Magento_CacheInvalidate mis-handles very large tag patterns when doing a PURGE

[moloughlin]

Preconditions:
\Magento\CacheInvalidate\Model\PurgeCache::sendPurgeRequest is responsible for taking arbitrarily large tag patterns and chunking it into HTTP PURGE requests based on the maximum header length Varnish will accept.

Currently, \Magento\CacheInvalidate\Observer\InvalidateVarnishObserver::execute will join tags with '|' and call sendPurgeRequest. Those tags themselves may include '|' characters.

\Magento\CacheInvalidate\Model\PurgeCache::splitTags will then separate on '|' and re-join components until the length of those components reaches the max. header length. This is too naive, since it may now split an individual tag (one which itself included '|' characters) across two requests. Both of those requests now have an invalid pattern - this would mean that at the very least, some purges will be missed. In the worst-case scenario, I expect this could also lead to Varnish being completely purged, depending on the tag that was incorrectly split.

For the sake of brevity, the example below has been limited to a small subset of my real-world example; as such let's pretend the max header size is 256 bytes. Note that this is a typical pattern that would result from a product save event.

((^|,)cat_p_1468(,|$))|((^|,)cat_p_1361(,|$))|((^|,)cat_p_1473(,|$))|((^|,)cat_p_930(,|$))|((^|,)cat_p_933(,|$))|((^|,)cat_p_934(,|$))|((^|,)cat_p_664(,|$))|((^|,)cat_p_487(,|$))|((^|,)cat_p_490(,|$))|((^|,)cat_p_491(,|$))|((^|,)cat_p_153(,|$))|((^|,)cat_p_156(,|$))|((^|,)cat_p_157(,|$))|((^|,)cat_p_497(,|$))|((^|,)cat_p_498(,|$))|((^|,)cat_p_499(,|$))|((^|,)cat_p_227(,|$))|((^|,)cat_p_228(,|$))|((^|,)cat_p_229(,|$))|((^|,)cat_p_242(,|$))|((^|,)cat_p_244(,|$))|((^|,)cat_p_245(,|$))

Based on my desired max. header size, this will be chunked into two separate requests:

((^|,)cat_p_1468(,|$))|((^|,)cat_p_1361(,|$))|((^|,)cat_p_1473(,|$))|((^|,)cat_p_930(,|$))|((^|,)cat_p_933(,|$))|((^|,)cat_p_934(,|$))|((^|,)cat_p_664(,|$))|((^|,)cat_p_487(,|$))|((^|,)cat_p_490(,|$))|((^|,)cat_p_491(,|$))|((^|,)cat_p_153(,|$))|((^

,)cat_p_156(,|$))|((^|,)cat_p_157(,|$))|((^|,)cat_p_497(,|$))|((^|,)cat_p_498(,|$))|((^|,)cat_p_499(,|$))|((^|,)cat_p_227(,|$))|((^|,)cat_p_228(,|$))|((^|,)cat_p_229(,|$))|((^|,)cat_p_242(,|$))|((^|,)cat_p_244(,|$))|((^|,)cat_p_245(,|$))|((^|,)cat_p_245(,|$))

Expected split behaviour:
((^|,)cat_p_1468(,|$))|((^|,)cat_p_1361(,|$))|((^|,)cat_p_1473(,|$))|((^|,)cat_p_930(,|$))|((^|,)cat_p_933(,|$))|((^|,)cat_p_934(,|$))|((^|,)cat_p_664(,|$))|((^|,)cat_p_487(,|$))|((^|,)cat_p_490(,|$))|((^|,)cat_p_491(,|$))|((^|,)cat_p_153(,|$))

((^|,)cat_p_156(,|$))|((^|,)cat_p_157(,|$))|((^|,)cat_p_497(,|$))|((^|,)cat_p_498(,|$))|((^|,)cat_p_499(,|$))|((^|,)cat_p_227(,|$))|((^|,)cat_p_228(,|$))|((^|,)cat_p_229(,|$))|((^|,)cat_p_242(,|$))|((^|,)cat_p_244(,|$))|((^|,)cat_p_245(,|$))

---

Steps to reproduce:

• Have full_page cache enabled
• Have caching application configured to Varnish
• Run code below

<?php

require __DIR__ . '/app/bootstrap.php';
$bootstrap = \Magento\Framework\App\Bootstrap::create(BP, $_SERVER);

class DummyEntity implements \Magento\Framework\DataObject\IdentityInterface
{
    public function getIdentities()
    {
        $ids = range(807, 1193);
        return array_map(function ($id) {
            return 'cat_p_' . $id;
        }, $ids);
    }
}

$objectManager = $bootstrap->getObjectManager();
$appState = $objectManager->get('Magento\Framework\App\State');
$appState->setAreaCode('frontend');
$eventManager = $objectManager->get('\Magento\Framework\Event\Manager');

$myObject = new DummyEntity();

$eventManager->dispatch('clean_cache_by_tags', ['object' => $myObject]);

Actual Result:
Observe two resulting PURGE requests sent to Varnish with invalid patterns (e.g. xdebug_break inside \Magento\CacheInvalidate\Model\PurgeCache::sendPurgeRequestToServers and observe the X-Magento-Tags-Pattern header)

Expected Result:

[m2-assistant]

Hi @moloughlin. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

@moloughlin do you confirm that you were able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• yes
• no

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[moloughlin]

Proposed solution: #26256

[ihor-sviziev]

This issue is duplicate of #8815, but as it have quite a good description I think it's good to keep it open.

[magento-engcom-team]

@engcom-Alfa Thank you for verifying the issue.

Unfortunately, not enough information was provided to acknowledge ticket. Please consider adding the following:

• Add "Component: " label(s) to this ticket based on verification result. If uncertain, you may follow the best guess

Once all required information is added, please add label "Issue: Confirmed" again.
Thanks!

[magento-engcom-team]

✅ Confirmed by @engcom-Alfa
Thank you for verifying the issue. Based on the provided information internal tickets MC-36618 were created

Issue Available: @engcom-Alfa, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

[magento-engcom-team]

Hi @moloughlin. Thank you for your report.
The issue has been fixed in #26256 by @moloughlin in 2.4-develop branch
Related commit(s):

• e2123017375ee389cb49514c7b1e3d60b9aa6e1b
• 4c223031e12bf6de7380ff8b13688d475f92f8d0
• fc2ce73d3352ad7180f64540c5369193ebb413f9
• a9d376d7c9c9a0e0dd37cb4c485ff7e93f56c764
• 5e69e824709d2a3eece6cffd4b04c82f9c891dd0
• 479c0cc0ec1a0e0af34efba671274102db7542e3
• d0ef6aea8f4e8690c6e164a4445c4847245c4ce9
• 85b3ffd90bfc9b415c6e8fcfb990caf92ab7980e
• 116c1910520fe2081e09e91dca6b04716f25dbbb
• 0926f5c008d0c272fc7828fbe2cfed60d93ac8cc
• ffb1174bfcd19b9f3de04a59ad1ec10c0b4913d2
• 99b26a4f9c22462e35d17ed1958d66fb31fe9e28
• a19274e203a884b92e0fcd2b0cda40ea4d4e8080

The fix will be available with the upcoming 2.4.1 release.