Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

X-Magento-Tags header too large #6401

Closed
katef opened this issue Aug 30, 2016 · 72 comments
Closed

X-Magento-Tags header too large #6401

katef opened this issue Aug 30, 2016 · 72 comments
Assignees
Labels
bug report Component: Framework/Cache Fixed in 2.3.x The issue has been fixed in 2.3 release line Fixed in 2.4.x The issue has been fixed in 2.4-develop branch improvement Issue: Confirmed Gate 3 Passed. Manual verification of the issue completed. Issue is confirmed Issue: Format is valid Gate 1 Passed. Automatic verification of issue format passed Issue: Ready for Work Gate 4. Acknowledged. Issue is added to backlog and ready for development Reproduced on 2.3.x The issue has been reproduced on latest 2.3 release Reproduced on 2.4.x The issue has been reproduced on latest 2.4-develop branch

Comments

@katef
Copy link

katef commented Aug 30, 2016

Preconditions

I've been reading about people using Magento2 and the size of the X-Magento-Tags header.

Steps to reproduce

For example http://www.maxbucknell.com/blog/2016/2/10/troubleshooting-varnish-with-magento-2 writes:

The rub here is that Magento adds a tag for every product in a category. And the tag is of the form "catalog_product_{{PRODUCT_ID}}".

Actual result

That's an integer and 20 bytes of string per product, which gives some users headers several kB in size, when many products are in one category. In that case, the author describes hitting Varnish's limit at 400 products.

Expected result

I work for a CDN where some of our users use Magento, and we cache their content with Varnish. Increasing Varnish's http_resp_hdr_len size may suit that one person's situation, but it is infeasible for us, because it would affect all of our users.

Proposed solution

I have a suggestion to offer, for how to use header space more efficiently:

Each product ID is an integer. The X-Magento-Tags header describes the presence or absence of products only. That information can be done with a single bit per product. The index of the bit would give the product ID.

That's usually known as a bit vector, or a bitfield, or a bitmap. My suggestion is to switch the header to use a bitmap, encoded in hex. That'd take one bit per product, instead of over 20 bytes.

More on bitmaps: https://en.wikipedia.org/wiki/Bit_array

@katef katef changed the title X-Magneto-Tags header too large X-Magento-Tags header too large Aug 30, 2016
@stamster
Copy link

Excellent point!
But I'm afraid such change is not going to happen anytime soon, maybe for Magento 3.0... and this is only one of the topics out there to discuss when it comes to optimization.

@katef
Copy link
Author

katef commented Aug 31, 2016

Thank you - Although I'd request that you consider this a blocking item, rather than an optimisation, because this header is so large that it prevents certain users from using Varnish.

@melnikovi
Copy link
Member

melnikovi commented Aug 31, 2016

@katef thank you for reporting this issue, internal ticket created MAGETWO-57851

@stamster
Copy link

@katef : I'm not representative of Magento Commerce Inc. ... :) I completely agree with you, of course it is blocking for certain use cases out there.

@pynej
Copy link

pynej commented Sep 7, 2016

Some more detail:
The catalog_category_product_ and catalog_category_ could easily be changed to 2 characters that would still be unique and cut down the length of this header by ~80%.

Of critical issue though is that in Magento 2.1 all versions of a configured product are incorrectly added to this header field. In 2.0 only the parent item was added. As a result the size of the field grows exponentially and makes varnish unusable. I am trying to find the code in question that is incorrectly adding all versions so I can comment it out and restore 2.0 functionality and and have an actual working site.

@pynej
Copy link

pynej commented Sep 7, 2016

This workaround fixes but wastes lots or memory.

@pynej
Copy link

pynej commented Sep 7, 2016

I did this workaround to get my site working again. Basically disabling the addition of the child products of configurable items to the tag. There are cons't defined that specify the catalog_product_ tag, so that should be pretty straightforward to shorten.

--- vendor/magento/module-configurable-product/Plugin/Model/Product.php.orig    2016-09-07 14:10:22.201010000 -0400
+++ vendor/magento/module-configurable-product/Plugin/Model/Product.php 2016-09-07 14:08:17.107250000 -0400
@@ -26,9 +26,9 @@
         /** @var Configurable $productType */
         $productType = $product->getTypeInstance();
         if ($productType instanceof Configurable) {
-            foreach ($productType->getChildrenIds($product->getId())[0] as $productId) {
-                $result[] = \Magento\Catalog\Model\Product::CACHE_TAG . '_' . $productId;
-            }
+            #foreach ($productType->getChildrenIds($product->getId())[0] as $productId) {
+            #    $result[] = \Magento\Catalog\Model\Product::CACHE_TAG . '_' . $productId;
+            #}
         }
         return $result;
     }

@pboisvert
Copy link

@choukalos can you ensure this item gets the magetwo ticket? pulled the improvement label

@choukalos
Copy link

Tracking internally - MAGETWO-58265 ; Thanks @pynej for diagnosing and the code diff.

@veloraven
Copy link
Contributor

Internal ticket for 2.1 - MAGETWO-58362

@veloraven veloraven added the Issue: Ready for Work Gate 4. Acknowledged. Issue is added to backlog and ready for development label Oct 26, 2016
@choukalos
Copy link

Closing; this was resolved in 2.1.3

@katef
Copy link
Author

katef commented Feb 1, 2017

@choukalos What was the resolution?

@djfordz
Copy link

djfordz commented Feb 24, 2017

I'm still having issue with this in version 2.1.3

Error 503 Backend fetch failed

With Category Pages that have tons of categories.

@dcabrejas
Copy link

dcabrejas commented Apr 25, 2017

@choukalos I think this ticket should be reopen, I have encountered this issue on a Magento site running on 2.1.5. Category pages containing a large number of products return a X-Magento-Tag header which is bigger than the size allowed by the HTTP specification.

We have implemented this workaround for now http://devdocs.magento.com/guides/v2.0/config-guide/varnish/tshoot-varnish-503.html

But this is not a solution, just a workaround because the http header size is still larger than the limit set by the HTTp specification.

@redelschaap
Copy link
Contributor

Any update on this? This is still present in 2.1.7.

@tedsalmon
Copy link

As previously stated, this is still an issue in 2.1.8. I have implemented a Plugin around Magento\Catalog\Model\Product like so:

<?php

namespace YourCompany\YourModule\Plugin;

class Product{

    protected $_request;

    public function __construct(\Magento\Framework\App\Request\Http $request){
       $this->_request = $request;
    }

    /*
     * Return no simple product tags when returning category pages
     */
    public function afterGetIdentities(\Magento\Catalog\Model\Product $subject, $tags){
        if(!$this->_request){
            return $tags;
        }
        $r = $this->_request;
        $fullRoute = sprintf(
            '%s_%s_%s', $r->getRouteName(), $r->getControllerName(), $r->getActionName()
        );
        if($fullRoute === 'catalog_category_view'){
            return [];
        }else{
            return $tags;
        }
    }
}

?>

I very much realize that this is a hack, but we simply have too many products (over 64kB of tags) for the tag shortening to work. This will still show any parent product tags, FWIW.

@satishgadhave
Copy link

Can you pls provide link to pull request?

@magento-engcom-team magento-engcom-team added the Issue: Format is not valid Gate 1 Failed. Automatic verification of issue format is failed label Nov 17, 2017
@katef
Copy link
Author

katef commented Apr 27, 2020

@magento-engcom-team Wonderful, thank you!

The commits you linked to 404 though. Was that branch rebased and now they have different hashes now, perhaps?

@magento-engcom-team
Copy link
Contributor

Hi @katef, @o-iegorov.

Thank you for your report and collaboration!

The issue was fixed by Magento team. The fix was delivered into magento/magento2:2.4-develop branch(es).
Related commit(s):

The fix will be available with the upcoming 2.4.0 release.

@magento-engcom-team magento-engcom-team added the Fixed in 2.4.x The issue has been fixed in 2.4-develop branch label May 4, 2020
@ghost ghost moved this from Dev in Progress to Done (last 30 days) in Community Backlog May 4, 2020
@magento-engcom-team magento-engcom-team removed this from Done (last 30 days) in Community Backlog May 5, 2020
@katef
Copy link
Author

katef commented May 5, 2020

Hi @magento-engcom-team! That's great, thank you!
All your links 404 for me, though. Are those private perhaps?

@bmitchell-ldg
Copy link

Hi @magento-engcom-team we would also like to see the related commits for 2.4.0 if you would be able to resolve those 404 links please?

@ecoprince
Copy link

Hi @katef, @o-iegorov.

Thank you for your report and collaboration!

The issue was fixed by Magento team. The fix was delivered into magento/magento2:2.4-develop branch(es).
Related commit(s):

The fix will be available with the upcoming 2.4.0 release.

@magento-engcom-team All links goes to 404. Please update it.

@ihor-sviziev
Copy link
Contributor

@ecoprince,
Now these commits are available. Here is merge commit that contains all these changes: 3b611a7

@katef
Copy link
Author

katef commented Jun 6, 2020

Thank you!

@websecureNL
Copy link

Any way to get this fixed on 2.3.5? I understand the fix is made for 2.4 develop but for our production store we are using 2.3.5-p1 and having this problem.

@ihor-sviziev
Copy link
Contributor

@websecureNL you could try to create patch and apply on top of 2.3.5.
You can use https://devdocs.magento.com/guides/v2.4/comp-mgr/patching.html or https://github.com/vaimo/composer-patches

@websecureNL
Copy link

@ihor-sviziev, I tried this and it seems to work properly. We will do some further testing. Since making patches was new to me a quick hint for other users. We used this file (based on merge commit 3b611a7) to patch using instructions https://devdocs.magento.com/guides/v2.4/comp-mgr/patching.html

@websecureNL
Copy link

The problem also occurs when using catalogsearch. No idea if this also impacts 2.4 but since the patch did not fix this it could be.

@ihor-sviziev
Copy link
Contributor

ihor-sviziev commented Jul 2, 2020 via email

@websecureNL
Copy link

@ihor-sviziev. Of course I can do this but isn't this originated from Magento 2.3? We are having problems with 2.3.5 and would like to get it fixed there. I will check 2.4-develop asap.

@elvinristi
Copy link
Contributor

if you compare commits then actually nothing was done. was reverted probably something what didn't exist yet in M2.1 and therefor no matter of the version, issue continues.

temporary fix can be currently #6401 (comment)

but longer run should be similar what was done here but closed and ignored due inactivity #12831 which splits headers.

@ihor-sviziev
Copy link
Contributor

ihor-sviziev commented Aug 6, 2020

if you compare commits then actually nothing was done. was reverted probably something what didn't exist yet in M2.1 and therefor no matter of the version, issue continues.

temporary fix can be currently #6401 (comment)

but longer run should be similar what was done here but closed and ignored due inactivity #12831 which splits headers.

@elevinskii #6401 (comment) is really hack and not a correct solution, it breaks automatic cache flushing for category pages when any product was changed in the admin. definitely don't recommend it.

#12831 this solution is quite good, but it's not finalized in order to be accepted. You could create new PR based on this PR and apply requested changes. Only one concern - such change might break some services like Fastly (for instance used in Magento Cloud), so probably it should be double checked additionally

Also would like to add that there was a fix for configurable products that mostly caused this issue, this fix you can see in #6401 (comment)

BTW if you have steps to reproduce of this issue on 2.4-develop - please report it separately

@toonvd
Copy link

toonvd commented Aug 13, 2021

I just landed on this issue, a more complete solution would be to implement #6401 (comment). And create an observer on product save that gets parents on simple products and invalidates those too.

@amenk
Copy link
Contributor

amenk commented Aug 26, 2021

Still happens on some circumstances on 2.4.2. Is this header only needed if varnish is used? Can it be safely removed, if not?

@hostep
Copy link
Contributor

hostep commented Aug 30, 2021

@amenk: as far as I'm aware: yes! The X-Magento-Tags header is only used by Varnish to indicate what tags are associated to a certain url. So when Magento tells Varnish to purge some tags later on, it will purge all url's from its cache where that particular tag got associated to it.
So I believe you should be able to safely remove the header if you aren't using Varnish (or a similar reverse proxy that works with this header).

@amenk
Copy link
Contributor

amenk commented Aug 30, 2021

@hostep Thanks - we have a patch on StackOverflow: https://magento.stackexchange.com/questions/287621/ah01070-error-parsing-script-headers-in-php-fpm-when-accessing-certain-product .

What should be noted is that the clearing should happen after the tags are extracted from the Header and used for the built-in cache as well.

@atty31
Copy link

atty31 commented Nov 22, 2021

This happens on 2.4.3-p1 . Any updates or patches ? Thanks

@mlaurense
Copy link

After having raised the http_resp_hdr_len to 1MB (derived by calculating the largest number of products in a category multiplied by 21, as advised by Magento: https://support.magento.com/hc/en-us/articles/360034631211-Troubleshooting-503-error-caused-by-necessity-to-change-default-Varnish-settings), we were still experiencing intermittent 503 errors. These would happen on static files, e.g. javascripts and images.

To further debug this, I opened a varnishlog using the following command:
varnishlog -g request -q "RespStatus >= 500"
This returns all requests with a response code of 500 or higher. Now we just had to wait for an error to appear.

Soon after, the errors appeared:

--  FetchError     Workspace overflow
--  FetchError     overflow
[...]
--  BerespStatus   503
--  BerespReason   Service Unavailable
--  BerespReason   Backend fetch failed
--  Error          out of workspace (bo)

This is indicating a clear error, and pointed me in the right direction to further raise the workspace parameters as well. I am now using these parameters and the errors haven't reappeared:

    -p workspace_backend=2097152
    -p http_resp_size=1082768
    -p http_resp_hdr_len=1050000
    -p workspace_client=2097152
    -p http_req_size=65536
    -p http_req_hdr_len=32768

Of course, this shouldn't be necessary, but given the fact that Magento uses these very large tag headers, I'm afraid we have no choice other than to tune these varnish parameters.

@hostep
Copy link
Contributor

hostep commented Jan 8, 2022

Potential fix: #33468

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug report Component: Framework/Cache Fixed in 2.3.x The issue has been fixed in 2.3 release line Fixed in 2.4.x The issue has been fixed in 2.4-develop branch improvement Issue: Confirmed Gate 3 Passed. Manual verification of the issue completed. Issue is confirmed Issue: Format is valid Gate 1 Passed. Automatic verification of issue format passed Issue: Ready for Work Gate 4. Acknowledged. Issue is added to backlog and ready for development Reproduced on 2.3.x The issue has been reproduced on latest 2.3 release Reproduced on 2.4.x The issue has been reproduced on latest 2.4-develop branch
Projects
None yet
Development

No branches or pull requests