-
Notifications
You must be signed in to change notification settings - Fork 9.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Varnish "Connection reset by peer" error when large catalog is reindexed on schedule #8815
Comments
My solution to this problem was to add 2 plugins to Magento: Both solutions check if the 'X-Magento-Tags' header exists, and if so, truncates it to 1000 characters. The obvious implication here is that if you do any operation that would have such a long tags header, you must manually clear your full page cache to make sure no stale content is improperly stored. Here is a snippet from my Layout plugin:
and here is a snippet from my PurgeCache plugin:
I realize that this may not be a great solution for the Magento core code, but I think it's fine to use for anyone who understands the implications. Personally, I disable all cache flushing short of the "everything" (.*) value. Magento invalidates too much too often IMO. This makes me responsible for clearing caches after any changes, and I'm fine with that since we only make changes once a day. Hope this helps. |
In my opinion the best approach is to send the purge request by fix batch size, based on amount of tags or request length. This can be achieved by replacing the observer Magento\CacheInvalidate\Observer\InvalidateVarnishObserver and adding the batching mechanism. I might as well do a PR if I have the time in the next days. |
@AirmanAJK Thanks for sharing your workaround, but as you mentioned, that solution is not a good solution for the core code. @Vedrillan A batching solution sounds like a good solution to this problem. |
@AirmanAJK Thanks but here are results of implementation on Magento 2.1.4 Enterprice `exception(s): Exception #0 (Exception): Notice: Undefined property: MageXo\Fixers\Plugin\Layout\Interceptor::$response in /app/app/code/MageXo/Fixers/Plugin/Layout.php on line 8 |
Any update of this issue ? I report you the same issue with Magento 2 EE 2.0.7 with > 10K products |
@erikhansen @AirmanAJK @Vedrillan Have you guys found a solution to this problem? Do you have any code I could try and implement? Thanks. |
@denisrpriebe No, I don't have any patch or solution to share. |
@denisrpriebe Actually, I do have a patch you could try, although I never tested it. If you want it, email me at Update: I was incorrect—the patch I had was for a different issue. |
@erikhansen Cool, just shot you an email. |
Hi, any news on this patch? Experiencing the same issue on Magento 2.1.8 |
I did a PR there #8919 but they can't reproduce the issue and as a result closed the PR x) You can adapt a patch from this https://patch-diff.githubusercontent.com/raw/magento/magento2/pull/8919.diff and apply it on your project. |
For anyone else experiencing this, I had to disabled "Full Page Caching" within cache management in the admin...or you can do it through the command line. Once I did that, I no longer received the notice. |
@denisrpriebe Simply not using the full page cache is hardly a solution. If CMS pages weren't loading, you surely wouldn't suggest deleting them all to fix the problem, right? |
@AirmanAJK I understand your point. I'm not proposing a solution but rather an alternative for what worked for me. I think some documentation is better than none when it comes to a Magento issue. |
@Vedrillan Thanks! I have been having this problem forever and still have it with 2.1.9. Your patch seems like the obvious fix. No tags are lost and simply sent in batches. Hopefully it will be added to 2.2. |
@Vedrillan Thank you for your pull request, and it seems not managed correctly in 2.2 too... They may need to improve their test coverages to the limit in this file : |
Any movement on this issue? We had this happening and then implemented a patch which sent over the requests in smaller batches, however, this sometimes ended up with way too many requests sent to varnish and overloaded the ephemeral ports eventually creating 500 errors on the site for users after getting slow to process. Our solution most likely will have to be to increase the |
@J-Fricke We also received a patch that separated the invalidation request into smaller batches and quickly discovered that every time the cron jobs run a PURGE request with almost every active product in the catalog was being sent to the Varnish server for processing. This happens even when there are no updates happening in the admin or via the API. Separating the large PURGE requests into batches caused too many PURGE requests to the varnish server each minute and caused memory issues with the Varnish process because of the constant requests to remove things from cache. I expect that increasing the |
@jdubbya We are currently testing removing the chunking patch and increasing the values Varnish can send/receive based on this: https://devdocs.magento.com/guides/v2.2/config-guide/varnish/tshoot-varnish-503.html - This guide only discusses http_resp_hdr_len & http_resp_size, though only increasing those values did not fix our issue. We have a current test we'll continue over the next couple of days with increasing http_req_hdr_len & http_req_size as well to matching values as the response ones. I'll update this thread once we are confident in the testing. As an aside, I'm curious what users would do if their catalog count * 21 exceeds the Varnish maximums would do to solve this issue? Perhaps a combinations of chunking and increasing the values to the max. I was pointed to a solution Fast.ly used for their extension which also leveraged some chunking, so that's probably the case. |
Hi @engcom-Delta. Thank you for working on this issue.
|
@joaolisboa @gigadesign1 @pczerkas Unfortunately, I cannot reproduce issue on 2.3-develop CE and 2.3-develop EE.
Result:
Result: Could you take a look if my steps are correct? |
From the 1st update |
@engcom-Delta, |
Hi @erikhansen. Thank you for your report.
The fix will be available with the upcoming 2.4.1 release. |
…-05-2024 [Support Tier-4-Kings glo16746] 03.05.2024 Regular delivery of bugfixes and improvements
Addition information for Re-Open
The Issue was reopened based on several complaints that it was not properly fixed and it still can be reproduced on the
2.2.8
and2.3.1
For more details and explanation see next comments:
Preconditions
http_req_hdr_len
andhttp_req_size
configured at their default values (per the documentation)Steps to reproduce
Expected result
The import will be processed without any errors.
Actual result
The cron will have an error when reindexing and will send an error like this to the cron email address:
The error above is triggered by line 375 of
vendor/zendframework/zend-http/src/Client/Adapter/Socket.php
In order to determine what request was causing the error, I edited the
Socket.php
file to add logging code at line 375 to log large requests, along with a backtrace:The requests that were causing errors were ~1.1MB in size, which exceeded the
http_req_size
default Varnish setting (which is 32k). While we could crank up that setting to allow the massive requests to be sent to Varnish, this would increase the Varnish memory footprint and is not an acceptable solution. Read this for context on the memory implications: https://www.varnish-cache.org/docs/4.0/reference/varnishd.html#http-req-sizeHere is a preview of what the 1.1MB request contained:
In case it is useful for troubleshooting this issue, here is the backtrace logged by the logging code I added to the
Socket.php
file:The text was updated successfully, but these errors were encountered: