Pages with many products return Error 500 when Varnish is enabled

[eldonbite]

Encountered an issue wherein certain product listing pages with 60 products per page loads properly when using M2's built-in FPC but returns Error 500 when using Varnish.

Preconditions

1. Magento 2.0.x up to 2.1.x
2. Store has products with many product options
3. All cache types are enabled

Steps to reproduce

1. Configure the store to make use of Varnish cache instead of built-in FPC
2. Set the number of products to display in the catalog to a high value (in our case, 60)
3. Load a product listing page

Expected result

The product listing page should display properly

Actual result

Page returns Error 500

After a lot of hours spent in investigating the issue and even trying to tune Varnish by configuring certain parameters, I noticed this recently submitted issue here in the M2 repository and tried to look into the X-Magento-Tags header.

#7640 X-Magento-Tags header containing whitespaces causes exception

I didn't find any pages that has spaces in the said header, nor blocks with block ID containing spaces, but I noticed something: perhaps due to the sheer amount of products that is being displayed in our product listing pages, some of them have X-Magento-Tags header lengths that reach up to more than 16k characters. I commented-out the "unset" statements in the default.vcl Varnish configuration file to check the contents of the said header and one of them are as follows:

store,cms_block,cms_block_header_mini_banner,catalog_category_42,catalog_category,catalog_category_product_42,catalog_product_13897,catalog_product,catalog_product_8512,catalog_product_8498,catalog_product_8499,catalog_product_8500,catalog_product_8501,catalog_product_8502,catalog_product_8503,catalog_product_8504,catalog_product_8513,catalog_product_8506,catalog_product_8507,catalog_product_8508,catalog_product_8509,catalog_product_8510,catalog_product_8511,catalog_product_13898,catalog_product_8514,catalog_product_8515,catalog_product_8516,catalog_product_8517,catalog_product_8518,catalog_product_8519,catalog_product_8520,catalog_product_8521,catalog_product_8522,catalog_product_8523,catalog_product_8524,catalog_product_8525,catalog_product_8526,catalog_product_8527,catalog_product_8528,catalog_product_13932,catalog_product_8590,catalog_product_8591,catalog_product_8592,catalog_product_8593,catalog_product_8594,catalog_product_8595,catalog_product_8603,catalog_product_8596,catalog_product_8597,catalog_product_8598,catalog_product_8599,catalog_product_8600,catalog_product_8601,catalog_product_8602,catalog_product_8632,catalog_product_8633,catalog_product_8634,catalog_product_8635,catalog_product_8636,catalog_product_8637,catalog_product_16189,catalog_product_16190,catalog_product_16191,catalog_product_16192,catalog_product_16193,catalog_product_16194,catalog_product_16195,catalog_product_16196,catalog_product_13959,catalog_product_8721,catalog_product_8722,catalog_product_8723,catalog_product_8724,catalog_product_8725,catalog_product_8726,catalog_product_8727,catalog_product_8728,catalog_product_8729,catalog_product_8730,catalog_product_15754,catalog_product_15755,catalog_product_15756,catalog_product_15757,catalog_product_15758,catalog_product_15759,catalog_product_15760,catalog_product_15761,catalog_product_15762,catalog_product_18991,catalog_product_18992,catalog_product_18993,catalog_product_18994,catalog_product_18995,catalog_product_13971,catalog_product_8793,catalog_product_8788,catalog_product_8789,catalog_product_8790,catalog_product_8791,catalog_product_8792,catalog_product_8794,catalog_product_8795,catalog_product_8796,catalog_product_8797,catalog_product_8798,catalog_product_8799,catalog_product_8812,catalog_product_8813,catalog_product_8814,catalog_product_8815,catalog_product_8816,catalog_product_8817,catalog_product_16026,catalog_product_16027,catalog_product_16028,catalog_product_16029,catalog_product_16030,catalog_product_16031,catalog_product_16032,catalog_product_16033,catalog_product_16034,catalog_product_16035,catalog_product_16036,catalog_product_16038,catalog_product_16039,catalog_product_16040,catalog_product_16041,catalog_product_16042,catalog_product_16043,catalog_product_17363,catalog_product_17364,catalog_product_17365,catalog_product_17366,catalog_product_17367,catalog_product_17368,catalog_product_13977,catalog_product_8837,catalog_product_8838,catalog_product_8839,catalog_product_8840,catalog_product_8841,catalog_product_8842,catalog_product_8845,catalog_product_8846,catalog_product_8847,catalog_product_8848,catalog_product_8849,catalog_product_8850,catalog_product_8851,catalog_product_8852,catalog_product_14004,catalog_product_8999,catalog_product_9000,catalog_product_9001,catalog_product_9002,catalog_product_9003,catalog_product_16175,catalog_product_9004,catalog_product_9005,catalog_product_9006,catalog_product_9007,catalog_product_9008,catalog_product_9009,catalog_product_9012,catalog_product_9013,catalog_product_9014,catalog_product_9015,catalog_product_9016,catalog_product_16176,catalog_product_16183,catalog_product_16184,catalog_product_16185,catalog_product_16186,catalog_product_16187,catalog_product_16188,catalog_product_16177,catalog_product_16178,catalog_product_16179,catalog_product_16180,catalog_product_16181,catalog_product_16182,catalog_product_17357,catalog_product_17358,catalog_product_17359,catalog_product_17360,catalog_product_17361,catalog_product_17362,catalog_product_14014,catalog_product_17411,catalog_product_9055,catalog_product_9056,catalog_product_9057,catalog_product_9058,catalog_product_9059,catalog_product_17412,catalog_product_9060,catalog_product_9061,catalog_product_9062,catalog_product_9063,catalog_product_9064,catalog_product_18526,catalog_product_18527,catalog_product_18528,catalog_product_18529,catalog_product_18530,catalog_product_18531,catalog_product_17413,catalog_product_9071,catalog_product_9072,catalog_product_9073,catalog_product_9074,catalog_product_9075,catalog_product_15866,catalog_product_15867,catalog_product_15868,catalog_product_15870,catalog_product_18532,catalog_product_18533,catalog_product_18534,catalog_product_18535,catalog_product_18536,catalog_product_18537,catalog_product_17415,catalog_product_17406,catalog_product_17407,catalog_product_18821,catalog_product_17410,catalog_product_14228,catalog_product_10288,catalog_product_10289,catalog_product_10290,catalog_product_10291,catalog_product_10292,catalog_product_10295,catalog_product_10296,catalog_product_10297,catalog_product_10298,catalog_product_10299,catalog_product_10300,catalog_product_10302,catalog_product_10303,catalog_product_10304,catalog_product_10305,catalog_product_10306,catalog_product_10307,catalog_product_10310,catalog_product_10311,catalog_product_10313,catalog_product_10314,catalog_product_15961,catalog_product_15962,catalog_product_15963,catalog_product_15964,catalog_product_15965,catalog_product_15966,catalog_product_15967,catalog_product_15968,catalog_product_15969,catalog_product_15971,catalog_product_15972,catalog_product_15974,catalog_product_14231,catalog_product_10318,catalog_product_10319,catalog_product_10320,catalog_product_10321,catalog_product_10322,catalog_product_10323,catalog_product_10324,catalog_product_14240,catalog_product_10406,catalog_product_10407,catalog_product_10408,catalog_product_10409,catalog_product_10410,catalog_product_10411,catalog_product_15860,catalog_product_15861,catalog_product_15862,catalog_product_15863,catalog_product_17530,catalog_product_17531,catalog_product_17532,catalog_product_17533,catalog_product_17534,catalog_product_14272,catalog_product_10660,catalog_product_10661,catalog_product_10662,catalog_product_10663,catalog_product_10664,catalog_product_10665,catalog_product_10666,catalog_product_10667,catalog_product_10668,catalog_product_10669,catalog_product_10670,catalog_product_10671,catalog_product_10672,catalog_product_10673,catalog_product_15883,catalog_product_15884,catalog_product_15885,catalog_product_15886,catalog_product_15887,catalog_product_15888,catalog_product_15889,catalog_product_14277,catalog_product_10701,catalog_product_10702,catalog_product_10703,catalog_product_10704,catalog_product_10705,catalog_product_10711,catalog_product_10712,catalog_product_10713,catalog_product_10714,catalog_product_10715,catalog_product_10706,catalog_product_10707,catalog_product_10708,catalog_product_10709,catalog_product_10710,catalog_product_15692,catalog_product_15693,catalog_product_15694,catalog_product_15695,catalog_product_15696,catalog_product_15697,catalog_product_15698,catalog_product_15699,catalog_product_15700,catalog_product_15701,catalog_product_17834,catalog_product_17835,catalog_product_17836,catalog_product_17837,catalog_product_17838,catalog_product_14343,catalog_product_11360,catalog_product_11361,catalog_product_11362,catalog_product_11363,catalog_product_11364,catalog_product_11365,catalog_product_11354,catalog_product_11355,catalog_product_11356,catalog_product_11357,catalog_product_11358,catalog_product_16012,catalog_product_16013,catalog_product_16014,catalog_product_16015,catalog_product_16016,catalog_product_16017,catalog_product_16019,catalog_product_16020,catalog_product_16021,catalog_product_16022,catalog_product_16023,catalog_product_17304,catalog_product_17305,catalog_product_17306,catalog_product_17307,catalog_product_17308,catalog_product_17309,catalog_product_17310,catalog_product_18996,catalog_product_18997,catalog_product_18998,catalog_product_18999,catalog_product_19000,catalog_product_19001,catalog_product_19002,catalog_product_19003,catalog_product_14357,catalog_product_11543,catalog_product_11544,catalog_product_11545,catalog_product_11546,catalog_product_11547,catalog_product_11548,catalog_product_15891,catalog_product_15893,catalog_product_15894,catalog_product_15895,catalog_product_15897,catalog_product_15898,catalog_product_15899,catalog_product_15901,catalog_product_18234,catalog_product_18235,catalog_product_18236,catalog_product_18237,catalog_product_18238,catalog_product_14359,catalog_product_11572,catalog_product_11573,catalog_product_11574,catalog_product_11575,catalog_product_11576,catalog_product_11577,catalog_product_11595,catalog_product_11596,catalog_product_11597,catalog_product_11598,catalog_product_11599,catalog_product_11600,catalog_product_11589,catalog_product_11590,catalog_product_11591,catalog_product_11592,catalog_product_11593,catalog_product_11594,catalog_product_16171,catalog_product_17313,catalog_product_17314,catalog_product_17315,catalog_product_17316,catalog_product_17317,catalog_product_17318,catalog_product_17344,catalog_product_17345,catalog_product_17346,catalog_product_17347,catalog_product_17348,catalog_product_17349,catalog_product_17828,catalog_product_17829,catalog_product_17830,catalog_product_17831,catalog_product_17832,catalog_product_14366,catalog_product_11653,catalog_product_11654,catalog_product_11655,catalog_product_11656,catalog_product_11657,catalog_product_11658,catalog_product_11659,catalog_product_11660,catalog_product_11661,catalog_product_11662,catalog_product_11663,catalog_product_11664,catalog_product_11680,catalog_product_11681,catalog_product_11682,catalog_product_11683,catalog_product_11684,catalog_product_11685,catalog_product_16071,catalog_product_16072,catalog_product_16073,catalog_product_16074,catalog_product_16075,catalog_product_18208,catalog_product_18209,catalog_product_18210,catalog_product_18211,catalog_product_18212,catalog_product_18213,catalog_product_18214,catalog_product_18215,catalog_product_18216,catalog_product_18217,catalog_product_14423,catalog_product_12004,catalog_product_12005,catalog_product_12006,catalog_product_12007,catalog_product_12008,catalog_product_12009,catalog_product_12010,catalog_product_12011,catalog_product_12012,catalog_product_12013,catalog_product_12014,catalog_product_12016,catalog_product_12017,catalog_product_12018,catalog_product_12019,catalog_product_12020,catalog_product_12021,catalog_product_12022,catalog_product_12023,catalog_product_16077,catalog_product_16078,catalog_product_16079,catalog_product_16080,catalog_product_16081,catalog_product_16082,catalog_product_16083,catalog_product_16084,catalog_product_16085,catalog_product_16086,catalog_product_16087,catalog_product_16088,catalog_product_18353,catalog_product_14436,catalog_product_12214,catalog_product_12215,catalog_product_12216,catalog_product_12217,catalog_product_12218,catalog_product_12219,catalog_product_14441,catalog_product_12293,catalog_product_12294,catalog_product_12295,catalog_product_12296,catalog_product_12297,catalog_product_12298,catalog_product_12305,catalog_product_12306,catalog_product_12307,catalog_product_12308,catalog_product_12309,catalog_product_12310,catalog_product_12311,catalog_product_12312,catalog_product_12313,catalog_product_12314,catalog_product_12315,catalog_product_12316,catalog_product_16094,catalog_product_16095,catalog_product_16096,catalog_product_16097,catalog_product_16098,catalog_product_16099,catalog_product_16101,catalog_product_16102,catalog_product_16103,catalog_product_16104,catalog_product_16105,catalog_product_16106,catalog_product_14456,catalog_product_12450,catalog_product_12451,catalog_product_12452,catalog_product_12453,catalog_product_12454,catalog_product_12455,catalog_product_15930,catalog_product_15924,catalog_product_15925,catalog_product_15926,catalog_product_15927,catalog_product_15928,catalog_product_15929,catalog_product_15931,catalog_product_15918,catalog_product_15919,catalog_product_15920,catalog_product_15921,catalog_product_15922,catalog_product_15923,catalog_product_15932,catalog_product_14513,catalog_product_12692,catalog_product_12693,catalog_product_12694,catalog_product_12695,catalog_product_12696,catalog_product_12697,catalog_product_15878,catalog_product_15879,catalog_product_15880,catalog_product_15881,catalog_product_15882,catalog_product_14515,catalog_product_12707,catalog_product_12708,catalog_product_12709,catalog_product_12710,catalog_product_12711,catalog_product_12712,catalog_product_12713,catalog_product_15910,catalog_product_15911,catalog_product_15912,catalog_product_15913,catalog_product_15914,catalog_product_15915,catalog_product_15916,catalog_product_15903,catalog_product_15904,catalog_product_15905,catalog_product_15906,catalog_product_15907,catalog_product_15908,catalog_product_15909,catalog_product_18411,catalog_product_18412,catalog_product_18413,catalog_product_18414,catalog_product_18415,catalog_product_18416,catalog_product_18405,catalog_product_18406,catalog_product_18407,catalog_product_18408,catalog_product_18409,catalog_product_18410,catalog_product_14516,catalog_product_12714,catalog_product_12715,catalog_product_12716,catalog_product_12717,catalog_product_12718,catalog_product_12719,catalog_product_15934,catalog_product_15935,catalog_product_15936,catalog_product_15937,catalog_product_15938,catalog_product_15939,catalog_product_15940,catalog_product_15941,catalog_product_15942,catalog_product_15943,catalog_product_15944,catalog_product_15945,catalog_product_15946,catalog_product_15947,catalog_product_15948,catalog_product_15949,catalog_product_15950,catalog_product_15951,catalog_product_15952,catalog_product_15953,catalog_product_15954,catalog_product_15955,catalog_product_17443,catalog_product_17444,catalog_product_17445,catalog_product_17446,catalog_product_17447,catalog_product_17448,catalog_product_17449,catalog_product_15460,catalog_product_15436,catalog_product_15437,catalog_product_15438,catalog_product_15439,catalog_product_15440,catalog_product_15441,catalog_product_15443,catalog_product_15444,catalog_product_15445,catalog_product_15446,catalog_product_15448,catalog_product_15449,catalog_product_15450,catalog_product_15452,catalog_product_15455,catalog_product_15456,catalog_product_15457,catalog_product_15458,catalog_product_15461,catalog_product_15462,catalog_product_15463,catalog_product_15464,catalog_product_17394,catalog_product_17395,catalog_product_17396,catalog_product_17397,catalog_product_17398,catalog_product_17399,catalog_product_16859,catalog_product_16838,catalog_product_16839,catalog_product_16840,catalog_product_16841,catalog_product_16842,catalog_product_16843,catalog_product_16844,catalog_product_16845,catalog_product_16846,catalog_product_16847,catalog_product_16848,catalog_product_16849,catalog_product_16850,catalog_product_16851,catalog_product_16852,catalog_product_16853,catalog_product_16854,catalog_product_16855,catalog_product_16856,catalog_product_16857,catalog_product_16858,catalog_product_18591,catalog_product_18592,catalog_product_18593,catalog_product_18594,catalog_product_18595,catalog_product_18596,catalog_product_18597,catalog_product_16964,catalog_product_16958,catalog_product_16959,catalog_product_16960,catalog_product_16961,catalog_product_16962,catalog_product_16963,catalog_product_16971,catalog_product_17043,catalog_product_17037,catalog_product_17038,catalog_product_17039,catalog_product_17040,catalog_product_17041,catalog_product_17042,catalog_product_17044,catalog_product_17051,catalog_product_17045,catalog_product_17046,catalog_product_17047,catalog_product_17048,catalog_product_17049,catalog_product_17050,catalog_product_18691,catalog_product_18488,catalog_product_18489,catalog_product_18490,catalog_product_18491,catalog_product_18692,catalog_product_18693,catalog_product_17053,catalog_product_17054,catalog_product_17055,catalog_product_17056,catalog_product_17057,catalog_product_17058,catalog_product_17060,catalog_product_18694,catalog_product_18493,catalog_product_18494,catalog_product_18495,catalog_product_18496,catalog_product_18497,catalog_product_18695,catalog_product_18696,catalog_product_16965,catalog_product_16967,catalog_product_16968,catalog_product_16969,catalog_product_18119,catalog_product_18120,catalog_product_18121,catalog_product_18122,catalog_product_18123,catalog_product_18124,catalog_product_19004,catalog_product_19005,catalog_product_19006,catalog_product_19007,catalog_product_19008,catalog_product_19009,cms_block_customer-register,cms_block_,cms_block_newsletter_popup

I paginated through the category and found a page that loads properly even with X-Magento-Tags header containing almost 5000 characters.

So what I did was to try modifying the setHeader() function of vendor/magento/framework/HTTP/PhpEnvironment/Response.php to create a sha1 hash of the header's value and then the pages started loading properly.

So I guess a possible fix for this is to either hash the value of the X-Magento-Tags or perhaps compress it if we want a reversible string, e.g.:

$value = base64_encode(gzencode($value));

I'm just not sure of the implications of this solution, like when cleaning or flushing the cache or whatever. I'll just leave it to you guys. Thanks!

[cizgidekiadam]

Same in here @eldonbite.
Magento version 2.1.2

When tried to enter a page with configurables, got 500 with varnish. Using built-in cache it seems ok.

[eldonbite]

Some updates: apparently changing certain Varnish parameters won't help. It's actually due to the maximum header size supported by Apache, which is 8KB:

http://stackoverflow.com/a/8623061

To ensure that having a huge number of configurable products in a product listing page works when Varnish is enabled, it's best to limit the size of the header to the said length (8190 bytes):

http://stackoverflow.com/a/6160643

In the meantime, as a workaround to our project, I created a patch for shortening the product cache tags both in generating the X-Magento-Tags header and when sending a purge request to Varnish. However, with the sheer amount of configurable products and swatches being displayed in our product listings, I had to lower the number of products to display per page down to 36.

diff --git 1/vendor/magento/framework/HTTP/PhpEnvironment/Response.php 2/vendor/magento/framework/HTTP/PhpEnvironment/Response.php
index 8c8a90a..81e98d6 100644
--- 1/vendor/magento/framework/HTTP/PhpEnvironment/Response.php
+++ 2/vendor/magento/framework/HTTP/PhpEnvironment/Response.php
@@ -89,6 +89,9 @@ class Response extends \Zend\Http\PhpEnvironment\Response implements \Magento\Fr
     public function setHeader($name, $value, $replace = false)
     {
         $value = (string)$value;
+        if ($name == 'X-Magento-Tags') {
+            $value = str_replace('catalog_product_', 'p', $value);
+        }
 
         if ($replace) {
             $this->clearHeader($name);
diff --git 1/vendor/magento/module-cache-invalidate/Observer/InvalidateVarnishObserver.php 2/vendor/magento/module-cache-invalidate/Observer/InvalidateVarnishObserver.php
index 44a9317..ef3f2bb 100644
--- 1/vendor/magento/module-cache-invalidate/Observer/InvalidateVarnishObserver.php
+++ 2/vendor/magento/module-cache-invalidate/Observer/InvalidateVarnishObserver.php
@@ -48,7 +48,7 @@ class InvalidateVarnishObserver implements ObserverInterface
                 $tags = [];
                 $pattern = "((^|,)%s(,|$))";
                 foreach ($object->getIdentities() as $tag) {
-                    $tags[] = sprintf($pattern, $tag);
+                    $tags[] = sprintf($pattern, str_replace('catalog_product_', 'p', $tag));
                 }
                 if (!empty($tags)) {
                     $this->purgeCache->sendPurgeRequest(implode('|', array_unique($tags)));

[eldonbite]

Some more updates:

By doing Huffman coding string replacement on the tags, I managed to further shorten the size of the X-Magento-Tags header by almost 80%. Some examples:

40 products per page:
Original length: 21962
Shortened length: 4904

60 products per page:
Original length: 26289
Shortened length: 5913

I hope the Magento team could implement some sort of cache tag compression similar to what I did.

[cizgidekiadam]

@eldonbite Thanks for info.

Sounds insteresting.
How do you implement Huffman? Is there a library for magento?
We are as 'Magento Folk', need too much compression like that one.

Platform great but it is a really resource eater.

[eldonbite]

Hi @rbostan,

It's not exactly a full-blown Huffman encoding algorithm. It's actually just doing a dictionary-based prefix replacement to the cache tags multiple times. For example:

Tags:
catalog_product_12345,catalog_product_12346..

Replacements:

• catalog_product_ => p
• p1 => q
• q2 => r

This will shorten the product cache tags to just:
r345,r346

HTH 👍

[cizgidekiadam]

@eldonbite Brilliant!
I will also try it

[eldonbite]

Didn't see this reported issue (due to the large stack of issues here):

X-Magento-Tags header too large #6401

Looks like that one already has a fix in M2.1.3, so I guess upgrading should fix this issue as well.

[eldonbite]

Just confirmed that Magento 2.1.3 has already fixed the issue with the large X-Magento-Tags header after I updated the staging instance of our project. Please feel free to close this issue, unless you also plan to work on compressing the tags and allow stores to display as many products per page as possible.

[choukalos]

Closing issue per comment from @eldonbite as confirmation.

[eldonbite]

After upgrading to 2.1.3, we were encountering problems similar to this reported issue: #8009

[binnykumar]

Hi All,

Anyone has fix for this issue ??
I am also getting this issue for both vernish and built-in cache.

Check this url:
http://beta.thebodyshop.in

[kurtdeserter]

Any reason, why this issue is closed. Issue is still there in 2.3.3