Occasionally getting JSON strings on category pages

[dyuk1987]

We have encountered this issue on a few different Magento projects. We understand we may not have sufficient information or very clear procedures for reproducing the issue, but as this turned out to be quite annoying and we have no clue about the root cause at all, we still decided to raise this here. And we have seen similar issues on StackOverflow.

1. https://magento.stackexchange.com/questions/296351/magento-2-intermittent-cache-issue-showing-a-json-code-instead-of-homepage
2. https://magento.stackexchange.com/questions/296944/magento2-category-page-cached-as-json-data-in-varnish

Preconditions (*)

Case 1:

1. Magento 2.2.10 on AWS
2. Varnish 4.0.5

Case 2:

1. Magento 2.2.5 on Azure
2. Varnish 4.0.5

Steps to reproduce (*)

1. Browse any category page
2. Reach a cached page with JSON strings occasionally

Expected result (*)

1. Show a normal category page

Actual result (*)

1. Reach a cached page with JSON strings occasionally

It happened (seemingly) randomly, and after restarting the varnish service, the issue would be gone.

[jbclaudio]

I've exact the same issue at some specific category pages. Has anyone find a solution?

[pinedamg]

I'm experiencing the same issue here, I thought it was the Amasty Extension Layered Navigation, but is still happening without the extension, definitely is something related with Varnish and Magento Core, and clearly this happens on a random way, and there is not a pattern or clue about how to reproduce it.

Does someone have any clue about it? thank you very much in advance.

Apache version: Apache/2.4.6 (CentOS)
Magento: Commerce 2.3.4
Varnish: (varnish-5.2.1 revision 67e562482)

Update:
I have tried disabling the whole varnish, just making a bypass, and the issue still happening. (of course cleared cache before/after)

If you are having the same issue, please could you let us know your current request flow?, I'm pretty sure this is a forwarding/redirect request issue with Magento of course.

Apache Proxy SSL Termination + Varnish = Same Server
Magento Instance = Another Server same setup (Apache)

The current setup is, Client -> CloudFlare (free) -> Apache (Proxy SSL termination) -> Varnish -> Magento

Thanks,

[jbclaudio]

I have found the cause and the solution. The cause is that when Varnish is cleared and a user comes to a filtered page first, the JSON is cached. I have excluded this for Varnish and this solved the problem.

[dyuk1987]

I have found the cause and the solution. The cause is that when Varnish is cleared and a user comes to a filtered page first, the JSON is cached. I have excluded this for Varnish and this solved the problem.

@jbclaudio
O-M-G finally someone came up with a solution!
That's some amazing findings there mate.
Do you mind sharing the VCL rules you have added?

[ZIPITRO]

I have found the cause and the solution. The cause is that when Varnish is cleared and a user comes to a filtered page first, the JSON is cached. I have excluded this for Varnish and this solved the problem.

Wow, great job! Would you please share with us your VCL rules?

[pinedamg]

I have found the cause and the solution. The cause is that when Varnish is cleared and a user comes to a filtered page first, the JSON is cached. I have excluded this for Varnish and this solved the problem.

@jbclaudio That sounds great, but I'm curious because I've tried disabling Varnish and this still happening, are you completely sure is fixed? Did you were able to reproduce the issue to verify if the fix is real? I would like to think that you are our savior in this very rare issue :P, and as our bug-mates said, please could you show us that portion of the VCL file?, thanks

[dyuk1987]

I have found the cause and the solution. The cause is that when Varnish is cleared and a user comes to a filtered page first, the JSON is cached. I have excluded this for Varnish and this solved the problem.

On second thought, I could not figure out the relationship between the filtered page and the unfiltered page.
Aren't those filtered pages standalone cache entries in Varnish?

And I tried to follow your findings, but still could not find a way to 100% reproduce the issue as well.
Could you please also advise how you could do that?

[jbclaudio]

I've added this to the .vcl file:

    if (req.url ~ "shopbyAjax") {
        return (pass);
    }

This project uses the Amasty Shopby module. This added the parameter to the filter pages, which I could use. You may be able to use another parameter that Magento itself adds.

After this has been added to the .vcl file, the problem no longer occurs.

[TJEliot]

I just encountered this and was able to quickly resolve it, so I thought I would share my solution here: I just reran the catalog_product_category and catalog_category_product indexers

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 14 days if no further activity occurs. Is this issue still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? Thank you for your contributions!

[dwestvik]

This happened, today, on our site and don't have Amasty.
Magento 2.3

[Daniel304]

Same here... I do found out that it is a combination of google pagespeed and varnish in my case.. and it is only on specific category pages... Really strange behaviour not using amasty shopby

[jsamhall]

Encountered this issue today on a category with many products. Clearing block_html cache resolved the issue.

Magento 2.4.0 Open Source
PHP 7.4.9 with Zend 3.4.0 and OpCache
Apache 2.4.29
Ubuntu 18.04.0.4 LTS

We are NOT using Varnish and instead just Magento built-in FPC, so the issue is not specific to Varnish, nor the Amasty ShopBy module as other comments have cited.

What might of interest is that we are using Smile's ElasticSuite which results in all categories being treated as Layered Navigation categories.

Perhaps layered navigation, Elasticsearch and Magento's FPC are the culprit here?

Prior to one of our team members informing me of this error, I received a number of 503 Exception emails due to a session_start failure: Warning: session_start(): Failed to read session data: user (path: /var/lib/php/sessions) in /var/www/deployments/releases/current/vendor/magento/framework/Session/SessionManager.php on line 204. We are using Redis for Session Management.`

One of the URL's cited in the 503 Exception emails was indeed the category that suffered the "JSON Output" titular bug. Unfortunately I did not test the other URLs for which I received an email prior to cleaning the block_html cache.

Very scary issue to encounter "randomly" with no means of testing / troubleshooting / workaround besides from regularly cleaning of block cache.... obviously not desirable for performance.... but the lesser of two evils for sure!!!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 14 days if no further activity occurs. Is this issue still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? Thank you for your contributions!

[Claudiusdev]

I've he same issue at some specific category pages. Has anyone find a solution?

[kronthto]

The JSON response can be triggered by requesting the category-page with the header X-Requested-With: XMLHttpRequest only, no other parameters.
The response includes a Content-Type: application/json but no Vary for that, this in theory allows a cache to use it in place for the same URL requested as html.
Built-in FPC likely adheres to similar rules.

[aaubets]

The JSON response can be triggered by requesting the category-page with the header X-Requested-With: XMLHttpRequest only, no other parameters. The response includes a Content-Type: application/json but no Vary for that, this in theory allows a cache to use it in place for the same URL requested as html. Built-in FPC likely adheres to similar rules.

This is the key!

I've the same issue after enabling multiple currencies in my site (2.3.5 + varnish + apache), randomly this error appears in random categories.

Adding:

if (req.http.X-Requested-With) { hash_data(req.http.X-Requested-With); }

to vcl_hash solve the problem for me.