Adding Varnish causes category menu to switch to https links

[joebusby]

Steps to reproduce

Using Magento 2.0.5 with Varnish, on separate Debian 8
MySQL version 5.6.28
LXC 6 cores
PHP 5.6.17-0+deb8u1

1. Create a few sub-catagories on the "default catagory" and add configurable products to each
2. Ensure the home page menu is working and displaying the products (like in sample data except only one menu layer. Verify that hovering over category menu selections shows "http://" link at the bottom of the screen.
3. Configure Varnish in Stores/Configuration/Advanced/System/Full Page Cache
4. Clear all caches in Magento 2, Varnish, and the browser

Expected result

Operation should be the same as before Varnish was enabled.

Actual result

1. Hovering over the same category menu items sometimes shows a URL starting with "Https://"
2. Viewing the page with Https instead of Http causes the menu to not appear.

This is not immediately obvious, since clicking the menu link may or may not lead you to using "https" to browse the site (depending on what you click next), but when it does, it causes problems with the menu displaying correctly. This appears to be 2 bugs:

Bug 1: The category menus should work as they did before adding Varnish
Bug 2: The Menu does not appear when browsing the site in secure "Https" mode.

The problem with browsing is caused by the interaction of the two bugs.

Temporary work around:

Adding redirects in the .htaccess file located in the root directory under the "redirects for https" section like the below samples will keep the site in http mode so that varnish can work and the bugs don't interact

RewriteCond %{HTTPS} on
RewriteCond %{REQUEST_URI} /yellow-fruit.html
RewriteRule (.*) http://%{HTTP_HOST}%{REQUEST_URI} [L]

RewriteCond %{HTTPS} on
RewriteCond %{REQUEST_URI} /red-fruit.html
RewriteRule (.*) http://%{HTTP_HOST}%{REQUEST_URI} [L]

Is there a configuration change I can make to prevent the menu selections from producing "https" or that will get the category menus to appear when using https?

Other suggestions?

Does this happen with PHP7?

[choukalos]

Internal ticket for tracking - MAGETWO-52923

[bwaters]

Thanks for reporting I am looking into this. Your Bug 2: The Menu does not appear when browsing the site in secure "Https" mode. Is reported here #3897.

The best workaround is to add -p feature=+esi_ignore_https to /etc/sysconfig/varnish

I still need to track down your Bug 1 --
Can you tell me your settings for Stores -> Configuration -> Web -> Base URLs
and Stores -> Configuration -> Web Base URLs (Secure section)

Also are your ssl requests going straight to your apache instance or are they being terminated by a load balancer in front of your varnish cache?

[joebusby]

How does the "-p feature=+esi_ignore_https" fix the problem of the menus putting the site into Https mode? That defeats the point of Varnish right? Https does not cache. Maybe I have to try it to see. . .

Do I run the https through the varnish server then?

my Config:

Base URL
http://www.mydomain.com/
Specify URL or {{base_url}} placeholder.
Base Link URL
{{unsecure_base_url}}
May start with {{unsecure_base_url}} placeholder.
Base URL for Static View Files
http://cdn.mydomain.com/pub/static/
May be empty or start with {{unsecure_base_url}} placeholder.
Base URL for User Media Files
http://cdn.mydomain.com/pub/media/
May be empty or start with {{unsecure_base_url}} placeholder.

Secure Base URL
https://www.mydomain.com/
Specify URL or {{base_url}}, or {{unsecure_base_url}} placeholder.
Secure Base Link URL
{{secure_base_url}}

Not trying to cache https with CDN
http goes through varnish and uses CDN
https goes straight to the server / no CDN

My fix using .htaccess works well even though I have to re-enter it after every upgrade.

[bwaters]

You need to pass both http and https traffic through varnish. You do this by terminating the ssl before varnish in a load balancer like haproxy, nginx, or Amazon elastic load balancer.

When the varnish cache is enabled in magento it starts sending esi:include tags for blocks of content. If a request is made bypassing varnish (like your https requests), these esi tags will not interpreted by varnish and will be passed to the browser where they will be ignored, resulting in missing content.

Can you try and rework the configuration as I described above with your ssl traffic terminating at a load balancer which then hits your port 80 on the varnish cache. Also make sure that the default.vcl from the magento admin interface is loaded.

Combined with the -p feature=+esi_ignore_https to /etc/sysconfig/varnish change this should remove any missing content problems.

I have not been able to duplicate the https links inside the http request problem. Please let me know if you are experiencing this with the setup described above.

[joebusby]

I set up nginx as an https proxy in front of varnish, added the " -p feature=+esi_ignore_https " cleared all caches, disabled CDN and The backend worked fine (everything https), but the problem however categories are missing in the frontend using https to access the frontend. The problem appears to be related to:

http://www.mydomain.com/
Specify URL or {{base_url}} placeholder.
Base Link URL

I might get it to work setting the unsecure base url to https://www.mydomain.com, but that would not allow CDN operation. My .httaccess mod does not seem to work using https on the nginx reverse proxy, but a similar redirect solution on the reverse proxy might do it. (just a thought)

I am still working on the solution provided here, but no success yet:

http://magento.stackexchange.com/questions/114383/magento2-with-varnish-4-and-nginx-as-ssl-termination-ends-in-mixed-content/114387

Other solutions all require https all the time.

Please check to see if the content menus show an https or http link on the bottom of the screen when you hover over them with the "Bug 1" problem. The site would work fine if that did not change when using varnish.

Will try other things when I get some days.

[bwaters]

Yes changing the unsecure base to https and doing https all the time works.

For mixed mode normal http and https operation there is a bug in the code that causes a redirect loop. I am working on resolving that now. Other steps involved are getting an "SSL-OFFLOADED: https" header added when nginx terminates the ssl (Or changing the header in the Magento admin to X_Forwarded_Proto which will handle the more common "X-Forwarded-Proto: https" header)

[joebusby]

@bwaters I switched apache2 to use php7 instead of php5.6.17 and the Bug 1 (category menu showing https) goes away. The site will still not work if the user enters "https" for a page that is not designated secure by magneto2, like the home page when the non-secure baseURL cofig says http://mydomain.com

To support php5, it appears there are two problems and for php7 there is one, although there may be a redirect that would fix the problem of someone using https when accessing http pages. If time permits I will experiment with that.

[bwaters]

@joebusby -- we delivered several fixes to Magento for SSL + Varnish.

1. Changed the vcl file to keep ssl and non ssl caches separate.
2. Changed the default value of the ssl offloaded header to "standard" X-Forwarded-Proto
3. Added correct ssl offload header to varnish vcl file.
4. Removed https calls from esi includes to fix missing content.
5. Made SSL detection consistent across the Magento codebase so that all calls to all request objects work with the configured SSL offload header.

I suggest you get the latest -- or get 2.1 when it is officially released, regenerate your vcl files and try again. Thanks for your help.

[KrystynaKabannyk]

Hello @joebusby, this issue has been fixed in the 2.1.0 Release, that's why I'm closing it. If you any questions or additional information regarding the issue feel free to reopen it or create a new one.