Multisite installation, default website slow (X-Magento-Vary)

[Flowlance]

Hi. We have a multisite installation. The default website is fast, but the non-default website is really slow. When we switch the default website to the slow one, that ones becomes fast. What I've found out so far is that the slow site always get a cookie named X-Magento-Vary, while the fast one get a PHPSESSID cookie.

Preconditions

Ubuntu 16.04
PHP 7.1.16
Aurora MySQL 5.6.10a (Aws RDS)
Magento 2.2.2
Nginx 1.14.0

Running on multiple AWS instances in OpsWorks with continuous integration (Jenkins).
Media folder is shared on EFS.

Steps to reproduce

The website codes are "base" and "exampleone". In the tests below we have "exampleone" set as the default website.

Nginx configs:

upstream fastcgi_backend {
  server   unix:/run/php/php7.1-fpm.sock;
}

fastcgi_cache_path /var/cache/nginx levels=1:2 keys_zone=magento:100m inactive=60m;
fastcgi_cache_key "$scheme$request_method$host$request_uri";

map $http_host $MAGE_RUN_CODE {
    default base;
    www.example1.com exampleone;
    example1.com exampleone;
}

server {
    listen 80 default_server;
    server_name _;
    set $MAGE_ROOT /srv/www/production/current/;
    set $MAGE_MODE production; # or developer
    index  index.php index.html index.htm;

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    include /srv/www/production/current/nginx.conf;
}

Relevant part of /srv/www/production/current/nginx.conf:

# PHP entry point for main application
location ~ (index|get|static|report|404|503|autocomplete|health_check)\.php$ {
    try_files $uri =404;
    fastcgi_pass   fastcgi_backend;
    fastcgi_buffers 1024 4k;
    fastcgi_param  MAGE_RUN_CODE $MAGE_RUN_CODE;
    fastcgi_param  MAGE_RUN_TYPE $MAGE_RUN_TYPE;
    fastcgi_param https on;
    fastcgi_param  PHP_FLAG  "session.auto_start=off \n suhosin.session.cryptua=off";
    fastcgi_param  PHP_VALUE "memory_limit=1024M \n max_execution_time=60";
    fastcgi_read_timeout 60s;
    fastcgi_connect_timeout 60s;

    fastcgi_param REMOTE_ADDR $http_x_forwarded_for;
    fastcgi_param Host $http_host;
    fastcgi_param X-Forwarded-Proto https;
    fastcgi_param X-Forwarded-Port 443;
    fastcgi_param X-Forwarded-Host $http_host;
    fastcgi_param X-Real-IP $remote_addr;
    fastcgi_param X-Forwarded-For $proxy_add_x_forwarded_for;
    add_header mage-run-code $MAGE_RUN_CODE;
    add_header mage-run-type $MAGE_RUN_TYPE;

    fastcgi_index  index.php;
    fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
    include        fastcgi_params;
}

1. Just by visiting the website, we can clearly tell that one website is much slower than the other one. TTFB is really high (8+ seconds).

2. By CURL'ing the websites, we can see the X-Magento-Vary and PHPSESSID cookies

curl -I https://www.example1.com/

HTTP/2 200
date: Tue, 05 Jun 2018 08:16:59 GMT
content-type: text/html; charset=UTF-8
server: nginx/1.14.0
vary: Accept-Encoding
set-cookie: PHPSESSID=kn4lppbu32dgfrd6ejnit2b307; expires=Wed, 06-Jun-2018 08:16:59 GMT; Max-Age=86400; path=/; domain=www.example1.com; secure; HttpOnly
pragma: no-cache
cache-control: max-age=0, must-revalidate, no-cache, no-store
expires: Mon, 05 Jun 2017 08:09:10 GMT
x-cache-warmer: HIT
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
mage-run-code: exampleone
mage-run-type: website

curl -I https://www.example2.com/

HTTP/2 200
date: Tue, 05 Jun 2018 08:17:16 GMT
content-type: text/html; charset=UTF-8
server: nginx/1.14.0
vary: Accept-Encoding
set-cookie: X-Magento-Vary=4dc58702c807c010b15fff2913b3c86682a31e1e; path=/; secure; HttpOnly
pragma: no-cache
cache-control: max-age=0, must-revalidate, no-cache, no-store
expires: Mon, 05 Jun 2017 08:17:16 GMT
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN

Expected result

Both websites should be fast. I'm not sure what X-Magento-Vary does, but every time the website is slow, this cookie is set. I was thinking it could be related to user specific caches, but we are testing on the index page as a non-logged in user so the same content should be delivered every time.

Actual result

Non-default website is slow.
As a note, magento does set the correct website and everything works, it's just slow.

[oefterdal]

I have the same issue, and if I flip what site should be the default the secondary site is slow. So It's nothing with the website.

[Flowlance]

@oefterdal This is exactly what happens here too. And it's holding back the launch of our second website.

[phoenix128]

Are you using Varnish or the Built-in Full page cache?

[Flowlance]

@phoenix128 The built in FPC. We do this because of the multiple servers / load balancing setup. FPC with Redis can share cache between servers. Having one Varnish per server would cause different caches to show when reloading the page. We can probably set up a separate Varnish server that all the servers communicate with, but it's a bit of a hassle and i doubt it would do anything with the x-magento-vary problem.

As a side note, we already experienced having different caches on every page reload using a shared Redis server. I digged in to the code and found the cache key prefix to be the issue.
The prefix was generated from the path to the web root (/srv/www/production/releases/20180528123754), where the version number changes between every deployment. So unless all the servers finishes deploying on the exact same second, they will have different cache. Even if the root is set to /srv/www/production/current which is a symlink into the current release, it looks like PHP sees the real path with version number. The solution here was to override the id_prefix in env.php. However, I'm not sure if it's related to the issue or not.

'page_cache' => array(
    'id_prefix' => 'PROD_',
    'backend' => 'Cm_Cache_Backend_Redis',
    'backend_options' => array(
        'server' => 'example.com',
        'port' => '6379',
        'database' => '1',
        'compress_data' => '1',
    ),
),

[phoenix128]

Thank you @Flowlance , I am investigating on it trying to reproduce the issue.

[phoenix128]

@Flowlance ,
how did you configure the multiwebsite? Did you configure your NGINX or you created multiple path endpoints?

Actually it is quite weird that you see X-Magento-Vary in your cookies while not running Magento with Varnish. As far as I know it shouldn't be there.

Are you sure your settings are correct and you configured Magento to use the built-in cache from Configuration > System > Full Page Cache ?

I am asking you because I was not able to reproduce it in any way.

[Flowlance]

@phoenix128 thanks for your reply.

I also find it weird that this cookie is being set. See the attached screenshot of our current cache configuration:

We have set specific url's per store. Also note in my nginx config we have this:

map $http_host $MAGE_RUN_CODE {
    default base;
    www.example1.com exampleone;
    example1.com exampleone;
}

Unless it matches one of those two url's, it defaults to the base website.

As a side note, it also happens in our development environment. Is there anything else I can provide you with to help you reproduce the issue?

[phoenix128]

Thank you @Flowlance ,
I think it should be enough. I am still trying to reproduce it locally.

[glwstor]

We have the same issue with Magento 2.2.4 with two store views.
The default store view has no X-Magento-Vary set and the page load is around 52ms.
The non-default store view has X-Magento-Vary set and the page load is around 352ms.
We do not use Varnish, only the build in FPC with Redis.

Tested on Ubuntu18.04
Apache2.4
MySQL 5.6
PHP7.1
Redis

and
Ubuntu 18.04
Nginx
MariaDB
PHP7.1
Redis

Is there any solution?

Best regards

[phoenix128]

Not yet, till now I was unable to reproduce it. I am setting up a new clean environment to give another chance.

[Hansmueller88]

@phoenix128 Hello, we have also the same problem.

1. web browser: Google Chrome; page A: one page of a non-default store view; page B: another page of the same non-default store view

2.1. clear the browsing data of the web browser, open page A, it is slow
2.2. reload the page A, the speed is fast
2.3. open page B, it is slow
2.4. reload the page B, the speed is fast
until now, page A and page B of this non-default store view should be cached

3.1. clear the browsing data of the web browser, open page A, it is slow
3.2. open page B, it is fast

4.1. clear the browsing data of the web browser, open page B, it is slow
4.2. open page A, it is fast

5.1. using PageSpeed Insights (or Pingdom) to test the speed of page A and page B, all these 2 pages get a very bad score

In my opinion, Magento cache system is not returning page from cache if the page is a non-default store view page. But this problem occurs only during the first page open after clearing the browsing data (see point 3.1 - 4.2).

It is not a big problem for the customers, because of only the first visit is slow, if the pages are cached (see point 3.1 - 4.2).

But it is a big problem for SEO(see point 5.1)