Multiple Vary headers emitted

[oschaaf]

On trunk-tracking, we somehow end up emitting three Vary: Accept-Encoding headers.

$ curl -I -X GET http://192.168.137.10:8090/mod_pagespeed_example/styles/index_style.css
HTTP/1.1 200 OK
Server: nginx/1.8.0
Content-Type: text/css
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5666b969-2c7"
Accept-Ranges: bytes
Date: Wed, 09 Dec 2015 15:51:53 GMT
Expires: Wed, 09 Dec 2015 15:56:53 GMT
Cache-Control: max-age=300
X-Original-Content-Length: 711
Vary: Accept-Encoding
Vary: Accept-Encoding
Content-Length: 711

nginx.conf:

server {
    listen       8090;
    server_name  localhost;
    pagespeed on;
    pagespeed FileCachePath /tmp/ps;
    pagespeed RewriteLevel PassThrough;
    pagespeed CssPreserveUrls on;
    pagespeed EnableFilters prioritize_critical_css,debug,extend_cache;

    location / {
        root   /var/www/;
        index  index.html index.htm;
    }
}

[deweydb]

Not sure if related, but i was having this issue when i had: gzip_vary on; in my http block, instead of the server block.

[oschaaf]

@jeffkaufman is this worthy of being on the work-prioritization list?

[jeffkaufman]

Added

[HansVanEijsden]

Reminder, please.. would love to have this fixed too (request of one of my clients). 😃

[Niwreg]

Same issue here. I'll follow the thread in case of updates :)

[oschaaf]

Fix for this is in progress at https://github.com/pagespeed/ngx_pagespeed/tree/oschaaf-trunk-tracking-issue-1064, working on tests to prevent regression.
I have yet to reproduce the situation again where we emit three Vary: Accept-Encoding headers which somehow seems hard now, but this fix gets rid of a reproducible double Vary: Accept-Encoding situation.
If anyone is up for trying the patch, here it is: https://github.com/pagespeed/ngx_pagespeed/compare/trunk-tracking...oschaaf-trunk-tracking-issue-1064.patch

[HansVanEijsden]

@oschaaf thanks, I applied the patch successfully (without errors) and recompiled Nginx with it.
Unfortunately, still the issue.
The double Vary headers appear after accessing the URL for the 2nd time.

First time:

$ curl -I https://www.jennifersauer.nl/wp-includes/js/jquery/jquery.js,qver=1.11.4.pagespeed.jm.zixJPNMRNN.js
HTTP/1.1 200 OK
Content-Type: application/javascript
Connection: keep-alive
Keep-Alive: timeout=300
Server: nginx
Content-Length: 95977
Last-Modified: Tue, 05 Jan 2016 19:30:28 GMT
Vary: Accept-Encoding
Access-Control-Allow-Origin: *
Accept-Ranges: bytes
Date: Fri, 29 Jan 2016 15:45:43 GMT
Expires: Fri, 29 Jan 2016 15:50:43 GMT
Cache-Control: max-age=300,private
X-Page-Speed: 1.10.33.2-7600

Second time:

$ curl -I https://www.jennifersauer.nl/wp-includes/js/jquery/jquery.js,qver=1.11.4.pagespeed.jm.zixJPNMRNN.js
HTTP/1.1 200 OK
Content-Type: application/javascript
Connection: keep-alive
Keep-Alive: timeout=300
Server: nginx
Access-Control-Allow-Origin: *
Accept-Ranges: bytes
Date: Fri, 29 Jan 2016 15:45:43 GMT
Expires: Sat, 28 Jan 2017 15:45:43 GMT
Cache-Control: max-age=31536000
ETag: W/"0"
Last-Modified: Fri, 29 Jan 2016 15:45:43 GMT
X-Original-Content-Length: 95889
Vary: Accept-Encoding
Vary: Accept-Encoding
Content-Length: 95889
X-Page-Speed: 1.10.33.2-7600

Just try it yourself, change qver=1.11.4 to qver=1.11.5 or 6 or what you want, it's just a variable. The double Vary headers appear after the 2nd access.

It's only with rewritten sources, normal sources are fine:

$ curl -I "https://www.jennifersauer.nl/wp-includes/js/wp-emoji-release.min.js?ver=4.4.2"
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 28 Jan 2016 23:14:10 GMT
Content-Type: application/javascript
Content-Length: 33713
Last-Modified: Thu, 07 Jan 2016 07:19:03 GMT
Connection: keep-alive
Keep-Alive: timeout=300
Vary: Accept-Encoding
ETag: "568e1167-83b1"
Expires: Thu, 31 Dec 2037 23:55:55 GMT
Cache-Control: max-age=315360000
Access-Control-Allow-Origin: *
Cache-Control: public, must-revalidate, proxy-revalidate
Accept-Ranges: bytes

I've purged all the caches (pagespeed via memcached) and gave the whole VPS a reboot to make sure not battling with the caches, but still no avail. Let me know if you need more information.

[oschaaf]

It took me a little while, but I found the old directory where I could reproduce writing out three Vary:Accept-Encoding response headers.

Applying #1105 there, the number of Vary: Accept-Encoding headers goes down from three to two on the optimized IPRO and .pagespeed. responses for me.

The two remaining Vary: Accept-Encoding headers originate from PSOL in both cases, and it seems this duplication has already been fixed on mod_pagespeed's master. With #1105 applied and build against the current state of mps master I can't reproduce the problem: a single Vary: Accept-Encoding gets emitted every time.

What is surprising is that your website continues to emit three Vary:Accept-Encoding headers with #1105 applied: I would at least have expected one of them to be removed.. Which makes me wonder: how did you clear PageSpeed's caches? The "official" way or by directly purging the memcached instance(s)?

[HansVanEijsden]

@oschaaf yes, strange isn't it? I cleared the cache by deleting the files inside /var/cache/ngx_pagespeed and I rebooted the server (so memcached would be 100% fresh and empty too).
There are no Wordpress cache plugins installed (except the nginx helper plugin). I also cleared the nginx fastcgi cache just to be sure, by clearing /var/run/nginx-cache.

[oschaaf]

@HansVanEijsden: Are you sure you fired up the patched nginx binary? Is it possible that the old (unpatched) nginx server was/is still running? I can't yet come up with a reason that would explain how it could be that the patch has zero effect, but there may be something in your configuration or module list that bars the patch from working correctly (though even if the patched worked, the problem would persist but in reduced form, because ngx_pagespeed would also need to be build against a newer PSOL version to eliminate the problem completely).

If you are sure we're looking at a patched nginx instance, what does nginx -V look like? Could you PM your configuration files?

[HansVanEijsden]

@oschaaf unfortunately.. yes, I'm sure about that.
To be sure, I've done a complete re-install of ngx_pagespeed and nginx.

I installed ngx_pagespeed following the instructions in #1099 and I also applied the cherry-pick.
Then, I applied your patch:

hans@vps:~/release-1.10.33.2-beta$ patch -p1 < trunk-tracking...oschaaf-trunk-tracking-issue-1064.patch
patching file src/ngx_pagespeed.cc
Hunk #1 succeeded at 571 (offset -1 lines).
Hunk #2 succeeded at 1274 (offset -12 lines).
Hunk #3 succeeded at 1845 (offset -12 lines).
Hunk #4 succeeded at 2163 (offset -12 lines).
patching file src/ngx_pagespeed.h
patching file test/nginx_system_test.sh
Hunk #1 succeeded at 1241 with fuzz 2 (offset -7 lines).
patching file test/pagespeed_test.conf.template
Hunk #1 succeeded at 1463 (offset -8 lines).

After re-installing nginx I gave the whole server a reboot, to make really really sure no old binaries to be active.
Here's my nginx -V:

$ /opt/nginx19/sbin/nginx -V
nginx version: nginx/1.9.10
built by gcc 4.9.2 (Debian 4.9.2-10)
built with OpenSSL 1.0.2f 28 Jan 2016
TLS SNI support enabled
configure arguments: --add-module=/home/hans/release-1.10.33.2-beta --prefix=/opt/nginx19 --user=www-data --group=www-data --with-http_ssl_module --with-http_v2_module --with-openssl=/usr/local/src/openssl-1.0.2f --with-openssl-opt='enable-ec_nistp_64_gcc_128 threads' --with-md5=/usr/local/src/openssl-1.0.2f--with-md5-asm --with-sha1=/usr/local/src/openssl-1.0.2f --with-sha1-asm --with-pcre-jit --with-file-aio --with-http_flv_module --with-http_geoip_module --with-http_gzip_static_module --with-http_gunzip_module --with-http_mp4_module --with-http_realip_module --with-http_stub_status_module --with-threads --with-ipv6 --add-module=/usr/local/src/nginx-rtmp-module --add-module=/usr/local/src/ngx_cache_purge-2.3 --add-module=/usr/local/src/ngx_http_substitutions_filter_module --with-cc-opt='-DTCP_FASTOPEN=23 -O3 -march=native'

$ uname -a
Linux vps 4.3.0-0.bpo.1-amd64 #1 SMP Debian 4.3.3-7~bpo8+1 (2016-01-19) x86_64 GNU/Linux

I sent you my config files by mail. 😃

[oschaaf]

Thanks @HansVanEijsden
I PM'd you another patch, let me know if that results in a single Vary: Accept-Encoding header being emitted consistently.
If it does, that means that #1105 is probably all we need for the nps side to resolve this.

[HansVanEijsden]

@oschaaf I own you a beer.
The patch you PM'ed works! 😄 😄