Missing `vary`header with Symfony proxy client

[thomaskrammer]

We are currently using the Symfony proxy client and from time to time the vary header is missing in the response headers. This occurs very randomly and I can't find the reason for the header disappearing. Could you help us with that issue?

Here's an example response:

age:2085
cache-control:max-age=0, no-cache, public, s-maxage=86400
content-length:77264
content-type:text/html; charset=UTF-8
date:Thu, 09 Nov 2017 10:45:04 GMT
...
status:200
strict-transport-security:max-age=31536000; includeSubDomains

Our config:

fos_http_cache:
    flash_message:
        enabled: true
        secure: true
        name: flashes
        path: /

    user_context:
        hash_cache_ttl: 600
        role_provider: true

    proxy_client:
        symfony:
            http:
                servers:
                    - 127.0.0.1

[dbu]

• could there be anything in your code that also touches headers, and possibly overwrites them?
• is it headers on any responses? or specifically with one context?
• do when the response comes from the cache, if it has no vary does it keep having no vary or is that also randomly flipping on and off?
• the cache-control header looks quite odd - no-cache but s-maxage=86400. as you configured the symfony proxy client, i assume you do not have varnish in front of your application? then the s-maxage might be used by random proxies somewhere (though no-cache should probably have priority, but its odd to have this information)

[dbu]

the relevant code is at

FOSHttpCacheBundle/src/EventListener/UserContextListener.php

Lines 177 to 202 in 7a4d77f

	if ($request->headers->has($this->options['user_hash_header'])) {
	// hash has changed, session has most certainly changed, prevent setting incorrect cache
	if (!is_null($this->hash) && $this->hash !== $request->headers->get($this->options['user_hash_header'])) {
	$response->setClientTtl(0);
	$response->headers->addCacheControlDirective('no-cache');
	return;
	}
	if ($this->options['add_vary_on_hash']
	&& !in_array($this->options['user_hash_header'], $vary)
	) {
	$vary[] = $this->options['user_hash_header'];
	}
	} elseif ($this->options['add_vary_on_hash']) {
	/*
	* Additional precaution: If for some reason we get requests without a user hash, vary
	* on user identifier headers to avoid the caching proxy mixing up caches between
	* users. For the hash lookup request, those Vary headers are already added in
	* onKernelRequest above.
	*/
	foreach ($this->options['user_identifier_headers'] as $header) {
	if (!in_array($header, $vary)) {
	$vary[] = $header;
	}
	}

actually, the odd header might come from this:

FOSHttpCacheBundle/src/EventListener/UserContextListener.php

Line 179 in 7a4d77f

if (!is_null($this->hash) && $this->hash !== $request->headers->get($this->options['user_hash_header'])) {

- this would be after logging in/logging out, when symfony recreated the session. that should not ever end up in the cache, so the vary header is irrelevant. which no-cache actually tells. and that makes me think we should maybe also overwrite max-age and s-maxage in the UserContextListener. would have to check if the url based configuration then would overwrite it again or not.

[thomaskrammer]

Thanks for your quick response. Regarding your questions:

1. We add a few headers to our response like this:

if ($event->isMasterRequest()) {
            $responseHeaders = $response->headers;
            $responseHeaders->set('strict-transport-security', 'max-age=31536000; includeSubDomains');
            $responseHeaders->set('X-XSS-Protection', '1; mode=block');
            $responseHeaders->set('X-Content-Type-Options', 'nosniff');
}

2. This occurs for logged in and logged out users
3. The vary header stays the same when the response comes from the cache
4. We do not use varnish. Here's what we have in our config for those routes:

headers:
       cache_control: { public: true, private: false, max_age: 0, s_maxage: 86400 }

What do you exactly mean by "random proxies"?

[thomaskrammer]

Does that mean the wrong headers maybe written to the cache right after logging in/out occurs? Thanks

[thomaskrammer]

We use render_esi in twig on those pages. Could that be related to this issue?

[dbu]

sorry, i think my original response does not make that much sense.

the "problem" is most likely not really a problem, but what i described in #399 (comment) . that it happens when the hash of the request does not match the hash we would do at the time the response is sent. this means that the roles of the user changed, or somehow the wrong hash is used. and in those cases, the listener tells to not cache at all (no-cache) because if we would cache, we would mix up the response with the wrong request hash.

unless you observed problems, i think you are seeing the expected behaviour here.

with random proxies i meant proxies on the internet which might look at s-maxage. but they should also look at no-cache. still i want to try if we can also set s-maxage to 0 in the listener in that case, to produce more consistent caching instructions.

[thomaskrammer]

I understand that it is the wanted behavior to send the no-cache header, but the problem that we are facing is that the no-cache response gets cached on the server-side. And that leads to mixed up caches in our case. Right?

So we should try to adapt the UserContextListener like this?

$response->setClientTtl(0);
$response->headers->addCacheControlDirective('no-cache');
$response->headers->addCacheControlDirective('s-maxage', 0);

[thomaskrammer]

Another weird thing that we experience are cached responses with a must-revalidate header. If this header is present it doesn't make any sense to also have the age header, does it?

E.g.
Age:247
Cache-Control:must-revalidate, no-cache, private
Connection:Keep-Alive
Content-Encoding:gzip
Content-Length:22914
Content-Type:text/html; charset=UTF-8
Date:Thu, 09 Nov 2017 13:49:03 GMT
Keep-Alive:timeout=5, max=100
strict-transport-security:max-age=31536000; includeSubDomains
vary:X-User-Context-Hash,Accept-Encoding
...

[dbu]

but the problem that we are facing is that the no-cache response gets cached on the server-side. And that leads to mixed up caches in our case. Right?

is that an observation or a question? ;-) looking at the code, it seems that if we set the s-maxage it should lead to the caching rules not applied anymore, so i will add that. looking at Symfony Response::isCacheable, i notice that it only checks for no-store and private, but not no-cache. i will add no-store too in our abort case, for good measure.

the age header is coming from symfony HttpCache and seems to always be set in the lookup method. you could unset it again if you want it gone, but it does not really hurt i think.

btw, s-maxage is problematic in general because it could be used by other proxies in the internet (although in the outside traffic there will be the session cookie so caching should never happen there). there is the reverse_proxy_ttl as well, together with the CustomTtlListener for HttpCache. if you use the user context, you might want to use that instead: http://foshttpcachebundle.readthedocs.io/en/1.3/reference/configuration/headers.html#reverse-proxy-ttl (also available with version 2)

[dbu]

are you on version 1.3 or 2 of this bundle? #400 should fix the issues - would be great if you can try that in your application to be sure. if you are on version 2, the branch won't work, but you could just copy the 2 lines into your code if you can reproduce the problem locally.

[thomaskrammer]

We are currently on version 2.

I did also just stumble upon that line in Response::isCacheable. Pretty sure #400 should solve our issue. I'll try locally although I cannot guarantee that I can reproduce the issue. Thanks!

[thomaskrammer]

Could you also add the fix to version 2? Thanks!

[dbu]

of course, but i will do that by merging 1.3 to master once its merged.

[thomaskrammer]

Just check it out on local and staging system, looks good now!