Generic graceful tag expiry (aka not using BAN)

[andrerom]

With work ongoing in #234 (PR #237) to generalize tags, and suggestions in #148 for how FOSHttpCache could in a general way keep track of tags and corresponding urls, this would allow us to support tagging generically across Symfony HttpCache, Nginx and Varnish, by using PURGE calls per url.

Besides supporting all cache proxies, this approach will also allow us to do graceful purging where supported, including Varnish. BAN does not support this, and there is afaik no indication that Varnish will implemented a SoftBan anytime soon.

Why is this important? For high load use cases (high traffic web site, front page), BAN will make Varnish not return the cache object for anything that matches the BAN regex, making it impossible to setup custom VCL rules to allow graceful cache handling on expired cache objects, potentially leading to massive dog-piling effects.

And given FOSHttpCache is the Reverse proxy solution for PHP and Symfony, this imho implies we should aim to cover those high load scenarios, ideally out of the box.

[dbu]

so your idea would be to build a comprehensive tag to url map in the symfony side to be able to purge on varnish instead of ban? i guess that is doable, but should probably not be the default.

• one would need to make sure that the store is really complete (multi-server setups will be tricky here)
• the cost of invalidating could be huge on a busy site, with possibly dozens or hundreds of purge requests clogging the response when done inside a symfony event listener. in the real case where the ban vs purge matters, this must probably go to a message queue which seems out of scope to me for this bundle.
• we would need to also record the ttl (which can be difficult if varnish is doing own things) to avoid purging tons of very-old tags that are not in the cache anymore

i wonder if we can just design our things here with this use case in mind and you build that system in a separate repository?

[andrerom]

one would need to make sure that the store is really complete (multi-server setups will be tricky here)

yes, cache systems like mentioned in #148 would need to be configured using memcached/redis shared across app servers.

the cost of invalidating could be huge on a busy site, with possibly dozens or hundreds of purge requests clogging the response when done inside a symfony event listener. in the real case where the ban vs purge matters, this must probably go to a message queue which seems out of scope to me for this bundle.

message queue does indeed not belong here, but could have the interfaces for async handling here, with purging being done on kernel terminate by default or similar.

i wonder if we can just design our things here with this use case in mind and you build that system in a separate repository?

could indeed be, if we model the tag interfaces in a way that it is possible to have own impl.

[dbu]

sounds good, yes. this could in my opinion be in the FOS organization if you want (we would need to propose it at https://github.com/FriendsOfSymfony/friendsofsymfony.github.com/issues), or at ez.

i am happy to give some input for the library but think mixing it into this library makes it hard to keep focus. and the separate library can then provide integration with message queues and memcached/redis while we just provide a trivial file-based thing to work with Symfony HttpCache.

[andrerom]

Ok, and then out of the box support nginx at later point by using one of the more advance modules then.

Should we then close this for now?