Ban subscriber for symfony HttpCache

[dbu]

See discussion at FriendsOfSymfony/FOSHttpCacheBundle#167

ezpublish also has an issue on this one: https://jira.ez.no/browse/EZP-22401

The idea is to use Stash, as the process would occur very early, I can propose that one would customize Stash driver by overriding a method, like we did in eZ to store the user context hash before FOSHttpCache

[lolautruche]

I'll try to take care of it soonish 😃

[dantleech]

@lolautruche any more news here? we need this quite soon -- I could make a PR based on the method you proposed, how does that sound?

[dbu]

would be very cool to see this happen.

[dantleech]

Have discussed with @lolautruche and we have decided to go with an "eager" approach, i.e. actively purging the URLs upon BAN requests. We will store a list of URLs against each tag using Stash.

We also considered the oposite, passive, approach, i.e. checking if a URL is banned against the incoming request and refereshing the cache, an approach which I detailed here.

Finally I think that eager purging is more performant, @dbu @ddeboer have you got any preferences between the eager and passive approaches?

[dbu]

I would say go ahead. call it EagerTagSubscriber or something like that to keep room for other implementation. to answer the question which is better, one would need to implement both and performance test them. then the outcome would most likely be "it depends". with not-so-frequent updates, eager deletion should be ok. with a very busy website, varnish is the better choice anyways. are you aware that BAN means invalidating the cache on any headers matched with regular expressions? and that the client will be sending a regular expression for the tag invalidation, not a list of tags to invalidate? this makes me wonder actually, if we should have a TagInterface along with BanInterface, and allow proxy clients to implement that and provide a method to invalidate tags. then the proxy client can decide how to execute tag invalidation. for varnish, it can generate a BAN with regular expression, but for HttpCache it could generate a PURGETAGS method or something and just send a comma-separated list of tags that you can use. for the full BAN in HttpCache, the best bet probably is leveraging the filesystem by placing all responses headers in files and calling `grep` with the regular expressions. as long as that section fits into RAM, the access should be pretty fast as well.

[dantleech]

Eager regex ban

FIrst, would using grep be a solution for an eager ban? I benchmarked finding cache entries and I would say it is at least 10x faster than the PHP solution, but to remove bans we would have to call something like sed I think. This would be platform specific I guess.

I have done some benchmarks for an eager ban using regex and native PHP. This method writes all the bans to a banlist.txt file as follows:

1426583165
node-936a2da6-14f6-4883-84d8-98b9fa9df72b,node-936a2da6-14f6-4883-84d8-98b9fa9df72b,node-936a2da6-14f6-4883-84d8-98b9fa9df72b,node-936a2da6-14f6-4883-84d8-98b9fa9df72b
cache-id-5507ee7df3b17

So entires are composed of three lines, the first is the timestamp, the second the ban and the third the ID of the cache file to purge.

331 entries, 0.0017211437225342s
662 entries, 0.0030980110168457s
1,324 entries, 0.0054359436035156s
2,648 entries, 0.012006998062134s
10,592 entries, 0.042278051376343s
338,944 entries, 1.5312321186066s

The benchmarks include removing the banned entires from the list. They don't include purging the cache, but I don't imagine that would add any additional noticable overhead.

So, not an enterprise solution, but adequate for small to medium websites - and because it is eager it is not a frontend penalty.

Eager tag ban

The tag ban would be much more scaleable as we would simply be looking up indexed tags from the cache. The value of the cache entry contains the purge IDs. It would be very fast.

The difference is that we are not using regex, and so, as @dbu says, it is not compatible with the existing BAN logic. We would have to implement a new method and interface. Which might make perfect sense.

Question

• Are the any use cases for the regex ban not covered by the tag ban here? i.e. is it worth creating it?
• Shall we extend the library to include the concept of tag bans?

[lolautruche]

The difference is that we are not using regex, and so, as @dbu says, it is not compatible with the existing BAN logic. We would have to implement a new method and interface. Which might make perfect sense.

This is an implementation detail in the Store. We have to take the regexp to comply what's done in the lib already. And it's not a big deal AFAICS.

[dantleech]

@lolautruche but how do we efficiently find the cached pages based on the regex? If we are using a key value store, how does that fit in with the concept of regex?

[lolautruche]

The regexp would be only needed to decode the BAN request, and retrieve the tags we want to purge. Then we look into the dictionary with the tags and purge corresponding cache files.

[dantleech]

So we would parse the regex? f.e. explode('|', $headers['X-Ban']) ?

[lolautruche]

That's the idea yes 😄 . This is what we currently do in eZ LocationAwareStore

[dantleech]

Also, as @dbu just pointed out on IRC, with grep we would not need to maintain any list. So that would be a pretty good strategy supporting full regex.