Use the Rails cache for everything

[tijmenb]

TLDR: This replaces the homegrown cache with the Rails cache. It turns out that the Cache class doesn't do anything, causing hundreds thousands of unnecessary requests to fetch locales, templates & components.

The story starts when @issyl0 and I were investigating high 404's on origin last week. We noticed that there we're a lot of requests to static for /templates/locales/en-GB, which returns 404:

By sampling a few requests (using the govuk_request_id) we figured out that most of those requests were coming from smart-answers. Moreover, each page request was sometimes responsible for 40 to 120 subsequent API requests:

The root cause of the 404's is that static doesn't have a en-GB locale, which is smart-answers default_locale. This was temporarily fixed in https://github.com/alphagov/static/tree/hack-locales-test.

But this didn't fix the issue, we just saw the 404 being replaced by 200:

Before: lots of 404s

After: lots of 200s

This meant that somehow, slimmer's caching was broken. To investigate further, we added a bunch of statsd instrumentation to slimmer, which sends stats for each cache hit/miss. The branch was deployed in smart-answers to staging. After a while we got this graph from grafana:

It shows that there were actually no cache hits in slimmer, just a lot of misses. We can only conclude that the Cache class (which uses Singleton from the standard library) isn't actually working. We haven't been able to figure out why.

The solution is to replace the homegrown solution with Rails.cache.

After deploying this in smart-answers, we see the following in kibana:

@fields.status:200 AND @fields.application:"static" AND @fields.request: "/templates/locales/en-GB"

Which is exactly what we expect for a cache configured with expires_in: 5.minutes (one request per process).

After deploying the gem branch to smart-answers, collections, multipage-frontend and collections, we see that traffic drops significantly:

My guess is that when deployed to all frontend applications we'll see just a few hundred requests per hour for the locales, govuk_components and templates.

https://trello.com/c/D9HmkJwI

[Davidslv]

I might be wrong but this looks like the cause of the cache not be working, I'm not sure where options[:use_cache] is coming from but it's likely that it got forgotten to be set, so the default (false) will be use (https://github.com/alphagov/slimmer/pull/183/files#diff-3edabcdd528cf7a4390669a093f8f185L12)

[tijmenb]

Yeah, use_cache is definitely false in the smartanswers:

tijmenbrommet@production-calculators-frontend-1:~$ govuk_app_console smartanswers
irb(main):001:0> Slimmer::Cache.instance
=> #<Slimmer::Cache:0x007fb87ee53150 @cache_last_reset=2017-02-22 09:15:09 +0000, @use_cache=false, @cache_ttl=300, @data_store={}>

But I don't understand why since it is looks like its turned on in production.

The same is true for collections:

irb(main):002:0> Slimmer::Cache.instance
=> #<Slimmer::Cache:0x007ff393677e68 @cache_last_reset=2017-02-22 09:21:01 +0000, @use_cache=false, @cache_ttl=300, @data_store={}>

But not government-frontend:

irb(main):004:0> Slimmer::Cache.instance
=> #<Slimmer::Cache:0x007f54a340cde8 @cache_last_reset=2017-02-22 09:21:42 +0000, @use_cache=true, @cache_ttl=300, @data_store={}>

😕

[boffbowsh]

Is the Rails cache always configured? For whitehall frontend there's a memcached instance on every box, but not for "normal" frontend. But we should configure this.

[tijmenb]

Good point. Rails.cache defaults to ActiveSupport::Cache::FileStore, which is maybe fine for this usage? Memcached won't grow indefinitely and will allow the processes to share the cache, so definitely better.

[NickColley]

Nice work! 😍

Would this also speed up local development of our frontend applications? We've had a small investigation into this to try and avoid the constant calls to static.

[tijmenb]

@NickColley yeah, I think that would prevent those calls in dev for most applications.

[fofr]

Awesome PR 💯 🎆
Good work @tijmenb and @issyl0