Re-enable the HTTP caching layer

[GUI]

The HTTP caching layer of api.data.gov is currently disabled. We need to get it re-enabed, but we need to validate some things first.

The caching layer got disabled when we encountered this bug with OPTIONS requests. We've worked around the issue on our end, and we've also got the underlying issue patched in NodeJS. However, while this was disabled, I discovered that some of our unrelated Varnish caching servers have been experiencing periodic failed requests due to this gzip+chunked responses+streaming issue in Varnish. The issue is very sporadic and seems to affect a very small percentage of requests, but it's still concerning, and I'd like to sort it out before re-enabling caching. I'm not 100% sure this same problem is happening on our api.data.gov servers, so that needs to be verified, but I suspect it might be if Varnish gets re-enabled (since we saw this issue on a few other different Varnish setups where streaming was enabled).

So before re-enabling caching:

• Verify our workaround for the failed OPTIONS requests works for the fbopen team.
• Try to come up with a test case to reproduce the sporadic issues seen when streaming is enabled in Varnish and the backend sends back chunked responses that are gzipped.
• Assuming the same Varnish streaming issue is present in our stack, I believe the issue is somewhat inherit with Varnish 3 and streaming enabled with no known workarounds. In that case, we have a few options:
  • Disable streaming in Varnish: I'm pretty sure this fixes the problem, but then streaming is disabled, which I'd like to avoid (since I believe some users are dependent on it).
  • Upgrade to Varnish 4: Varnish 4 makes streaming the default (it was not in Varnish 3), so I'd assume this issue has been fixed in Varnish 4 (but more testing would be needed).
  • Switch the HTTP cache to Apache Traffic Server. I've started using this on another project and have been impressed. I've tested for the same gzip+streaming issues and haven't seen it. Some of Traffic Server's other features like it's clustering and persistent cache are also interesting. But it's doesn't seem quite as configurable as what Varnish allows with VCL, so we'd need to figure out how to handle things like Surrogate-Control headers.

[GUI]

Rolled out as part of #123