Respect and handle cache headers

[sindresorhus]

Would be very handy if Got had builtin support for response caching. Most APIs you interact with implement either ETag or Last-Modified headers for caching and it's annoying to implement manually, but usually required for efficiency and rate limiting reasons. It could be a Got wrapper, but in reality most requests will need this, so I think it should be in core. We should not be opinionated in how you cache the request however. We could simply say that it supports any module that exposes .set(), .get(), .has() methods. Most will probably want to use a LRU cache for in-memory storage, or Redis higher scale requirements.

Relevant GitHub docs: https://developer.github.com/v3/#rate-limiting

Prior art:

• https://github.com/export-mike/f-etag/blob/master/src/index.js
• https://github.com/gitterHQ/request-http-cache
• https://github.com/kaelzhang/node-modified
• https://github.com/ForbesLindesay/http-basic/blob/febede32150b99874f2c327a19b206b73cdec377/lib/cache-utils.js
• https://github.com/spalger/http-asset
• https://github.com/developit/request-easy-cache (only normal caching)

---

// @floatdrop @alextes @lukechilds @julien-f @reconbot

[lukechilds]

I've had this on my todo list for a while, I've already implemented this on top of Got for lukechilds/onionoo-node-client. I was gonna pull my implementation out and do it as a wrapper but happy to have a bash it implementing it in core. I used an in memory store by default but you can pass options in to use Redis/MongoDB for production.

There's a possibility I'll be quite busy with paid work over the next month so I might not have time for a while. If anyone else wants to start on it then go ahead. You may also find this helpful: lukechilds/expired.

And btw, for the caching, I used node-cache-manager, but I don't think I'd recommend it. The different store engines don't all implement the full API and have inconsistent behaviour so they're not just drop in config options like they appear to be. I had to submit a PR to get Redis working at all. cacheman and jugglingdb look like better options, although I haven't used them.

[lukechilds]

Apologies for messy code in onionoo-node-client btw, I know it's not the neatest. I kinda struggled with the async control flow.

[reconbot]

I love this, yeah a pluggable key value store would make this pretty straightforward and flexible. I wonder if using a leveldown compatible adaptor would be the way to go. Off hand we have memdown and redisdown which are both battle tested and have many friends.

It would be pretty small if people didn't use it and didn't want to bring in a storage adaptor. I'm all for this in core.

[lukechilds]

Yeah LevelUP + LevelDOWN adapter looks good.

Should LevelUP + memdown be a dependency of Got so we can cache OOB? e.g:

const got = require('got');

got('foo.com', { cache: true });

Then { cache } could also be a LevelUP config object to use an adapter other than memory.

Or should we require a LevelUP DB instance to be passed in to Got? e.g:

const got = require('got');
const levelup = require('levelup');
const redisdown = require('redisdown');
const db = levelup('mydb', { db: redisdown, host: 'localhost', port: 6379 });

got('foo.com', {cache: db});

I think the second method is better, it's a bit more verbose but it keeps Got slim and we're also not then tied to LevelUP. Any module that exposes .put(key, val) and .get(key) could be passed in to { cache } like in Sindre's first comment.

Thoughts?

[reconbot]

I'd prefer the second, needs more documentation but keeps got tiny.

…

On Mon, Mar 20, 2017, 11:28 PM Luke Childs ***@***.***> wrote: Yeah LevelUP + LevelDOWN adapter looks good. Should LevelUP + memdown be a dependency of Got so we can cache OOB? e.g: const got = require('got'); got('foo.com', { cache: true }); Then { cache } could also be a LevelUP config object to use an adapter other than memory. Or should we require a LevelUP DB instance to be passed in to Got? e.g: const got = require('got');const levelup = require('levelup');const redisdown = require('redisdown');const db = levelup('mydb', { db: redisdown, host: 'localhost', port: 6379 }); got('foo.com', {cache: db}); I think the second method is better, it's a bit more verbose but it keeps Got slim and we're also not then tied to LevelUP. Any module that exposes .put(key, val) and .get(key) could be passed in to { cache } like in Sindre's first comment. Thoughts? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#281 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABlbuU25ix7YjuQp8ENNvE2BrDmRkyBks5rn0PMgaJpZM4McPJs> .

[alextes]

Second approach as per Sindre's first comment.

I'd much prefer not to have Got concern itself with managing a store. Picking a store means deciding on how to store. Better to leave that up to the user methinks.

(Also nice to see a familiar face Luke, thanks for the zsh plugin 😉)