Cache support

[rprieto]

Hi,

It would be great if unirest had support for a pluggable cache, just like the browser's local cache.

I see many packages on npm that support specifying a TTL for each request - but I'd much rather take advantage of HTTP and actually read the response headers, and cache the response in memory for the appropriate amount of time (either expire, or max-age - age).

In a sense, this is very similar to a reverse-proxy. Do you think this could be part of unirest? Could it be some sort of plugin, where every request/response traverses the proxy and it decides whether to call the backend or returned cached data?

If the interface is nice & clean, I could imagine several implementation of that proxy, e.g. in memory, backed by Redis... Ideally they would each have smart limits depending on their constrains (memory limit, number of responses, LRU....).

Happy to help implement this if you think it would be useful.

[mashapedeployment]

this issue was actually raised from another company too, they could not use unirest-node bcz of no cache.

[rprieto]

Thanks. Our current stop-gap solution is to call unirest, but wrap every call in a special handler to decide whether to make the call, cache the response, etc. Some built-in support would be amazing.

[nijikokun]

This is not something that is within the scope of the library, this should be done outside of the library (a plugin would be fine, separate repo).

[nijikokun]

Thanks. Our current stop-gap solution is to call unirest, but wrap every call in a special handler to decide whether to make the call, cache the response, etc. Some built-in support would be amazing.

This is the correct way to do it, as it is a per case basis, not everyone will have the same requirements or situation as yours.

Hence why being a plugin would be a better solution for this situation.

[rprieto]

Fair point @nijikokun. About a "plugin", I can imagine two different approaches:

• we have to wrap unirest

This is pretty much what we currently do, but it quite painful to implement if you want caching across the board. If we want an easy drop-in solution, we could create a library that wraps unirest to support typical caching/proxy concerns. On the downside it would have to delegate the entire unirest API. For example, in the end function below, the proxy would check if that request is in cache, call unirest if it's not, and cache the response on the way back before calling the callback.

var proxy = new InMemoryProxy();
proxy.get('http://awesome/api')
         .set('Accept', 'application/json')
         .end(function(res) {});

• pluggable plugin support

This would let unirest take an optional reverse proxy to pass every request/response through. The proxy receives every Request object on the way out, and can optionally cancel them (e.g. if it was returned from cache), and gets every Response object on the way in as well.

unirest.get('http://awesome/api')
          .set('Accept', 'application/json')
          .through(myProxy)
          .end(function(res) {});

[rprieto]

Also minor comment on not everyone will have the same requirements or situation as yours, I wouldn't think these are crazy requirements. We're talking about caching, which is one of the fundamental pillars of the web, and has well defined standards across browsers, reverse proxies, etc... But agreed that a plugin of some sort would make sure people can handle any type of requirements they have 😃

[nijikokun]

Plugins could be hooked like middleware, global plugins, and local plugins,
I like the .through but maybe it could be .plugin

On Wed, Feb 11, 2015 at 4:39 PM, Romain Prieto notifications@github.com
wrote:

Fair point @nijikokun https://github.com/Nijikokun. About a "plugin", I
can imagine two different approaches:

• we have to wrap unirest

This is pretty much what we currently do, but it quite painful to
implement if you want caching across the board. If we want an easy drop-in
solution, we could create a library that wraps unirest to support typical
caching/proxy concerns. On the downside it would have to delegate the
entire unirest API. For example, in the end function below, the proxy
would check if that request is in cache, call unirest if it's not, and
cache the response on the way back before calling the callback.

var proxy = new InMemoryProxy();
proxy.get('http://awesome/api')
.set('Accept', 'application/json')
.end(function(res) {});

• pluggable plugin support

This would let unirest take an optional reverse proxy to pass every
request/response through. The proxy receives every Request object on the
way out, and can optionally cancel them (e.g. if it was returned from
cache), and gets every Response object on the way in as well.

unirest.get('http://awesome/api')
.set('Accept', 'application/json')
.through(myProxy)
.end(function(res) {});

—
Reply to this email directly or view it on GitHub
#47 (comment)
.

[rprieto]

👍 on plugin, which could be as simple as plugin.request(request) that returns true if the plugin handled it, and plugin.response(response) just before calling the end callback.

[nijikokun]

Closing since this was agreed to be a plugin

[rprieto]

Hi @Nijikikun, I'm happy to help write the plugin... is there a plugin interface / system already? I thought that was the current blocker (e.g. no request.get('/foo').plugin(bar).send()). Given plugins are likely to want to inspect/manipulate both requests and responses, something similar to Connect/Koa middleware would be great.

[nijikokun]

I think I will open an issue to address perhaps a plugin specification, that way we can come up with something, I haven't had time to think about as of right now. Working on a new release with all these backlogged items