Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Proposal: Stale while revalidate - prevent multiple generate calls #149
In a situation where a fast response is desireable, and a generate function is slow, multiple unnecessary calls to the generate function are currently made.
Assume there is already a stale value for a key in the cache.
Multiple requests come in.
For the duration of the stale timeout, only a single call to the generate function is made.
However, once the stale timeout expires, and those pending requests are responded to, there is an outstanding call to the generate function in progress, but further calls will lead to another unnecessary call to the generate function.
I'm suggesting a new option, pendingTimeout, which if set, would have to be greater than stateTimeout.
If set, when the staleTimeout is reached, the pending responses would be sent, and the callbacks removed from the array, but the key would remain in the pending list, meaning that additional calls would wait for the same call to the generate function to complete.
When the pendingTimeout was reached, THEN the key would be deleted from the pending list.
This would allow a short staleTimeout (say 200ms) for a long running generate function (say 5s), with only a single call to the generate function, as opposed to the current situation where that setup could generate up to 25 calls to the generate function.
If this seems reasonable, I'm happy to implement and submit a pull request.
This feels complicated and if your staleTimeout is so short, you are problem using the wrong tool. catbox was created for caching of HTTP related activities with normal timeouts in seconds (really minutes and hours). What are you doing that you need such short timeouts?
We are using it for caching http related activities.
However, we're using it (quite successfully generally) to deliver stale-while-revalidate functionality.
So we have an http call that can take around 10 seconds.
So by setting a staleTimeout of 250ms, we get a quick (but stale) response if there is one in the cache, and in the meantime the value is updated in the background once the http request completes.
The only issue is that when under load we generate a lot of unnecessary http calls while the initial long running call is being processed.
Just out of interest, would you usually expect long staleTimeout (as opposed to staleIn) values?
We use long staleIn values (minutes to hours depending on use case), but very short staleTimeout values (time to wait for the generateFunction to execute before returning a stale response).
I'm curious as to the use case/benefit of a long staleTimeout (rather than staleIn value) of minutes or hours.
@primitive-type I've got a change that implements that ready to go here:
I'm just testing it out to make sure it works as expected before submitting a PR.
@hueniverse not sure if you want to wait for a PR, or take a look now and see if you think the approach fits.