Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[idea] HttpCacheMiddleware could be further enhanced #2744

starrify opened this issue May 17, 2017 · 2 comments


None yet
4 participants
Copy link

commented May 17, 2017

The current design of HttpCacheMiddleware:

  • Checks whether a request hits the cache in process_request
  • Stores a response to the cache storage in process_response

That makes it possible to download a same resource multiple times within a same job, given that a following request to the same resource comes before the first one returns.

It's possible to enhance this middleware by storing the state and returning a deferred when there is a pending request of the same resource.


This comment has been minimized.

Copy link

commented Feb 16, 2018

This could make a really nice performance gain on some kinds of site, that have very dense webs of internal links. Do you think this would be a good candidate for the "good first issue" label? I'm not familiar enough with those APIs yet to say. :)


This comment has been minimized.

Copy link

commented Mar 12, 2018

Hey, can you guys elaborate more on this issue :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.