Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide DownloaderMiddleware an interface to read raw HTTP requests and responses #440

odie5533 opened this issue Oct 24, 2013 · 1 comment


Copy link

@odie5533 odie5533 commented Oct 24, 2013

Hello! I would like to request that Scrapy provide an interface to the DownloaderMiddlewares so that they can read (and possibly edit) the raw HTTP request and response data. An example name would be process_raw_request and process_raw_response which would provide the middleware with the raw GET / ... data.

The new HTTP 1.1 parts seem to use a producer pattern, so these could be process_raw_response_chunk/process_raw_response_chunk_decoded methods depending on where in the chain the callback is sent.

Right now I am accessing the data through a very roundabout way, so providing a direct interface through DownloaderMiddleware would be very useful both for debugging and for saving entire dumps.

Copy link

@dangra dangra commented Nov 19, 2013

I'm afraid there aren't plans for an streaming API. Design proposals are welcome.

@cathalgarvey cathalgarvey added the stale label Feb 20, 2018
@kmike kmike removed the stale label Jul 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.