Provide DownloaderMiddleware an interface to read raw HTTP requests and responses #440

Open
odie5533 opened this Issue Oct 24, 2013 · 1 comment

Comments

Projects
None yet
3 participants
@odie5533

Hello! I would like to request that Scrapy provide an interface to the DownloaderMiddlewares so that they can read (and possibly edit) the raw HTTP request and response data. An example name would be process_raw_request and process_raw_response which would provide the middleware with the raw GET / ... data.

The new HTTP 1.1 parts seem to use a producer pattern, so these could be process_raw_response_chunk/process_raw_response_chunk_decoded methods depending on where in the chain the callback is sent.

Right now I am accessing the data through a very roundabout way, so providing a direct interface through DownloaderMiddleware would be very useful both for debugging and for saving entire dumps.

@dangra

This comment has been minimized.

Show comment
Hide comment
@dangra

dangra Nov 19, 2013

Member

I'm afraid there aren't plans for an streaming API. Design proposals are welcome.

Member

dangra commented Nov 19, 2013

I'm afraid there aren't plans for an streaming API. Design proposals are welcome.

@cathalgarvey cathalgarvey added the stale label Feb 20, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment