Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
Provide DownloaderMiddleware an interface to read raw HTTP requests and responses #440
Hello! I would like to request that Scrapy provide an interface to the DownloaderMiddlewares so that they can read (and possibly edit) the raw HTTP request and response data. An example name would be process_raw_request and process_raw_response which would provide the middleware with the raw GET / ... data.
The new HTTP 1.1 parts seem to use a producer pattern, so these could be process_raw_response_chunk/process_raw_response_chunk_decoded methods depending on where in the chain the callback is sent.
Right now I am accessing the data through a very roundabout way, so providing a direct interface through DownloaderMiddleware would be very useful both for debugging and for saving entire dumps.