You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.
I confirm @wRAR 's statement - headers from CONNECTION_CONTROL_HEADERS (including Content-Length header`) is not copied to headers of response objects.
Solved it by this monkeypatch:
importscrapyfromscrapy.crawlerimportCrawlerProcessfromtwisted.web._newclientimportHTTPParser# HTTPParser.CONNECTION_CONTROL_HEADERS.clear() # <- works (not as expected)# initially I tried to use the same approach as for _caseMappings# mentioned in this comment# https://github.com/scrapy/scrapy/issues/2711#issuecomment-367342284classHTTTParser_H(HTTPParser):
defheaderReceived(self, name, value):
Interesting, many thanks for the research @wRAR and @GeorgeA92 😄
Looks like a design decision rather than a bug, do you know what is the reason behind it?
Should we consider doing this in Scrapy itself? I was able to work around this in 84e91b6, so I have no further need for this specifically.