[MRG] Buffer CONNECT response bytes from proxy until all HTTP headers are received #2495
Conversation
Codecov Report
@@ Coverage Diff @@
## master #2495 +/- ##
==========================================
- Coverage 83.46% 83.44% -0.02%
==========================================
Files 161 161
Lines 8780 8784 +4
Branches 1288 1289 +1
==========================================
+ Hits 7328 7330 +2
- Misses 1204 1205 +1
- Partials 248 249 +1
Continue to review full report at Codecov.
|
d44ef57
to
a586243
@@ -121,8 +122,16 @@ def processProxyResponse(self, rcvd_bytes): | |||
created, notifies the client that we are ready to send requests. If not | |||
raises a TunnelError. | |||
""" | |||
self._connectBuffer += rcvd_bytes |
kmike
Feb 2, 2017
Member
This could have a different algorithmic complexity in pypy; it seems using bytearray could help. I'm not sure if this is a real problem though.
This could have a different algorithmic complexity in pypy; it seems using bytearray could help. I'm not sure if this is a real problem though.
LGTM. Relevant document: http://www.freeproxy.ru/ru/free_proxy/faq/draft-luotonen-web-proxy-tunneling-01.txt:
so this PR does as specified - waits for an empty line. |
# from the proxy so that we don't send those bytes to the TLS layer | ||
# | ||
# see https://github.com/scrapy/scrapy/issues/2491 | ||
if b'\r\n\r\n' not in self._connectBuffer: |
kmike
Feb 20, 2017
Member
This also can be O(N^2) if connectBuffer is extended byte-by-byte, but I'm not sure how to fix it, and likely it shouldn't be a problem as the response should be small.
This also can be O(N^2) if connectBuffer is extended byte-by-byte, but I'm not sure how to fix it, and likely it shouldn't be a problem as the response should be small.
Fixes #2491
See #2491 (comment) for rationale.
A simple fix, but ideally, we'd want to use a proper HTTP parser.
Something for another PR I believe.