use enforce_content_length in Requests #3563
Conversation
timeout=timeout | ||
timeout=timeout, | ||
enforce_content_length=True, | ||
request_method=request.method |
Lukasa
Sep 8, 2016
Member
Do we need to explicitly provide this here?
Do we need to explicitly provide this here?
nateprewitt
Sep 8, 2016
•
Author
Member
No, this was an oversight in which class we were calling urlopen
on.
No, this was an oversight in which class we were calling urlopen
on.
decode_content=False | ||
decode_content=False, | ||
enforce_content_length=True, | ||
request_method=request.method |
Lukasa
Sep 8, 2016
Member
Do we need to explicitly provide this here?
Do we need to explicitly provide this here?
nateprewitt
Sep 8, 2016
•
Author
Member
Unfortunately yes, there's specific logic for a HEAD request in length_remaining and it's not accessible from the response object in urllib3. We're passing this information in manually in urllib3 in urlopen
but not from_httplib
.
I'd say this is probably an oversight I made in urllib3/urllib3#949 though, I'd be happy to pass the method along there so we're not having to do this manually from outside of urllib3. I'll open a quick amendment patch in a second if that seems acceptable @Lukasa.
Unfortunately yes, there's specific logic for a HEAD request in length_remaining and it's not accessible from the response object in urllib3. We're passing this information in manually in urllib3 in urlopen
but not from_httplib
.
I'd say this is probably an oversight I made in urllib3/urllib3#949 though, I'd be happy to pass the method along there so we're not having to do this manually from outside of urllib3. I'll open a quick amendment patch in a second if that seems acceptable @Lukasa.
nateprewitt
Sep 8, 2016
Author
Member
Alright, the method is available inside of from_httplib
but only as a private variable _method
on the httplib
object. I don't know if we want to use that inside of urllib3, so this would be the alternative option. from_httplib
is a lower level method than urlopen
, so we should probably be providing the method like this in the response_kw
list, rather than trying to reimplement functionality that's already happening at a higher level.
Alright, the method is available inside of from_httplib
but only as a private variable _method
on the httplib
object. I don't know if we want to use that inside of urllib3, so this would be the alternative option. from_httplib
is a lower level method than urlopen
, so we should probably be providing the method like this in the response_kw
list, rather than trying to reimplement functionality that's already happening at a higher level.
8c7b12a
to
d356e0e
@Lukasa, here's a bit of a hiccup I'd appreciate some input on. Currently all data reads in Requests end up using Currently, regardless of whether or not the content is streamed, we end up initially running it through the We could parse the exception string for 'IncompleteRead', but that seems clunky to me. We could also add a conditional inside to raise |
...since when does |
Oh, nevermind, I see! There's no need for all that, we can just check the response headers for a transfer encoding. =) |
Ok, great! To clarify, are you suggesting a conditional in the Edit: The reason I ask is it would adversely affect |
We must raise a Requests exception. |
Alright, I'll run with that then. Thanks! |
b2e99f8
to
0fcc119
@Lukasa we'll need to merge master into Proposed/3.0.0 to run the tests normally, but with 1.19 in Requests 2.12 this should be ready for a look. |
That merge is done. =) |
Thanks @Lukasa! I'll merge it in now. |
0fcc119
to
84d99f0
Branch updated and tests are running clean. I think this is ready for a review whenever you find a spare moment. |
Ok, I think this looks good! Thanks so much @nateprewitt! |
This is a DO NOT MERGE until we bring urllib3 1.17 into the Requests
proposed/3.0.0
branch, which should likely happen before 3.0.0 gets out the door.This is the final step for the work in urllib3/urllib3#949 and addresses the issues originally raised in urllib3/urllib3#723 and #2833.