Add `enforce_content_length` for responses by nateprewitt · Pull Request #949 · urllib3/urllib3

nateprewitt · 2016-08-17T18:56:19Z

So here's a pass at #723. This is kind of a weird edge case but it particularly prominent in the default configuration of Requests. Most calls performed by urllib3 will raise a IncompleteRead error from httplib when the number of bytes in the body doesn't match the Content-Length.

The Skinny

httplib raises IncompleteReads appropriately everywhere except on incrementally read data. This is the primary way Requests uses urlopen with preload_content=False and then reading with iter_content(). Retrieving data this way hits the flaw in httplib. I've added a flag to enable this functionality, so as not to break stream(amt) and read(amt) calls presently. In the next major release, I would advise the flag being removed to make all read operations uniform by default.

Notes:

My unfamiliarity with the testing harness is definitely showing in test_strict_content_length but the test does prove the changes are working correctly. Tornado won't allow you to send uneven data, so this was the only other solution I could come up with. Any suggestions on alternative methods of simulating this problem would be appreciated.
I implemented length as a property to match the attribute nature of httplib.HTTPResponse.length. I realize an int that we modify may be preferred to a property, but felt it would be more likely to break if we implement int updates everywhere IO might happen in the code.

Lukasa · 2016-08-17T19:48:33Z

Nate: while you're working on a fix for this I'll hold off from reviewing. No reason beyond the fact that I'm trying to manage my workload!

nateprewitt · 2016-08-17T19:52:57Z

Totally! I think I've got the fix worked out, but doing some final testing. No rush on this at all.

nateprewitt · 2016-08-17T22:10:29Z

Keep hitting socket overlap, can someone kick the tests when they get a chance? Thanks!

Lukasa · 2016-08-18T08:29:02Z

Restarted.

nateprewitt · 2016-08-23T16:14:26Z

                 strict=0, preload_content=True, decode_content=True,
-                 original_response=None, pool=None, connection=None, retries=None):
+                 original_response=None, pool=None, connection=None,
+                 strict_content_length=False, retries=None, request_method=None):


Should request_method be replaced with **response_kw so we're not defining minimal contact params?

I don't think so. However, please put strict_content_length after retries.

nateprewitt · 2016-08-23T16:40:44Z

Alright, I think this is ready for a glance whenever you have a spare moment @Lukasa.

This still has some rough edges, so I left a few comments inline, as well as my initial comment in the opening post.

Thanks!

Lukasa · 2016-08-24T19:17:32Z

+
+    def test_length_when_chunked(self):
+        headers = {'content-length': '5',
+                   'transfer-encoding': 'chunked'}


This combination of headers is forbidden by RFC 7230 Section 3.3.2:

A sender MUST NOT send a Content-Length header field in any message that contains a Transfer-Encoding header field.

I added this test because the initial httplib length logic checks to make sure things aren't chunked. I view this test in the same vein as receiving a Content-Length of "foo". It shouldn't happen but do we want to actually return the content length in the event urllib3 happens to receive both? I'd say no, the length should be None because it can't be determined.

Alternatively, we could raise an exception here but I'm not sure if that would buy us anything useful other than aborting the operation.

Yeah, this boils down to "how do we want to deal with this"? The options are as follows:

Raise an exception explaining what went wrong. This fails fast and clearly.

Fall back to no content-length. That means we'll treat the body as chunked. If it's not, we'll fail fast (IncompleteRead, usually). If it is, everything works.

I think that's probably ok in this case, but I'd like a comment explaining the rationale.

Possibly also a log at warning level to explain what we're doing.

Lukasa · 2016-08-24T19:38:17Z

Cool, this is a really good patch so far! I've added some notes here for strictness and other things which I think are fairly important: let me know if you have thoughts!

sigmavirus24 · 2016-08-27T12:11:50Z

Sorry for the confusion Nat. I didn't expand the diff enough and it looked as if you were adding to the DeflateDecoder class. Feel free to ignore that comment.

nateprewitt · 2016-08-27T12:23:51Z

Thanks for the feedback Ian! Things should be updated.

sigmavirus24 · 2016-08-27T12:32:27Z

@nateprewitt it looks like someone updated urllib3 to require your branch to be consistently rebased on top of master (easily one of GitHub's most annoying mistakes/misfeatures as it ties into other things people generally want). Can you rebase this as well please?

nateprewitt · 2016-08-28T00:50:07Z

Ok cool, things are rebased onto current master. All of @sigmavirus24's proposed changes should be in place. Once @haikuginger gives the thumbs up, I'll squash things down to a more manageable commit list and update CHANGES.

haikuginger · 2016-08-28T12:29:33Z

+                        "chunked.")
+            return None
+
+        if length is not None:


This conditional is entirely supplemental to the one above. Possibly do elif length is not None?

haikuginger · 2016-08-28T12:46:36Z

Couple minor nits left.

nateprewitt · 2016-08-28T17:31:33Z

Alright, @Lukasa, @sigmavirus24, @haikuginger, I think everything has been addressed and I squashed the commits down into two separate feature commits. One for the length_remaining attribute and then building on that to implement enforce_content_length.

Lukasa · 2016-08-28T23:56:16Z

Cool, I still like this. @haikuginger @sigmavirus24, are you two happy?

sigmavirus24 · 2016-08-29T00:15:37Z

I've skimmed through (haven't downloaded it and played with it) but it 👀-only looks good.

haikuginger · 2016-08-29T01:22:07Z

I'm on the same page as @sigmavirus24; haven't played with it, but it looks good.

Lukasa · 2016-08-29T01:28:04Z

Ok, cool, I'm happy with this then. Thanks for the great work @nateprewitt!

Lukasa · 2016-08-29T01:28:17Z

And thanks so much for the reviews @sigmavirus24 and @haikuginger, fantastic team job all around.

haikuginger · 2016-08-29T01:32:14Z

Thanks for the excellent work, @nateprewitt! Way to drive this PR through. ✨🍻✨

…ngth Add `enforce_content_length` for responses

nateprewitt force-pushed the 723_strict_content_length branch 5 times, most recently from 6fc439a to a92dfda Compare August 17, 2016 21:55

nateprewitt force-pushed the 723_strict_content_length branch 4 times, most recently from 09addb7 to 1632b96 Compare August 23, 2016 03:31

nateprewitt reviewed Aug 23, 2016
View reviewed changes

nateprewitt force-pushed the 723_strict_content_length branch 2 times, most recently from 7bf3882 to 5c8becc Compare August 23, 2016 16:34

nateprewitt force-pushed the 723_strict_content_length branch 3 times, most recently from 5d087f6 to 2015f77 Compare August 23, 2016 21:38

Lukasa reviewed Aug 24, 2016
View reviewed changes

nateprewitt force-pushed the 723_strict_content_length branch 2 times, most recently from 1b05327 to d442647 Compare August 25, 2016 18:03

nateprewitt force-pushed the 723_strict_content_length branch from 2bc4abe to 5b351aa Compare August 28, 2016 00:43

sigmavirus24 changed the title ~~strict_content_length enforcement~~ Add enforce_content_length for responses Aug 28, 2016

haikuginger reviewed Aug 28, 2016
View reviewed changes

nateprewitt force-pushed the 723_strict_content_length branch 2 times, most recently from 84b6911 to 7c6c226 Compare August 28, 2016 17:06

nateprewitt added 2 commits August 28, 2016 11:25

adding length_remaining functionality to HTTPResponse

e5637bb

enforce_content_length for incrementally read responses

0a2a2dc

nateprewitt force-pushed the 723_strict_content_length branch from 7c6c226 to 0a2a2dc Compare August 28, 2016 17:26

Lukasa merged commit 65b8c52 into urllib3:master Aug 29, 2016

nateprewitt deleted the 723_strict_content_length branch August 29, 2016 04:00

nateprewitt mentioned this pull request Aug 30, 2016

Add a way to check if number of bytes received matches declared transfer length(s) #723

Closed

nateprewitt mentioned this pull request Sep 8, 2016

use enforce_content_length in Requests psf/requests#3563

Merged

debugmiller mentioned this pull request Feb 1, 2019

Content-Length is not checked, resulting in short reads with no error psf/requests#4956

Closed

nateprewitt mentioned this pull request Nov 16, 2020

RFC: What features should be included in v2? #1944

Closed

nateprewitt mentioned this pull request Jan 2, 2022

Enable enforce_content_length by default #2514

Merged

Dobatymo pushed a commit to Dobatymo/urllib3 that referenced this pull request Mar 16, 2022

Merge pull request urllib3#949 from nateprewitt/723_strict_content_le…

8dc27b5

…ngth Add `enforce_content_length` for responses

pquentin mentioned this pull request May 31, 2022

Retry incomplete HTTP downloads elastic/rally#1504

Closed

Uh oh!

Conversation

nateprewitt commented Aug 17, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The Skinny

Uh oh!

Lukasa commented Aug 17, 2016

Uh oh!

nateprewitt commented Aug 17, 2016

Uh oh!

nateprewitt commented Aug 17, 2016

Uh oh!

Lukasa commented Aug 18, 2016

Uh oh!

nateprewitt Aug 23, 2016

Choose a reason for hiding this comment

Uh oh!

Lukasa Aug 24, 2016

Choose a reason for hiding this comment

Uh oh!

nateprewitt commented Aug 23, 2016

Uh oh!

Lukasa Aug 24, 2016

Choose a reason for hiding this comment

Uh oh!

nateprewitt Aug 24, 2016

Choose a reason for hiding this comment

Uh oh!

Lukasa Aug 24, 2016

Choose a reason for hiding this comment

Uh oh!

Lukasa Aug 24, 2016

Choose a reason for hiding this comment

Uh oh!

Lukasa commented Aug 24, 2016

Uh oh!

sigmavirus24 commented Aug 27, 2016

Uh oh!

nateprewitt commented Aug 27, 2016

Uh oh!

sigmavirus24 commented Aug 27, 2016

Uh oh!

nateprewitt commented Aug 28, 2016

Uh oh!

haikuginger Aug 28, 2016

Choose a reason for hiding this comment

Uh oh!

haikuginger commented Aug 28, 2016

Uh oh!

nateprewitt commented Aug 28, 2016

Uh oh!

Lukasa commented Aug 28, 2016

Uh oh!

sigmavirus24 commented Aug 29, 2016

Uh oh!

haikuginger commented Aug 29, 2016

Uh oh!

Lukasa commented Aug 29, 2016

Uh oh!

Lukasa commented Aug 29, 2016

Uh oh!

haikuginger commented Aug 29, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nateprewitt commented Aug 17, 2016 •

edited

Loading