httplib.HTTPResponse.read could potentially leave the socket opened forever

[eranrund]

BPO	16298
Nosy	@orsenthil, @pitrou, @asvetlov, @slingamn
Files	httplib-no-content-length-close-sock-fix.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-12-15.18:30:17.609>
created_at = <Date 2012-10-22.11:32:32.357>
labels = ['type-bug', 'library']
title = 'httplib.HTTPResponse.read could potentially leave the socket opened forever'
updated_at = <Date 2012-12-15.18:56:56.117>
user = 'https://bugs.python.org/eranrund'

bugs.python.org fields:

activity = <Date 2012-12-15.18:56:56.117>
actor = 'eranrund'
assignee = 'none'
closed = True
closed_date = <Date 2012-12-15.18:30:17.609>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2012-10-22.11:32:32.357>
creator = 'eranrund'
dependencies = []
files = ['27913']
hgrepos = []
issue_num = 16298
keywords = ['patch']
message_count = 10.0
messages = ['173505', '174037', '174613', '174807', '175039', '177539', '177540', '177545', '177546', '177548']
nosy_count = 7.0
nosy_names = ['orsenthil', 'pitrou', 'asvetlov', 'python-dev', 'piotr.dobrogost', 'eranrund', 'slingamn']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16298'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.4']

[eranrund]

When calling HTTPResponse.read() on a response that is:
a. not chunked
b. contains no content-length header
the underlying socket (referenced by self.fp) will never get closed (through self.close())

The offending code is at the bottom of the read() function:
s = self.fp.read(amt)
if self.length is not None:
self.length -= len(s)
if not self.length:
self.close()
return s
As seen, if self.length is None, even when the server closes the connection (causing self.fp.read to return ''), the socket will not get closed.

btw, this may be the cause of bpo-15633 (http://bugs.python.org/issue15633)

[asvetlov]

Please publish patch for the issue.

[eranrund]

The patch is probably trivial - however I would still like some verification.
Would it be correct to call self.close() when fp.read returns ''? In case self.length is not present, I don't see a way around this anyway. When it is present, and fp.read returns '', how should we go about that? We can either return less data, or raise an exception to indicate that the connection terminated prematurely.

Thanks

[pitrou]

The patch is probably trivial - however I would still like some
verification.
Would it be correct to call self.close() when fp.read returns ''? In
case self.length is not present, I don't see a way around this anyway.
When it is present, and fp.read returns '', how should we go about
that? We can either return less data, or raise an exception to
indicate that the connection terminated prematurely.

It's probably better to return less data. No need to break user programs
when they download from a slightly misbehaved Web site.

The patch should include some kind of unit test, if possible. See
Lib/test/test_httplib.py.

[eranrund]

Hello

I have attached a patch that includes a (slightly broken) fix and a test case. Note that there is currently an unresolved issue:
If the user reads the exact amount of bytes the server sent, read() on the socket will never have a chance to return '' and inform the user about the connection termination. This will also happen if the user reads more data than the server is sending. In that case, again, read() will not get called again so we will never know the socket is gone. One possible way to address this is perform the actual read() calls in a loop, attempting to read the exact amount the user specified. If we do this and the user attempts to read more bytes than the server sent, we will properly detect it. If, however, the user reads the exact length, then we still have a problem.

I am not sure what would be the correct way of solving this.

[pitrou]

Sorry for the delay. I am not sure I understand your concern here:

If the user reads the exact amount of bytes the server sent, read() on > the socket will never have a chance to return '' and inform the user
about the connection termination.

Certainly read(), called once again, would return ''? That's how I understand your unit test anyway.

[eranrund]

Hm, it's been a while and I'm no longer sure :(
You're right - since there is no length the user will have to call read() until he gets back ''. It's possible I forgot that assumption when speculating about this.

[python-dev]

New changeset 2186f7b99c28 by Antoine Pitrou in branch '2.7':
Issue bpo-16298: In HTTPResponse.read(), close the socket when there is no Content-Length and the incoming stream is finished.
http://hg.python.org/cpython/rev/2186f7b99c28

New changeset b47d342c449b by Antoine Pitrou in branch '3.2':
Issue bpo-16298: In HTTPResponse.read(), close the socket when there is no Content-Length and the incoming stream is finished.
http://hg.python.org/cpython/rev/b47d342c449b

New changeset 59358f991c00 by Antoine Pitrou in branch '3.3':
Issue bpo-16298: In HTTPResponse.read(), close the socket when there is no Content-Length and the incoming stream is finished.
http://hg.python.org/cpython/rev/59358f991c00

New changeset 5d6c2c7bc5d4 by Antoine Pitrou in branch 'default':
Issue bpo-16298: In HTTPResponse.read(), close the socket when there is no Content-Length and the incoming stream is finished.
http://hg.python.org/cpython/rev/5d6c2c7bc5d4

[pitrou]

Ok, the patch looked fine to me, so I've committed it to 2.7 and adapted it for 3.x. Thank you!
By the way, it's probably easier to produce patches using Mercurial rather than using manual diff. You can have a look at the devguide for more information: http://docs.python.org/devguide/

[eranrund]

My pleasure.
I had no idea about the Mercurial patch, this is the first time I have submitted a Python bug report :)
I'll have a look.

Thanks for merging the fix!