urllib2 does not catch httplib.BadStatusLine

[adamnelson]

BPO	8823
Nosy	@orsenthil, @vadmium
Superseder	bpo-13736: urllib.request.urlopen leaks exceptions from socket and httplib.client
Files	bad_status_urlerror.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/orsenthil'
closed_at = <Date 2015-11-16.22:09:16.062>
created_at = <Date 2010-05-26.13:44:09.147>
labels = ['type-bug', 'library']
title = 'urllib2 does not catch httplib.BadStatusLine'
updated_at = <Date 2015-11-16.22:09:16.061>
user = 'https://bugs.python.org/adamnelson'

bugs.python.org fields:

activity = <Date 2015-11-16.22:09:16.061>
actor = 'martin.panter'
assignee = 'orsenthil'
closed = True
closed_date = <Date 2015-11-16.22:09:16.062>
closer = 'martin.panter'
components = ['Library (Lib)']
creation = <Date 2010-05-26.13:44:09.147>
creator = 'adamnelson'
dependencies = []
files = ['26298']
hgrepos = []
issue_num = 8823
keywords = ['patch']
message_count = 6.0
messages = ['106526', '106539', '164851', '165255', '240471', '254760']
nosy_count = 4.0
nosy_names = ['orsenthil', 'adamnelson', 'martin.panter', 'groodt']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = None
status = 'closed'
superseder = '13736'
type = 'behavior'
url = 'https://bugs.python.org/issue8823'
versions = ['Python 2.6']

[adamnelson]

When running urllib2 and getting a BadStatus from an http server, this error is raised:

File "/var/www/pinax-env/pline/apps/page/models.py", line 303, in render
content = urllib2.urlopen(self.url,timeout=10).read()
File "/usr/lib/python2.6/urllib2.py", line 124, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.6/urllib2.py", line 389, in open
response = self._open(req, data)
File "/usr/lib/python2.6/urllib2.py", line 407, in _open
'_open', req)
File "/usr/lib/python2.6/urllib2.py", line 367, in _call_chain
result = func(*args)
File "/usr/lib/python2.6/urllib2.py", line 1146, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.6/urllib2.py", line 1119, in do_open
r = h.getresponse()
File "/usr/lib/python2.6/httplib.py", line 974, in getresponse
response.begin()
File "/usr/lib/python2.6/httplib.py", line 391, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.6/httplib.py", line 355, in _read_status
raise BadStatusLine(line)
httplib.BadStatusLine

Because urllib2 doesn't catch it with this:

lines 1116-1120

try:
h.request(req.get_method(), req.get_selector(), req.data, headers)
r = h.getresponse()
except socket.error, err: # XXX what error?
raise URLError(err)

It is not caught anywhere else and the call blows up unless I make a special exception for all httplib exceptions. The specific url that blew this up is:

http://phoenix.untd.com/oasx/rqst/type=sx/rdb=8203014d740000555355533a415a2d2d2d2d2d2d2d2d2d2d0100001d494a0901000000000000/version=3/origin=uol/isp=et_cau

[orsenthil]

urllib2 is currently catching the socket.error exceptions and presenting the reason as custom URLError exception. To address this issue, the module should catch the httplib raised exceptions also and present it.

[groodt]

I've made a small change to urllib2 to catch the httplib.BadStatusLine and raise as URLError. This exception should rarely happen as it means the server is returning invalid responses. Nevertheless, I've added a test and hopefully fixed the issue.

Patch is attached. I will check if this needs to be added to newer versions of Python.

[groodt]

By the way, the issue can be recreated by running the following:

netcat -l -p 9999 -e "echo HTTP/1.1 1000 OK" &
python -c "import urllib2; urllib2.urlopen('http://localhost:9999')"

This happens on 2.6, 2.7 and 3 by the looks of it.

[vadmium]

See also bpo-22797, about documenting that non-URLError exceptions may be raised.

[vadmium]

bpo-13736 also proposes to wrap HTTP client exceptions, although I personally don’t really endorse this.

The other option is to fix the documentation: bpo-25633, bpo-22797.