IncompleteRead error with urllib2 or urllib.request -- fine with urllib, wget, or curl

[AlexQuinn]

BPO	14044
Nosy	@orsenthil, @pitrou, @vadmium, @serhiy-storchaka, @msornay, @apocalyptech

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-11-26.01:12:52.021>
created_at = <Date 2012-02-17.17:36:16.734>
labels = ['type-bug', 'library']
title = 'IncompleteRead error with urllib2 or urllib.request -- fine with urllib, wget, or curl'
updated_at = <Date 2017-02-12.21:40:30.311>
user = 'https://bugs.python.org/AlexQuinn'

bugs.python.org fields:

activity = <Date 2017-02-12.21:40:30.311>
actor = 'apocalyptech'
assignee = 'none'
closed = True
closed_date = <Date 2015-11-26.01:12:52.021>
closer = 'martin.panter'
components = ['Library (Lib)']
creation = <Date 2012-02-17.17:36:16.734>
creator = 'Alex Quinn'
dependencies = []
files = []
hgrepos = []
issue_num = 14044
keywords = []
message_count = 10.0
messages = ['153581', '171263', '191087', '208169', '210813', '231406', '231434', '255396', '287644', '287653']
nosy_count = 9.0
nosy_names = ['orsenthil', 'pitrou', 'Alex Quinn', 'martin.panter', 'serhiy.storchaka', 'msornay', 'raylu', 'laurento.frittella', 'apocalyptech']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue14044'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4', 'Python 3.6']

[AlexQuinn]

When accessing this URL, both urllib2 (Py2) and urlib.client (Py3) raise an IncompleteRead error.
http://info.kingcounty.gov/health/ehs/foodsafety/inspections/XmlRest.aspx?Zip_Code=98199

Previous discussions about similar errors suggest that this may be due to a problem with the server and chunked data transfer. (See links below.) I can't understand what that means. However, this works fine with urllib (Py2), curl, wget, and all regular web browsers I've tried it with. Thus, I would have expected urllib2 (Py2) and urllib.request (Py3) to cope with it similarly.

Versions I've tested with:

• Fails with urllib2 + Python 2.5.4, 2.6.1, 2.7.2 (Error messages vary.)
• Fails with urllib.request + Python 3.1.2, 3.2.2
• Succeeds with urllib + Python 2.5.4, 2.6.1, 2.7.2
• Succeeds with wget 1.11.1
• Succeeds with curl 7.15.5

---

TEST CASES

# FAILS - Python 2.7, 2.6, 2.5
import urllib2
url = "http://info.kingcounty.gov/health/ehs/foodsafety/inspections/XmlRest.aspx?Zip_Code=98199"
xml_str = urllib2.urlopen(url).read() # Raises httplib.IncompleteRead

# FAILS - Python 3.2, 3.1
import urllib.request
url = "http://info.kingcounty.gov/health/ehs/foodsafety/inspections/XmlRest.aspx?Zip_Code=98199"
xml_str = urllib.request.urlopen(url).read() # Raises http.client.IncompleteRead

# SUCCEEDS - Python 2.7, 2.6, 2.5
import urllib
url = "http://info.kingcounty.gov/health/ehs/foodsafety/inspections/XmlRest.aspx?Zip_Code=98199"
xml_str = urllib.urlopen(url).read()
dom = xml.dom.minidom.parseString(xml_str) # Verify XML is complete
print("urllib:  %d bytes received and parsed successfully"%len(xml_str))

# SUCCEEDS - wget
wget -O- "http://info.kingcounty.gov/health/ehs/foodsafety/inspections/XmlRest.aspx?Zip_Code=98199" | wc

# SUCCEEDS - curl - prints an error, but returns the full data anyway
curl "http://info.kingcounty.gov/health/ehs/foodsafety/inspections/XmlRest.aspx?Zip_Code=98199" | wc

---

RELATED DISCUSSIONS

http://www.gossamer-threads.com/lists/python/python/847985
http://bugs.python.org/issue11463 (closed)
http://bugs.python.org/issue6785 (closed)
http://bugs.python.org/issue6312 (closed)

[pitrou]

The example URL doesn't seem to work anymore. Do you have another example to test with?

[raylu]

The URL works for me.

While wget does download it successfully, I get the following output:

$ wget http://info.kingcounty.gov/health/ehs/foodsafety/inspections/XmlRest.aspx\?Zip_Code\=98199
--2013-06-13 12:15:21--  http://info.kingcounty.gov/health/ehs/foodsafety/inspections/XmlRest.aspx?Zip_Code=98199
Resolving info.kingcounty.gov (info.kingcounty.gov)... 146.129.240.75
Connecting to info.kingcounty.gov (info.kingcounty.gov)|146.129.240.75|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/xml]
Saving to: ‘XmlRest.aspx?Zip_Code=98199’

[      <=>                                               ] 515,315      448KB/s   in 1.1s   

2013-06-13 12:15:23 (448 KB/s) - Read error at byte 515315 (Success).Retrying.

--2013-06-13 12:15:24-- (try: 2) http://info.kingcounty.gov/health/ehs/foodsafety/inspections/XmlRest.aspx?Zip_Code=98199
Connecting to info.kingcounty.gov (info.kingcounty.gov)|146.129.240.75|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/xml]
Saving to: ‘XmlRest.aspx?Zip_Code=98199’

[ <=>                                                    ] 0           --.-K/s   in 0s      

Cannot write to ‘XmlRest.aspx?Zip_Code=98199’ (Success).

Similarly, curl gives

$ curl http://info.kingcounty.gov/health/ehs/foodsafety/inspections/XmlRest.aspx\?Zip_Code\=98199 > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  503k    0  503k    0     0   222k      0 --:--:--  0:00:02 --:--:--  229k
curl: (18) transfer closed with outstanding read data remaining

$ wget --version
GNU Wget 1.14 built on linux-gnu.

$ curl --version
curl 7.30.0 (x86_64-pc-linux-gnu) libcurl/7.30.0 OpenSSL/1.0.1e zlib/1.2.8 libidn/1.25 libssh2/1.4.2 librtmp/2.3

[laurentofrittella]

I had the same problem using urllib2 and the following trick worked for me

import httplib
httplib.HTTPConnection._http_vsn = 10
httplib.HTTPConnection._http_vsn_str = 'HTTP/1.0'

Source: http://stackoverflow.com/a/20645845

[vadmium]

The server in question is sending a chunked response, but seems to be closing the connection when it is done, without sending a zero-length chunk (which I understand it is meant to according to the HTTP protocol).

My Firefox shows the XML without any indication of error. But then if I manually truncate a chunked response to Firefox it doesn’t indicate an error either, which I would probably want to know about.

[vadmium]

I suggest this is the same situation as bpo-6785, and is not a bug in Python. However it might be reasonable to allow forcing a HTTP client connection to version 1.0, which could be used as a workaround.

[laurentofrittella]

Even if forcing the HTTP/1.0 workaround works it can end up in weird issues, especially if used in something more than a small script, like the one I tried to describe in this issue report[1] for the "requests" python library.

[1] https://github.com/kennethreitz/requests/issues/2341

[vadmium]

Closing this as being a bug in the web server, rather than Python.

If someone wants to add a way to force a HTTP 1.0 response, or a way to get all valid data before raising the exception, I suggest opening a new report.

[apocalyptech]

I've just encountered this problem on Python 3.6, on a different URL. The difference being that it's not encountered with EVERY page load, though I'd say it happens with at least half:

import urllib.request
html = urllib.request.urlopen('http://www.basicinstructions.net/').read()
print('Succeeded!')

I realize that the root problem here may be an HTTP server doing something improper, but I've got no way of fixing someone else's webserver. It'd be really nice if there was a reasonable way of handling this in Python itself. As mentioned in the original report, other methods of retreiving this URL work without fail (curl/wget/etc). As it is, the only way for me to be sure of retreiving the entire page contents is by looping until I don't get an IncompleteRead, which is hardly ideal.

[apocalyptech]

Ah, well, actually I suppose I'll rescind that a bit - other pages about this bug around the internet had been claiming that the 'requests' module uses urllib in the backend and was subject to this bug as well, but after experimenting myself, it seems like if that IS the case, they're working around it somehow, because using requests makes this succeed 100% of the time. I probably should've tried that first!

So anyway, there's a reasonable workaround, at least. Sorry for the bugspam!