urllib.request.urlretrieve hangs waiting for connection close after a redirect

[andyharrington]

BPO	8035
Nosy	@amauryfa, @orsenthil, @pitrou, @merwok, @bitdancer
Files	tests.zip: 2.6 file that works and 3.1 file that hangs on urlretrieve in some cases urllib_redirect.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-12-18.16:12:02.226>
created_at = <Date 2010-03-01.19:01:51.985>
labels = ['type-bug', 'library']
title = 'urllib.request.urlretrieve hangs waiting for connection close after a redirect'
updated_at = <Date 2011-12-30.21:30:21.990>
user = 'https://bugs.python.org/andyharrington'

bugs.python.org fields:

activity = <Date 2011-12-30.21:30:21.990>
actor = 'eric.araujo'
assignee = 'none'
closed = True
closed_date = <Date 2011-12-18.16:12:02.226>
closer = 'neologix'
components = ['Library (Lib)']
creation = <Date 2010-03-01.19:01:51.985>
creator = 'andyharrington'
dependencies = []
files = ['16407', '22040']
hgrepos = []
issue_num = 8035
keywords = ['patch', 'needs review']
message_count = 13.0
messages = ['100259', '102351', '102354', '111093', '111108', '111129', '126180', '136386', '149765', '149774', '150016', '150022', '150396']
nosy_count = 8.0
nosy_names = ['amaury.forgeotdarc', 'orsenthil', 'pitrou', 'andyharrington', 'eric.araujo', 'r.david.murray', 'neologix', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue8035'
versions = ['Python 3.2']

[andyharrington]

I give simple 2.6 code test26.py working with urllib.urlretrieve and corresponding 3.1 code test31.py with urllib.request.urlretrieve that hangs (close enough to your category 'crash'?) on Ubuntu 1.9.1 and Windows XP, service pack 2.

Both work on an html page in google sites,
http://sites.google.com/a/luc.edu/comp150/filebin/
but only the 3.1 verson fails when retrieving a file in a Google filebin.
http://sites.google.com/a/luc.edu/comp150/filebin/addition1.py
The file in the filebin downloads simply in the Firefox browser GUI and in the corresponding 2.6 program.

[neologix]

Alright, what happens is the following:

• the file you're trying to retrieve is actually redirected, so the server send a HTTP/1.X 302 Moved Temporarily
• in urllib, when we get a redirection, we call redirect_internal:
  def redirect_internal(self, url, fp, errcode, errmsg, headers, data):
  if 'location' in headers:
  newurl = headers['location']
  elif 'uri' in headers:
  newurl = headers['uri']
  else:
  return
  void = fp.read()
  fp.close()
  # In case the server sent a relative URL, join with original:
  newurl = basejoin(self.type + ":" + url, newurl)
  return self.open(newurl)

the fp.read() is there to wait for the remote end to close connection
The problem, in this case, is that with Python 3.1, httplib uses HTTP/1.1 instead of HTTP/1.0 in version 2.6, and with HTTP/1.1 the server doesn't close the connection after sending the redirect (shown by tcpdump).
So, the process remains stuck on fp.read().
Now, in version 3.1, if we simply change Lib/http/client.py:628
from
class HTTPConnection:

    _http_vsn = 11
    _http_vsn_str = 'HTTP/1.1'

to
class HTTPConnection:

    _http_vsn = 11
    _http_vsn_str = 'HTTP/1.0'

to use HTTP/1.0 instead, the retrieval works fine.

Obviously, this is not a good solution. Since the RFC doesn't seem to require the server to close the connection after sending a redirect, we'd probably better close the connection ourselves.

That's what the attached patch does, it simply removes the call to fp.read() before closing the connection. It also removes this for http_error_default, since if an error occurs, we probably want to close the connection as soon as possible instead of waiting for server to do so.

[bitdancer]

@Andyharrington: No, crash is when the interpreter segfaults.
I'm making it priority high, though, since it is a hang during an operation that is likely to happen fairly frequently. Senthil may want to bump it up even higher.

@neologix: Thanks for figuring this out.

[amauryfa]

This URL does not seem to return a 302 code. Is there another example?

[neologix]

2010/7/21 Amaury Forgeot d'Arc <report@bugs.python.org>

Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:

This URL does not seem to return a 302 code. Is there another example?

There are probably many of them, but a simple way to reproduce it
could be to use netcat on your machine and have it send back:

HTTP/1.1 302 Found
Location: http://www.wikipedia.org/index.php

This should do the trick.

----------
nosy: +amaury.forgeotdarc

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue8035\>

---

[amauryfa]

I would have hoped a Windows way...

[pitrou]

bpo-10577 is a duplicate. See an URL allowing reproducing in msg122831.

[neologix]

Those URLs don't trigger the problem anymore, but AFAICT from the code, this problem is still present in py3k.
Here's an updated patch.

[python-dev]

New changeset 038616802b65 by Charles-François Natali in branch '2.7':
Issue bpo-8035: urllib: Fix a bug where the client could remain stuck after a
http://hg.python.org/cpython/rev/038616802b65

New changeset a420b27a86d9 by Charles-François Natali in branch '3.2':
Issue bpo-8035: urllib: Fix a bug where the client could remain stuck after a
http://hg.python.org/cpython/rev/a420b27a86d9

New changeset 764234983120 by Charles-François Natali in branch 'default':
Issue bpo-8035: urllib: Fix a bug where the client could remain stuck after a
http://hg.python.org/cpython/rev/764234983120

[neologix]

Alright, should be fixed now.