urllib hangs when closing connection

[danielugra]

BPO	16270
Nosy	@orsenthil, @giampaolo, @bitdancer
PRs	gh-81403: Fix for CacheFTPHandler in urllib #13951
Files	urllib.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/giampaolo'
closed_at = <Date 2012-10-19.11:41:53.772>
created_at = <Date 2012-10-17.21:38:49.669>
labels = ['type-bug', 'library']
title = 'urllib hangs when closing connection'
updated_at = <Date 2019-06-10.23:44:24.970>
user = 'https://bugs.python.org/danielugra'

bugs.python.org fields:

activity = <Date 2019-06-10.23:44:24.970>
actor = 'danh'
assignee = 'giampaolo.rodola'
closed = True
closed_date = <Date 2012-10-19.11:41:53.772>
closer = 'giampaolo.rodola'
components = ['Library (Lib)']
creation = <Date 2012-10-17.21:38:49.669>
creator = 'daniel.ugra'
dependencies = []
files = ['27606']
hgrepos = []
issue_num = 16270
keywords = ['patch', '3.3regression']
message_count = 9.0
messages = ['173209', '173212', '173223', '173224', '173323', '173325', '173326', '265227', '286020']
nosy_count = 5.0
nosy_names = ['orsenthil', 'giampaolo.rodola', 'r.david.murray', 'daniel.ugra', 'python-dev']
pr_nums = ['13951']
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16270'
versions = ['Python 3.2', 'Python 3.3', 'Python 3.4']

[danielugra]

With version 3.3 (64 bit on Arch Linux) Python now hangs when closing FTP connection on some sites:

url = "ftp://ftp.fu-berlin.de/pub/misc/movies/database/ratings.list.gz"

with urllib.request.urlopen( url ):
	pass

[giampaolo]

I guess this is about urllib, not ftplib.

[bitdancer]

Well, urllib calls ftplib for ftp urls, so it might be about ftplib. But in that case it would be better to have a reproducer that *just* uses ftplib.

[giampaolo]

I can reproduce the issue on python 3.3.
To have a clearer understanding on what's going on use:

import urllib.request
import ftplib
ftplib.FTP.debugging = 4
url = "ftp://ftp.fu-berlin.de/pub/misc/movies/database/ratings.list.gz"
with urllib.request.urlopen(url):
    pass

The problem appears to be endtransfer() erroneously calling ftp.voidresp() (it just shouldn't):
http://hg.python.org/cpython/file/f6fcff683866/Lib/urllib/request.py#l2328
Patch in attachment fixes the issue.

[python-dev]

New changeset edeeb727df86 by Giampaolo Rodola' in branch 'default':
Fix issue bpo-16270: urllib may hang when used for retrieving files via FTP by using a context manager.
http://hg.python.org/cpython/rev/edeeb727df86

[python-dev]

New changeset 2e6bcbb0ff59 by Giampaolo Rodola' in branch '3.3':
Fix issue bpo-16270: urllib may hang when used for retrieving files via FTP by using a context manager.
http://hg.python.org/cpython/rev/2e6bcbb0ff59

[python-dev]

New changeset 7f0d9637a3ad by Giampaolo Rodola' in branch '3.2':
Fix issue bpo-16270: urllib may hang when used for retrieving files via FTP by using a context manager.
http://hg.python.org/cpython/rev/7f0d9637a3ad

[python-dev]

New changeset 44d02a5d59fb by Senthil Kumaran in branch '2.7':
Closes bpo-26960.
https://hg.python.org/cpython/rev/44d02a5d59fb

[orsenthil]

The original problem here was retrieving files failed (hung) when it was tried via context-manager

1. For e.g after "reverting" the changes, if you use the original file without the context manager, it will succeed.

fobj = urllib.request.urlopen( url )

That indicates the fix in changing something in endtransfer method was improper. I have given more information here: http://bugs.python.org/issue27973#msg286016

The proper fix will be revert the endtransfer changes, but fix it in contextmanager scenario. Add test cases to cover all these scenarios.