test_urllib2net: issues with ftp://gatekeeper.research.compaq.com/, ftp://ftp.mirror.nl/ and ftp://ftp.kernel.org/

[vstinner]

BPO	23881
Nosy	@vstinner

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-04-07.13:22:37.156>
created_at = <Date 2015-04-07.10:43:02.531>
labels = ['tests']
title = 'test_urllib2net: issues with ftp://gatekeeper.research.compaq.com/, ftp://ftp.mirror.nl/ and ftp://ftp.kernel.org/'
updated_at = <Date 2015-04-07.13:22:37.154>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2015-04-07.13:22:37.154>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2015-04-07.13:22:37.156>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2015-04-07.10:43:02.531>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 23881
keywords = []
message_count = 8.0
messages = ['240198', '240199', '240200', '240201', '240202', '240203', '240204', '240209']
nosy_count = 2.0
nosy_names = ['vstinner', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue23881'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[vstinner]

test_urllib2net uses multiple public FTP servers. Since yesterday, there are issues with 2 FTP servers:

(1) ftp://gatekeeper.research.compaq.com/pub/DEC/SRC/research-reports/00README-Legal-Rules-Regs
=> DNS server error

$ host gatekeeper.research.compaq.com 8.8.8.8
Host gatekeeper.research.compaq.com not found: 2(SERVFAIL)
$ host gatekeeper.research.compaq.com 212.27.40.240
Host gatekeeper.research.compaq.com not found: 2(SERVFAIL)

(2) ftp://ftp.mirror.nl/pub/gnu/

• The FTP connection succeeded
• Changing the directory to /pub/gnu succeeded
• but listing the content of /pub/gnu/ takes forever...

In Firefox, I get the error "425 Failed to establish connection".

[python-dev]

New changeset 69d4e199b88e by Victor Stinner in branch '2.7':
Issue bpo-23881: urllib.ftpwrapper constructor now closes the socket if the FTP
https://hg.python.org/cpython/rev/69d4e199b88e

New changeset 7b168db16e67 by Victor Stinner in branch '3.4':
Issue bpo-23881: urllib.request.ftpwrapper constructor now closes the socket if
https://hg.python.org/cpython/rev/7b168db16e67

New changeset 1a72c0a1a50f by Victor Stinner in branch 'default':
(Merge 3.4) Issue bpo-23881: urllib.request.ftpwrapper constructor now closes the
https://hg.python.org/cpython/rev/1a72c0a1a50f

[python-dev]

New changeset 40ddcd785f7b by Victor Stinner in branch '3.4':
Issue bpo-23881: ftp://gatekeeper.research.compaq.com/ and ftp://ftp.debian.org/
https://hg.python.org/cpython/rev/40ddcd785f7b

New changeset 225be6be3893 by Victor Stinner in branch '2.7':
ssue bpo-23881: ftp://gatekeeper.research.compaq.com/ and ftp://ftp.debian.org/
https://hg.python.org/cpython/rev/225be6be3893

[vstinner]

Python 2.7 used ftp://ftp.kernel.org/pub/linux/kernel/README whereas Python 3.4/3.6 uses ftp://ftp.debian.org/debian/README.

Right now, ftp://ftp.kernel.org/pub/linux/kernel/README doesn't respond. So I also modified Python 2.7 to uses ftp.debian.org as Python 3.4/3.5.

Is there a FTP vulnerability in the wild and some public FTP servers are down because of that? Or is it a network connectivity issue?

[vstinner]

For ftp.kernel.org, it may be related to a FS data corruption bug:

2015-02-03: FTP limited on mirrors.kernel.org
https://www.kernel.org/ftp-limited-on-mirrorskernelorg.html
"We've had to temporarily limit FTP access to mirrors.kernel.org due to high IO load."

[vstinner]

$ host gatekeeper.research.compaq.com 212.27.40.240
Host gatekeeper.research.compaq.com not found: 2(SERVFAIL)

"What happened to gatekeeper.dec.com?"
http://gatekeeper.dec.com/what-happened-to-gatekeeper.html

Maybe we should use ftp://apotheca.hpl.hp.com/ instead?

[vstinner]

(2) ftp://ftp.mirror.nl/pub/gnu/

DNS resolution:

$ host ftp.mirror.nl
ftp.mirror.nl is an alias for download.xs4all.nl.
download.xs4all.nl is an alias for dl.xs4all.nl.
dl.xs4all.nl has address 194.109.21.66
dl.xs4all.nl has IPv6 address 2001:888:0:25:194:109:21:66

http://dl.xs4all.nl/ works well, but this site has no /pub/ directory. Is it related?

See also "Dead mirror: ftp.demon.nl"
https://lists.gnupg.org/pipermail/gnupg-devel/2014-November/029113.html
(but it doesn't look to be related)

xs4all hosted python.org, I don't know if it still host the new CDN of python.org.
https://en.wikipedia.org/wiki/XS4ALL#Corporate_culture
"XS4ALL also sponsors and hosts the sites of many free software projects, like Python, Squirrelmail and Debian."

[vstinner]

test_urllib2net looks to pass on enough buildbots, I close the issue.