buildbot: test_urllib2_localnet failures (Connection refused) on Tiger buildbot #52701

vstinner · 2010-04-19T11:29:03Z

BPO	8455
Nosy	@db3l, @ronaldoussoren, @mdickinson, @abalkin, @orsenthil, @vstinner, @ned-deily
Files	issue8455.diff issue8455-scproxy.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ronaldoussoren'
closed_at = <Date 2010-06-27.12:52:27.634>
created_at = <Date 2010-04-19.11:29:03.287>
labels = ['tests', 'type-bug', 'library']
title = 'buildbot: test_urllib2_localnet failures (Connection refused) on Tiger buildbot'
updated_at = <Date 2010-08-11.08:18:07.583>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2010-08-11.08:18:07.583>
actor = 'ned.deily'
assignee = 'ronaldoussoren'
closed = True
closed_date = <Date 2010-06-27.12:52:27.634>
closer = 'ronaldoussoren'
components = ['Library (Lib)', 'Tests']
creation = <Date 2010-04-19.11:29:03.287>
creator = 'vstinner'
dependencies = []
files = ['17737', '17740']
hgrepos = []
issue_num = 8455
keywords = ['patch', 'buildbot']
message_count = 15.0
messages = ['103577', '103599', '103600', '103605', '103907', '104784', '104785', '108330', '108331', '108336', '108341', '108347', '108405', '108785', '113593']
nosy_count = 7.0
nosy_names = ['db3l', 'ronaldoussoren', 'mark.dickinson', 'belopolsky', 'orsenthil', 'vstinner', 'ned.deily']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue8455'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2']

vstinner · 2010-04-19T11:29:02Z

http://www.python.org/dev/buildbot/builders/x86 Tiger 3.x/builds/25/steps/test/logs/stdio

test_urllib2_localnet
test test_urllib2_localnet failed -- multiple errors occurred; run in verbose mode for details
Re-running test test_urllib2_localnet in verbose mode
test_proxy_qop_auth_int_works_or_throws_urlerror (test.test_urllib2_localnet.ProxyAuthTests) ... ok
test_proxy_qop_auth_works (test.test_urllib2_localnet.ProxyAuthTests) ... ERROR
test_proxy_with_bad_password_raises_httperror (test.test_urllib2_localnet.ProxyAuthTests) ... ERROR
test_proxy_with_no_password_raises_httperror (test.test_urllib2_localnet.ProxyAuthTests) ... ERROR
test_200 (test.test_urllib2_localnet.TestUrlopen) ... ok
test_200_with_parameters (test.test_urllib2_localnet.TestUrlopen) ... ok
test_404 (test.test_urllib2_localnet.TestUrlopen) ... ok
test_bad_address (test.test_urllib2_localnet.TestUrlopen) ... ok
test_basic (test.test_urllib2_localnet.TestUrlopen) ... ok
test_chunked (test.test_urllib2_localnet.TestUrlopen) ... ok
test_geturl (test.test_urllib2_localnet.TestUrlopen) ... ok
test_info (test.test_urllib2_localnet.TestUrlopen) ... ok
test_redirection (test.test_urllib2_localnet.TestUrlopen) ... ok
test_sending_headers (test.test_urllib2_localnet.TestUrlopen) ... ok

======================================================================
ERROR: test_proxy_qop_auth_works (test.test_urllib2_localnet.ProxyAuthTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1072, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 938, in request
    self._send_request(method, url, body, headers)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 976, in _send_request
    self.endheaders(body)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 934, in endheaders
    self._send_output(message_body)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 788, in _send_output
    self.send(msg)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 729, in send
    self.connect()
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 711, in connect
    self.timeout, self.source_address)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/socket.py", line 321, in create_connection
    raise error(msg)
socket.error: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_urllib2_localnet.py", line 278, in test_proxy_qop_auth_works
    result = self.opener.open(self.URL)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 349, in open
    response = self._open(req, data)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 367, in _open
    '_open', req)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 327, in _call_chain
    result = func(*args)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1090, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1075, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 61] Connection refused>

======================================================================
ERROR: test_proxy_with_bad_password_raises_httperror (test.test_urllib2_localnet.ProxyAuthTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1072, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 938, in request
    self._send_request(method, url, body, headers)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 976, in _send_request
    self.endheaders(body)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 934, in endheaders
    self._send_output(message_body)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 788, in _send_output
    self.send(msg)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 729, in send
    self.connect()
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 711, in connect
    self.timeout, self.source_address)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/socket.py", line 321, in create_connection
    raise error(msg)
socket.error: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_urllib2_localnet.py", line 266, in test_proxy_with_bad_password_raises_httperror
    self.URL)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 456, in assertRaises
    callableObj(*args, **kwargs)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 349, in open
    response = self._open(req, data)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 367, in _open
    '_open', req)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 327, in _call_chain
    result = func(*args)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1090, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1075, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 61] Connection refused>

======================================================================
ERROR: test_proxy_with_no_password_raises_httperror (test.test_urllib2_localnet.ProxyAuthTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1072, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 938, in request
    self._send_request(method, url, body, headers)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 976, in _send_request
    self.endheaders(body)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 934, in endheaders
    self._send_output(message_body)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 788, in _send_output
    self.send(msg)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 729, in send
    self.connect()
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 711, in connect
    self.timeout, self.source_address)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/socket.py", line 321, in create_connection
    raise error(msg)
socket.error: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_urllib2_localnet.py", line 272, in test_proxy_with_no_password_raises_httperror
    self.URL)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 456, in assertRaises
    callableObj(*args, **kwargs)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 349, in open
    response = self._open(req, data)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 367, in _open
    '_open', req)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 327, in _call_chain
    result = func(*args)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1090, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1075, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 61] Connection refused>

Ran 14 tests in 10.076s

ned-deily · 2010-04-19T14:20:26Z

This appears to be a recently introduced failure; the same failure is seen on current trunk build on 10.5 and 10.6 as well.

vstinner · 2010-04-19T14:22:13Z

Maybe r80198 of bpo-7154?

ned-deily · 2010-04-19T14:29:03Z

That wouldn't explain trunk failures. The _scproxy code has been in 26 and trunk for a long time.

db3l · 2010-04-21T20:25:08Z

For what it's worth, I added a few debugging statements to a local trunk checkout, and it looks like the port is getting lost somewhere along the way to setup the proxy... So the refused error is accurate and because it's trying the default port 80. Note also that I think all the Proxy tests fail, even those saying they're ok, because the refused error shows up as a URLError which is what the test expects, but for other reasons.

Oh, and small item I noted while doing the test - the urllib2_localnet module doesn't clean up properly during the tests - all the local servers are left listening on their ports until all the tests are done. Adding a call to self.httpd.server_close() at the end of run() in LoopbackHttpServerThread appears to fix that.

Here's a sample of one test...

test_proxy_qop_auth_int_works_or_throws_urlerror (test.test_urllib2_localnet.ProxyAuthTests) ... Init done
Serving HTTP on localhost port 55161
Thread running
Calling handle request
proxy_url: http://localhost:55161
proxy_open: http None None localhost:55161
httplib _set_hostport(localhost,None)
httplib _set_hostport done host=localhost port=80
httplib.connect: Connecting to ('localhost', 80)
back from handle request

mdickinson · 2010-05-02T11:16:43Z

I'm also seeing this on OS X 10.6. It seems to have started with r80243.

mdickinson · 2010-05-02T11:20:47Z

To clarify, that last message was about trunk, where this test is failing for me since r80243. Adding 2.7 to the versions.

abalkin · 2010-06-22T00:37:12Z

Apparently, the failure on OSX is due to the fact that urllib.proxy_bypass('localhost') returns True. This makes the opener to bypass the ProxyHandler that is explicitly set up to use the correct server port:

    def setUp(self):
        ..
        proxy_url = "http://127.0.0.1:%d" % self.server.port
        handler = urllib2.ProxyHandler({"http" : proxy_url})
        self.proxy_digest_handler = urllib2.ProxyDigestAuthHandler()
        self.opener = urllib2.build_opener(handler, self.proxy_digest_handler)

instead, the opener skips to the default HTTPHandler which attempts to connect on port 80 with sad results.

Interestingly,

>>> urllib.proxy_bypass('127.0.0.1')
False

So the simplest fix is s/localhost/127.0.0.1/ as done in the attached patch (bpo-8455.diff).

abalkin · 2010-06-22T00:55:13Z

In fact, urllib.proxy_bypass() returns True for any simple host name (i.e. any string without '.' in it):

>>> proxy_bypass('xyz')
True
>>> proxy_bypass('')
True
>>> proxy_bypass('whatever')
True

I think the fix I am proposing makes sense regardless of platform because proxy_url is set up numerically:

proxy_url = "http://127.0.0.1:%d" % self.server.port

Maybe the above should be changed to

proxy_url = "%s:%d" % (self.URL, self.server.port)

once URL is changed to numerical form.

orsenthil · 2010-06-22T04:23:10Z

Replacing 'http://localhost' with 'http://127.0.0.1' for this test is fine.

But, urllib.proxy_bypass returning for True for any string, is not correct. In the code, I see that Proxy Settings from Mac OSX system configuration is obtained by calling _get_proxy_settings().
proxy_settings = _get_proxy_settings()

The is the check to to see if proxy_settings['exclude_simple'] is set.
It seems that, it is set under conditions, which either might be a wrong behavior, or we should not rely on this property.

ronaldoussoren · 2010-06-22T08:08:55Z

The mac-related code in urllib is correct, it uses the SystemConfiguration framework on MacOSX to get the proxy settings. That said, the code in _scproxy is not entirly correct, it makes the wrong assumption w.r.t. the value of exclude_simple when the corresponding value is not in the settings retrieved from the SystemConfiguration framework.

The tests seem to make assumptions on how the proxy-detecting code functions, although I don't understand what's going on here (that is, I haven't waded through the test code and urllib to trace the code)

The patch "issue-8455-scproxy.patch" ensures that the tests pass, unless the user changed their proxy settings and selected the checkbox "Exclude simple hostnames" (System Preferernces -> Network -> First network adaptor -> Advanced... -> Proxies)

Please test if the patch solves the issue for you as well.

NOTE: as I mention in the second paragraph the tests seem to make unwarrented assumptions about their environment. Further proof for this:
when you set "http_proxy=localhost:12345" in the environment on a Unixy system the tests will fail as well (although a different subset of the tests).

ronaldoussoren · 2010-06-22T09:33:41Z

I've applied my patch in r82150, which fixes the issue for me in the trunk.

I'll merge the patch into the other branches if this does indeed fix the buildbot issue.

mdickinson · 2010-06-22T18:10:55Z

Just to confirm that with this fix test_urllib2_localnet is now passing for me. Thanks!

ronaldoussoren · 2010-06-27T12:52:27Z

Applied to 2.6 (r82280), r82281 (3.2), r82282 (3.1).

ned-deily · 2010-08-11T08:18:07Z

The problem still exists on OS X 10.3. See bpo-9568 for patches.

vstinner added stdlib Python modules in the Lib dir tests Tests in the Lib/test dir labels Apr 19, 2010

ronaldoussoren self-assigned this May 13, 2010

ronaldoussoren closed this as completed Jun 27, 2010

ronaldoussoren added the type-bug An unexpected behavior, bug, or error label Jun 27, 2010

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buildbot: test_urllib2_localnet failures (Connection refused) on Tiger buildbot #52701

buildbot: test_urllib2_localnet failures (Connection refused) on Tiger buildbot #52701

vstinner commented Apr 19, 2010

vstinner commented Apr 19, 2010

ned-deily commented Apr 19, 2010

vstinner commented Apr 19, 2010

ned-deily commented Apr 19, 2010

db3l commented Apr 21, 2010

mdickinson commented May 2, 2010

mdickinson commented May 2, 2010

abalkin commented Jun 22, 2010

abalkin commented Jun 22, 2010

orsenthil commented Jun 22, 2010

ronaldoussoren commented Jun 22, 2010

ronaldoussoren commented Jun 22, 2010

mdickinson commented Jun 22, 2010

ronaldoussoren commented Jun 27, 2010

ned-deily commented Aug 11, 2010

buildbot: test_urllib2_localnet failures (Connection refused) on Tiger buildbot #52701

buildbot: test_urllib2_localnet failures (Connection refused) on Tiger buildbot #52701

Comments

vstinner commented Apr 19, 2010

vstinner commented Apr 19, 2010

ned-deily commented Apr 19, 2010

vstinner commented Apr 19, 2010

ned-deily commented Apr 19, 2010

db3l commented Apr 21, 2010

mdickinson commented May 2, 2010

mdickinson commented May 2, 2010

abalkin commented Jun 22, 2010

abalkin commented Jun 22, 2010

orsenthil commented Jun 22, 2010

ronaldoussoren commented Jun 22, 2010

ronaldoussoren commented Jun 22, 2010

mdickinson commented Jun 22, 2010

ronaldoussoren commented Jun 27, 2010

ned-deily commented Aug 11, 2010