testPythonOrg() of test_robotparser fails on validating python.org HTTPS certificate

[vstinner]

BPO	28151
Nosy	@vstinner, @tiran, @berkerpeksag, @zware
Files	issue28151.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-09-18.11:08:46.058>
created_at = <Date 2016-09-14.13:34:34.372>
labels = ['3.7', 'type-feature', 'tests']
title = 'testPythonOrg() of test_robotparser fails on validating python.org HTTPS certificate'
updated_at = <Date 2016-09-18.20:51:28.285>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2016-09-18.20:51:28.285>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2016-09-18.11:08:46.058>
closer = 'berker.peksag'
components = ['Tests']
creation = <Date 2016-09-14.13:34:34.372>
creator = 'vstinner'
dependencies = []
files = ['44695']
hgrepos = []
issue_num = 28151
keywords = ['patch', 'buildbot']
message_count = 7.0
messages = ['276442', '276443', '276445', '276725', '276874', '276880', '276909']
nosy_count = 5.0
nosy_names = ['vstinner', 'christian.heimes', 'python-dev', 'berker.peksag', 'zach.ware']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue28151'
versions = ['Python 3.6', 'Python 3.7']

[vstinner]

The "x86 Windows7 3.x" buildbot fails to validate python.org HTTPS certificate.

The unit test uses the clear text HTTP URL, but http://www.python.org/robots.txt returns "301 Moved Permanently" with the new link: https://www.python.org/robots.txt

Firefox sees the cert as valid on my Fedora 24.

Firefox tells me that the cert organization (O) is "Python Software Foundation" and it was emitted by (CN) "DigiCert SHA2 Extended Validation Server CA", (O) "DigiCert Inc", (OU) "www.digicert.com".

Does this buildbot lack DigitCert root CA?

http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/11636/steps/test/logs/stdio

======================================================================
ERROR: testPythonOrg (test.test_robotparser.NetworkTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\urllib\request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\http\client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\http\client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\http\client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\http\client.py", line 1026, in _send_output
    self.send(msg)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\http\client.py", line 964, in send
    self.connect()
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\http\client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\ssl.py", line 401, in wrap_socket
    _context=self, _session=session)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\ssl.py", line 808, in __init__
    self.do_handshake()
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:747)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_robotparser.py", line 280, in testPythonOrg
    parser.read()
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\urllib\robotparser.py", line 58, in read
    f = urllib.request.urlopen(self.url)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\urllib\request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\urllib\request.py", line 532, in open
    response = meth(req, response)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\urllib\request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\urllib\request.py", line 564, in error
    result = self._call_chain(*args)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\urllib\request.py", line 756, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\urllib\request.py", line 526, in open
    response = self._open(req, data)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\urllib\request.py", line 544, in _open
    '_open', req)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\urllib\request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\urllib\request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:747)>

[vstinner]

A workaround is to catch the ssl.SSLError and skip the unit test...

Another fix is to avoid completely network connection to the Internet and only uses the temporary HTTP server used by other unit tests.

[berkerpeksag]

Note that I'm planning to rewrite the test to use pythontest.net.

[berkerpeksag]

Here's a patch that uses pythontest.net.

[python-dev]

New changeset 83bca958adc9 by Berker Peksag in branch '3.6':
Issue bpo-28151: Use pythontest.net in test_robotparser
https://hg.python.org/cpython/rev/83bca958adc9

New changeset 4bfd91a45c81 by Berker Peksag in branch 'default':
Issue bpo-28151: Merge from 3.6
https://hg.python.org/cpython/rev/4bfd91a45c81

[berkerpeksag]

test_robotparser is now green. I've opened bpo-28195 for test_huntrleaks_fd_leak.

[vstinner]

Thank you Berker! I'm quite sure that the issue is very old, like 1 year
old or more. And this fix is welcomed!