Included cacert.pm doesn't contain current pypi.python.org CA certificate

[tazle]

It appears that the cacert.pem bundled in pip 1.5.6 (through requests) doesn't contain current pypi.python.org CA certificate:

tul@karju:~/piptest> py/bin/pip --version
pip 1.5.6 from /u/tul/piptest/py/lib/python2.6/site-packages (python 2.6)
tul@karju:~/piptest> py/bin/pip install pg8000
Downloading/unpacking pg8000
  Cannot fetch index base URL https://pypi.python.org/simple/
  Could not find any downloads that satisfy the requirement pg8000
Cleaning up...
No distributions at all found for pg8000
Storing debug log for failure in /u/tul/.pip/pip.log
tul@karju:~/piptest> py/bin/pip -v install pg8000
Downloading/unpacking pg8000
  Could not fetch URL https://pypi.python.org/simple/pg8000/: connection error: [Errno 1] _ssl.c:491: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
  Will skip URL https://pypi.python.org/simple/pg8000/ when looking for download links for pg8000
  Could not fetch URL https://pypi.python.org/simple/: connection error: [Errno 1] _ssl.c:491: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
  Will skip URL https://pypi.python.org/simple/ when looking for download links for pg8000
  Cannot fetch index base URL https://pypi.python.org/simple/
  Could not fetch URL https://pypi.python.org/simple/pg8000/: connection error: [Errno 1] _ssl.c:491: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
  Will skip URL https://pypi.python.org/simple/pg8000/ when looking for download links for pg8000
  Could not find any downloads that satisfy the requirement pg8000
Cleaning up...
  Removing temporary dir /u/tul/piptest/py/build...
No distributions at all found for pg8000
Exception information:
Traceback (most recent call last):
  File "/u/tul/piptest/py/lib/python2.6/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/u/tul/piptest/py/lib/python2.6/site-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/u/tul/piptest/py/lib/python2.6/site-packages/pip/req.py", line 1177, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/u/tul/piptest/py/lib/python2.6/site-packages/pip/index.py", line 277, in find_requirement
    raise DistributionNotFound('No distributions at all found for %s' % req)
DistributionNotFound: No distributions at all found for pg8000

Storing debug log for failure in /u/tul/.pip/pip.log

Curl by default uses system certificates, and with curl, pypi works:

tul@karju:~/piptest> curl -v 'https://pypi.python.org/simple/pg8000/' > /dev/null
<clip>
* SSL connection using RC4-SHA
* Server certificate:
*    subject: 2.5.4.15=Private Organization; 1.3.6.1.4.1.311.60.2.1.3=US; 1.3.6.1.4.1.311.60.2.1.2=Delaware; serialNumber=3359300; streetAddress=16 Allen Rd; postalCode=03894-4801; C=US; ST=NH; L=Wolfeboro,; O=Python Software Foundation; CN=www.python.org
*    start date: 2014-09-05 00:00:00 GMT
*    expire date: 2016-09-09 12:00:00 GMT
*    subjectAltName: pypi.python.org matched
*    issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA
*    SSL certificate verify ok.
> GET /simple/pg8000/ HTTP/1.1

Curl with pip CA certificates fails:

tul@karju:~/piptest> curl -v --capath /tmp --cacert /u/tul/piptest/py/lib/python2.6/site-packages/pip/_vendor/requests/cacert.pem 'https://pypi.python.org/simple/pg8000/' > /dev/null
<clip>
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS alert, Server hello (2):
} [data not shown]
* SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Closing connection #0

[dstufft]

Hm, this doesn't make any sense. If the cacert.pem didn't include most everyone would be broken. Also PyPI's root certificate is:

DigiCert High Assurance EV Root CA
SHA1: 5fb7ee0633e259dbad0c4c9ae6d38f1a61c7dc25 

and this is in the cacert.pem bundle for 1.5.6: https://github.com/pypa/pip/blob/1.5.6/pip/_vendor/requests/cacert.pem#L1896-L1925

It sounds like something might have gotten corrupted with your pip install maybe?

[tazle]

Indeed the root CA seems to be there. I wonder why curl is failing as well, then, with the pip/requests cacert.pem but not with system certificates. Will investigate.

[tazle]

This might be some kind of PEM file size limit on my OpenSSL library.

[tazle]

Couldn't find root cause. Unable to do a custom build of OpenSSL. OpenSSL appears to load all the CA certificates, but for some reason verification fails when there are over 150 of them.

[s0l4r]

I can verify that the ca-bundle seems to be to big...

python -c "import requests; r = requests.get('https://pypi.python.org/');"
will not work with this cert store:
python2.7/site-packages/certifi:
315159 Nov 26 00:26 cacert.pem
grep "BEGIN CERTI" cacert.pem | wc -l
152

python -c "import requests; r = requests.get('https://pypi.python.org/');"                                                                                                              
Traceback (most recent call last):                                                                                     
 File "", line 1, in                                                                                 
 File "/opt/venv/devpi/lib/python2.7/site-packages/requests/api.py", line 60, in get                          
   return request('get', url, *_kwargs)                                                                               
 File "/opt/venv/devpi/lib/python2.7/site-packages/requests/api.py", line 49, in request                      
   return session.request(method=method, url=url, *_kwargs)                                                           
 File "/opt/venv/devpi/lib/python2.7/site-packages/requests/sessions.py", line 457, in request                
   resp = self.send(prep, *_send_kwargs)                                                                              
 File "/opt/venv/devpi/lib/python2.7/site-packages/requests/sessions.py", line 569, in send                   
   r = adapter.send(request, *_kwargs)                                                                                
 File "/opt/venv/devpi/lib/python2.7/site-packages/requests/adapters.py", line 420, in send                   
   raise SSLError(e, request=request)                                                                                 
requests.exceptions.SSLError: [Errno bad handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

If I remove some entries off the top:
grep "BEGIN CERTI" cacert.pem | wc -l
114
242776 Nov 26 00:35 cacert.pem

python -c "import requests; r = requests.get('https://pypi.python.org/');"
^Works fine.

python -c 'import requests; print(requests.certs.where())'
/opt/venv/devpi/lib/python2.7/site-packages/certifi/cacert.pem

python -c "import ssl; print ssl.OPENSSL_VERSION"
OpenSSL 0.9.8j-fips 07 Jan 2009

I am using SuSE Enterprise Linux SP3, so unfortunately the included OpenSSL is not very up to date :-/

Just to clarify, even with the latest pip installed it does not work.

To fix this I need to create another CA bundle. And then set this before running pip:
export REQUESTS_CA_BUNDLE=~/lib/python2.7/site-packages/certifi/cacert-fixed.pem

also pip --cert cacert-fixed.pem would work (if it's less than 150 I believe).

[tazle]

Indeed my issues were on SLES11SP3 as well. I tried to find the cause of the limit in OpenSSL, but didn't get very far.

[bitliner]

+1
Cannot install anything on AWS (while locally it goes like a charm), because SSL verify fails.
I tried even to fix the time (according to this comment), but still it does not work.
I tried specifying manually the list of cacert.pem, but nothing changes. The file has this header

# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

The error is:

:     raise SSLError(e, request=request)
remote: SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
remote: Collecting Jinja2==2.7.3 (from -r requirements.txt (line 1))
remote:   Getting page https://pypi.python.org/simple/jinja2/
remote:   Starting new HTTPS connection (1): pypi.python.org
remote:   Could not fetch URL https://pypi.python.org/simple/jinja2/: connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
remote:   Will skip URL https://pypi.python.org/simple/jinja2/ when looking for download links for Jinja2==2.7.3 (from -r requirements.txt (line 1))
remote:   URLs to search for versions for Jinja2==2.7.3 (from -r requirements.txt (line 1)):
remote:   * https://pypi.python.org/simple/jinja2/
remote:   Getting page https://pypi.python.org/simple/jinja2/
remote:   Starting new HTTPS connection (2): pypi.python.org
remote:   Could not fetch URL https://pypi.python.org/simple/jinja2/: connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
remote:   Will skip URL https://pypi.python.org/simple/jinja2/ when looking for download links for Jinja2==2.7.3 (from -r requirements.txt (line 1))
remote:   Could not find any downloads that satisfy the requirement Jinja2==2.7.3 (from -r requirements.txt (line 1))
remote:   Cleaning up...
remote:   No distributions at all found for Jinja2==2.7.3 (from -r requirements.txt (line 1))
remote:   Exception information:
remote:   Traceback (most recent call last):
remote:     File "/usr/local/lib/python2.7/site-packages/pip/basecommand.py", line 232, in main
remote:       status = self.run(options, args)
remote:     File "/usr/local/lib/python2.7/site-packages/pip/commands/install.py", line 339, in run
remote:       requirement_set.prepare_files(finder)
remote:     File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 333, in prepare_files
remote:       upgrade=self.upgrade,
remote:     File "/usr/local/lib/python2.7/site-packages/pip/index.py", line 397, in find_requirement
remote:       'No distributions at all found for %s' % req
remote:   DistributionNotFound: No distributions at all found for Jinja2==2.7.3 (from -r requirements.txt (line 1))

[Ivoz]

@bitliner what OS are you running this under?

[bitliner]

debian - jessie.

[bitliner]

Or let's change question: which are the OS where it works normally?

[bitliner]

Furthermore: locally it works, when I run it on Amazon AWS, it stops to work. It could be an issue related to AWS

[cagdascirit]

@bitliner try to install ca-certificates package first and try again.
apt-get update &&
apt-get install ca-certificates

[miguelgr83]

My problem was slightly different. It looks like the https://pypi.python.org/simple/ certificate was renewed in June 2016, so if the date of your OS is prior to that date, you'll have problems validating the certificate, no matter if the proper CA certs are installed.