python module certifi==2015.9.6.2 is breaking the AWS CLI - SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

[schhibber-bcs]

We had a number of boxes boot up in the last couple days and upgrade the python module certifi to 2015.9.6.2

After the upgrade we started getting the following error when trying to run any commands against a SSL based endpoint: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

e.g.: aws s3 ls
[Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

The following was our fix:
sudo pip uninstall certifi
sudo pip install certifi==2015.04.28

[rayluo]

On the one hand, I quickly tested this one. I can not reproduce the error.

$ aws s3 ls
2015-08-31 14:20:17 awscli-s3test-m3s9biblah

$ pip install certifi
Installing collected packages: certifi
Successfully installed certifi-2015.9.6.2

$ aws s3 ls
2015-08-31 14:20:17 awscli-s3test-m3s9biblah

I dig deeper. AWS CLI does not directly use certifi, but one of the underlying 3rd party module named requests is aware of certifi. I wrote the following test script and ran it against all known S3 endpoints, after install certifi. Again, they are run well (you will see 404 error because bucket not exists, but it also means the SSL connection does work). Would you mind to run it on your box and see what will happen?

import logging
import requests
logging.basicConfig(level=logging.DEBUG)
bucket_name = 'doesnotmatter_asdfasfasdfasdf'
endpionts = [  # http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
    's3.amazonaws.com',
    's3-external-1.amazonaws.com',
    ]
for endpoint in endpionts:
    print(requests.get('https://%s/%s' % (endpoint, bucket_name)))

On the other hand, the root cause seems to be this relevant issue in certifi. There is not much we can do here from the AWS CLI side. I'll leave this issue open for a while, so that other people encounter same problem can refer to it.

[sigmavirus24]

@rayluo check the version of OpenSSL you have installed and the version of Python. If they're new enough that issue probably doesn't affect you which is why you can't reproduce the issue.

[brianwebb01]

@rayluo I encountered the same issue, and the solution provided by @schhibber-bcs worked for me.

[rayluo]

@brianwebb01 Yes, downgrading certifi is a feasible workaround at this moment.

By the way, as suggested by @sigmavirus24 at 2 posts above, I am sharing my version of OpenSSL and Python, for what its worth. These combination seemingly work fine for me.

$ openssl version
OpenSSL 0.9.8zg 14 July 2015

$ python --version
Python 2.7.10

[brianwebb01]

@rayluo For reference, here are the versions where I experienced the issue and certifi downgrade fixed it:

# openssl version
OpenSSL 1.0.1f 6 Jan 2014

# python --version
Python 2.7.6

[myniva]

I ran into the same issue and the workaround provided by @schhibber-bcs works for me as well.

$ openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013

$ python --version
Python 2.7.5

[jamesls]

I'm just catching up on this issue here. From reading the linked certifi issue, and I correct in saying that there's no actionable thing we can do on the AWS CLI side? If so, I'm inclined to close this issue.

[rayluo]

Summary for future reference:

• This symptom may not happen, if you are using new enough version of OpenSSL and/or Python. (Quoted from @sigmavirus24's earlier comment )
• When it happens, a workaround is to downgrade certifi to previous version (pip install certifi==2015.04.28)

There is not much aws-cli can do here. Agree to close this issue.

[sigmavirus24]

I would say the best option that all of you have is to document a user's options. One option is to pin certifi, another is for them to create a wheel of cryptography that statically links to a recent enough version of OpenSSL and use requests[security]. @Lukasa was working on documentation for this second option.

[rcousens]

Certifi downgrade fixed it for me.

Close this issue.

[ajmath]

Ran into this same problem and spent awhile debugging it. The certifi issue is over here certifi/python-certifi#26

Is there anything that can be done on the awscli side to mitigate this?

[jamesls]

@ajmath Per @sigmavirus24's comment above, the best option we have is to document the options a user has.

I think we can make sure that a plain install of AWS CLI works without user action, but if a user has also installed certifi in their python environment, I would be hesitant to override that behavior because they've presumably installed certifi because they want to use that cert bundle.

[ajmath]

@jamesls Thanks for the response. I was under the assumption that certifi was getting installed by awscli. After looking into it, I realized this is coming from salt-master. Unfortunately I need both tools. I'll continue conversation with the certifi and salt teams.