Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aiohttp ignoring SSL_CERT_DIR and or SSL_CERT_FILE environment vars. Results in [SSL: CERTIFICATE_VERIFY_FAILED] #3180

Closed
creslinux opened this issue Aug 8, 2018 · 51 comments
Labels
invalid This doesn't seem right outdated question StackOverflow

Comments

@creslinux
Copy link

creslinux commented Aug 8, 2018

Long story short

The CA file is working with cURL, Python Requests, but not aiohttp, when using SSL_CERT_DIR and or SSL_CERT_FILE environment variables.

Our environment uses its own CA root used to decode/encode HTTPS API requests/responses to provide a short lived cache to prevent excessing external requests.

The environment has the following set:

$ (set -o posix; set) | egrep 'SSL|_CA'
CURL_CA_BUNDLE=/home/creslin/poo/freqcache/cert/ca.pem
REQUESTS_CA_BUNDLE=/home/creslin/poo/freqcache/cert/ca.pem
SSL_CERT_DIR=/home/creslin/poo/freqcache/cert/
SSL_CERT_FILE=/home/creslin/poo/freqcache/cert/ca.pem

The ca.pem can be successfully used by cURL - with both a positive and negative test shown:

curl --cacert /home/creslin/poo/freqcache/cert/ca.pem https://api.binance.com/api/v1/time
{"serverTime":1533719563552}

curl --cacert /home/creslin/NODIRHERE/ca.pem https://api.binance.com/api/v1/time
curl: (77) error setting certificate verify locations:
  CAfile: /home/creslin/NODIRHERE/ca.pem
  CApath: /etc/ssl/certs

A simple python requests script req.py also works as expected, positive and negative tests

cat req.py 
import requests
req=requests.get('https://api.binance.com/api/v1/time', verify=True)
print(req.content)
CURL_CA_BUNDLE=/home/creslin/poo/freqcache/cert/ca.pem
REQUESTS_CA_BUNDLE=/home/creslin/poo/freqcache/cert/ca.pem
SSL_CERT_DIR=/home/creslin/poo/freqcache/cert/
SSL_CERT_FILE=/home/creslin/poo/freqcache/cert/ca.pem

python3 req.py 
b'{"serverTime":1533720141278}'
CURL_CA_BUNDLE=/
REQUESTS_CA_BUNDLE=/
SSL_CERT_DIR=/
SSL_CERT_FILE=/

python3 req.py 
Traceback (most recent call last):
  File "/home/creslin/freqt .......

..... ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)

Using aysnc/aiohttp [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833) is returned always.
The environment settings pointing to the ca.pem shown to work for both cURL and requests are seemingly ignored

CURL_CA_BUNDLE=/home/creslin/poo/freqcache/cert/ca.pem
REQUESTS_CA_BUNDLE=/home/creslin/poo/freqcache/cert/ca.pem
SSL_CERT_DIR=/home/creslin/poo/freqcache/cert/
SSL_CERT_FILE=/home/creslin/poo/freqcache/cert/ca.pem

I have the test script a.py as

cat a.py 
import aiohttp
import ssl
import asyncio
import requests

print("\n requests.certs.where", requests.certs.where())
print("\n ssl version", ssl.OPENSSL_VERSION)
print("\n ssl Paths", ssl.get_default_verify_paths() ,"\n")
f = open('/home/creslin/poo/freqcache/cert/ca.crt', 'r') # check perms are ok
f.close()

async def main():
    session = aiohttp.ClientSession()
    async with session.get('https://api.binance.com/api/v1/time') as response:
        print(await response.text())
    await session.close()

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Which will always produce the failure - output in full:

 requests.certs.where /home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/certifi/cacert.pem

 ssl version OpenSSL 1.1.0g  2 Nov 2017

 ssl Paths DefaultVerifyPaths(cafile=None, capath='/home/creslin/poo/freqcache/cert/', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/lib/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/lib/ssl/certs') 

Traceback (most recent call last):
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/connector.py", line 822, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 804, in create_connection
    sock, protocol_factory, ssl, server_hostname)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 830, in _create_connection_transport
    yield from waiter
  File "/usr/lib/python3.6/asyncio/sslproto.py", line 505, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/lib/python3.6/asyncio/sslproto.py", line 201, in feed_ssldata
    self._sslobj.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "a.py", line 20, in <module>
    loop.run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "a.py", line 14, in main
    async with session.get('https://api.binance.com/api/v1/time') as response:
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/client.py", line 843, in __aenter__
    self._resp = await self._coro
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/client.py", line 366, in _request
    timeout=timeout
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/connector.py", line 445, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/connector.py", line 757, in _create_connection
    req, traces, timeout)
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/connector.py", line 879, in _create_direct_connection
    raise last_exc
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/connector.py", line 862, in _create_direct_connection
    req=req, client_error=client_error)
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/connector.py", line 827, in _wrap_create_connection
    raise ClientConnectorSSLError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host api.binance.com:443 ssl:None [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)]
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fa6bf9de898>

Expected behaviour

aiohttp does not reject server certificate.

Actual behaviour

SSL verification error

Steps to reproduce

use own CA root certificate to trust HTTPS server.

Your environment

Ubuntu 18.04
Python 3.6.5
ssl version OpenSSL 1.1.0g 2 Nov 2017
Name: aiohttp Version: 3.3.2
Name: requests Version: 2.19.1
Name: certifi Version: 2018.4.16

@creslinux creslinux changed the title Own root ca.pem - ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833) aiohttp ignoring SSL_CERT_DIR and or SSL_CERT_FILE environment vars. Results in [SSL: CERTIFICATE_VERIFY_FAILED] Aug 8, 2018
@webknjaz
Copy link
Member

webknjaz commented Aug 8, 2018

There's this PR #2735 hanging since February. Do you mind picking it up? It'd allow you to have a custom ssl.SSLContext, where you could configure trust as you wish.

@creslinux
Copy link
Author

Hi @webknjaz

Is 2735 related to aiohttp not using SSL_CERT_DIR and or SSL_CERT_FILE environment variables to set a trusted root ca locaton?

The PR does not read this way, i may *easily be mistaken

@asvetlov
Copy link
Member

asvetlov commented Aug 8, 2018

Looks like @webknjaz misread the mentioned PR.

aiohttp has no specific code for SSL certificates loading.
It is out of the scope of the library.

Please read the standard documentation to figure out how to create SSLContext with custom certs.
Using environment variables for this is an error-prone and leaking way, please learn how to create proper SSL context by Python API calls.

@asvetlov asvetlov closed this as completed Aug 8, 2018
@asvetlov asvetlov added the invalid This doesn't seem right label Aug 8, 2018
@creslinux
Copy link
Author

I dont think its as simple as that - there is a catch 22 here.

requests and aiohttp are commonly used by other libraries - changing those are out of scope for many users and use-cases

To explain further
requests does not use the system key store, they can be pointed at CA file via ENV to work around this.
This then also updates where SSL is looking, printing out ssl.get_default_verify_paths() shows this.

aiohttp does use the system key store, via SSL.
But if an application has requests calls this has changed where SSL looks for ca certs.
and seemingly aiohttp does not follow where SSL is looking

We're left where an application can make no obvious use of requests and aiohttp and a companies own root_ca certificate.

@asvetlov
Copy link
Member

asvetlov commented Aug 8, 2018

aiohttp follows Python decisions which use system store.
certifi approach has own benefits and pitfalls.
I prefer to follow Python.
If Python core devs will decide to switch to a custom certs store or respect environment variables -- aiohttp will do it too. This discussion raised in the Python dev team several times and proposals were constantly declined for security reasons.

If you want requests-like configuration by default -- you can assemble a thin wrapper on top of aiohttp to do it. Personally I dont like this approach

@creslinux
Copy link
Author

OK - thanks for the response.

@webknjaz
Copy link
Member

webknjaz commented Aug 9, 2018

@asvetlov AFAIR aiohttp does not have any hooks where the end-user could supply their own SSLContext objects and that PR adds one. Am I right?

@creslinux
Copy link
Author

creslinux commented Aug 9, 2018

system certs are a mess. an unmanageable real mess if we're honest once scratch the surface of it.

OSX and Windows you have half a chance, even then its problematic as certs are lazy loaded via s-channel so may or may not be there.

Linux is beyond a lost cause - system certs can be in any 1 of 10 locations reported, finding one those isn't enough to assume its "THE" one.
Debian and thereafter ubuntu (so thats 90% of users in my linux world) certs are ridiculously out of date, unmanaged which moots entirely the point of having it - if you're trusting root CAs even the root ca authority are trying to revoke for years.....
Then anaconda may have trodden all over openssl anyway etc

Pip tries to use system certs, but falls back to mozillas certifi bundle because system certs are anything anywhere

None of that is criticism of aiohttp or python.
Being able to simply and absolutely point at a store, cognitive that aiohttp/asynio are now often a couple libraries down from the library or app users are building upon would be tremendously useful.

There is no portability in it without - code on one os is not going to behave the same on another on a lift and shift.

@creslinux
Copy link
Author

Google couldn't make using system certs work on linux for Chrome, the average python scripter - myself in this bracket - has no hope.

@webknjaz
Copy link
Member

webknjaz commented Aug 9, 2018

@creslinux recently I've been trying to automate (with ansible) adding a coprorate certificate to my laptop and found out that neither Chrome nor Firefox can read system folder, only user's one: turns out that underlying nsslib only promises to read /etc, but in fact does not do that.

@creslinux
Copy link
Author

it all seems a bit religious.
Whats wrong with options such as pip install aiohttp[certifi] or pip install requests[system] and let the users / deployment profile choose.

@asvetlov
Copy link
Member

asvetlov commented Aug 9, 2018

Help yourself.
Say again, for using certifi you don't need aiohttp source code modification, just make a wrapper and use it happily.

@webknjaz
Copy link
Member

webknjaz commented Aug 9, 2018

What about SSLContext? Only monkey-patching?

@creslinux
Copy link
Author

creslinux commented Aug 9, 2018 via email

@asvetlov
Copy link
Member

asvetlov commented Aug 9, 2018

@webknjaz await client.get(url, ssl=...) supports SSLContext argument, no need for monkey patching

@webknjaz
Copy link
Member

webknjaz commented Aug 9, 2018

Okay, that's what I was asking about :)

@asvetlov
Copy link
Member

asvetlov commented Aug 9, 2018

@creslinux your arguments are applicable to any TSL connection, not only HTTP.
I have a feeling that the problem should be fixed on OS or Python level.

Would you try to convince python core devs to ship Python with custom root certs?
I recall @tiran had objections for this approach.
Honestly, I don't remember the whole pros/cons list.
The security is very sensitive matter, I don't want to make the decision fast.

@creslinux
Copy link
Author

creslinux commented Aug 9, 2018 via email

@webknjaz
Copy link
Member

webknjaz commented Aug 9, 2018

@creslinux while there are various projects supporting such feature, it's out of the scope of aiohttp to do so.

Moreover, as @asvetlov outlined, there's a security risk related to this feature. Given that aiohttp is and underlying building block for apps it would be unwise to have a library-specific env var for hijacking root CA. I can imagine that user having two apps in the same env wanting to add a trusted CA to one of them would unintentionally (probably lacking some knowledge) influence the other one as well, creating an extra "invisible" security hole. Also, we would like to avoid creation of the false sense of protection among users.

On the other hand, most of the software built on top of aiohttp will probably want their own env vars (unique names etc.), which would we non-conflicting and won't abuse global settings.

Based on the above I strongly encourage you to implement this on the level of your app.

@webknjaz webknjaz added the question StackOverflow label Aug 9, 2018
@creslinux
Copy link
Author

creslinux commented Aug 9, 2018

its not risk when the OS its build on uses them....
its aiohttp saying we'll use this part of the OS but not the other

Its true requests, twisted, aiohttp, ssl - are pillars that many apps are built upon.
Only 1 in that list does not support the vars.

Linux has no trusted key-store, its misnomer to point out it should be used.
Tell me my trusted keystore in linux ill point to 20 linux distributions that say different and the major ones that use the VARs to manage that problem.

@creslinux
Copy link
Author

creslinux commented Aug 9, 2018

So in summary

  • requests - wrong
  • pip - wrong
  • ssl - wrong
  • twisted - wrong
  • openssl - wrong
  • redhatd - wrong
  • ubuntu - wrong
  • curl - wrong
  • wget - wrong
  • aiohttp - correct

@creslinux
Copy link
Author

The thing is the Python SSL module does support the Env Vars - aiohttp simply ignores SSLs when its using them.

Thats the part i dont understand - why use SSL if going to ignore that its instructing to do.

https://access.redhat.com/articles/2039753#modifying-python-programs-to-control-certificate-verification-9

screen shot 2018-08-09 at 7 36 01 pm

@MyNameIsCosmo
Copy link

Correct me if I'm wrong, but aiohttp uses SSL Context and does not directly handle ssl.wrap_context()
So global variables shouldn't apply to aiohttp

@creslinux
Copy link
Author

creslinux commented Aug 9, 2018

python SSL sees the path to be used.
aiohttp *seemingly ignores it and only uses only 1 of the paths returned.

It is deciding which parts of the OS to honour and is unusual in this as other apps behave as the OS would intend.

SSL_CERT_DIR = /tmp
export SSL_CERT_DIR
import ssl
print("\n ssl Paths", ssl.get_default_verify_paths() ,"\n")

returns:

ssl Paths DefaultVerifyPaths(cafile=None, capath='/tmp', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/lib/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/lib/ssl/certs')

@MyNameIsCosmo
Copy link

MyNameIsCosmo commented Aug 9, 2018

https://docs.python.org/3/library/ssl.html#libressl-support

"SSLContext.set_default_verify_paths() ignores the env vars SSL_CERT_FILE and SSL_CERT_PATH although get_default_verify_paths() still reports them."

I think it makes sense to pass the cert from get_default_verify_paths to your ssl context.

If you think aiohttp should handle this, make a pull request with your changes for review.

Edit:
I just realized the link above is to LibreSSL support.
I'm unsure if the ssl module supports the global path variables.

@webknjaz
Copy link
Member

webknjaz commented Aug 9, 2018

@MyNameIsCosmo true

@creslinux (1) it depends on the SSL implementation (particularly, python doc emphasizes that LibreSSL ignores env vars) and (2) methods used from there (see more docs).

@webknjaz
Copy link
Member

webknjaz commented Aug 9, 2018

@creslinux as for your "wrong" accusations, it's not about being "right" or "incorrect". It's a design decision and keeping the responsibility for breaches on the user's side.
Following your suggestions, I could say that you're wrong trying to supersede maintainers' design decisions: we need to stick to certain design patterns to prevent the project from becoming a mess just because everyone wants their bells and whistles there. If you step back and take a look at the ecosystem here, you'll see that the currently encouraged way to go is to create a third-party extension to aiohttp.

@creslinux
Copy link
Author

creslinux commented Aug 9, 2018

Thats a very tight cut and past - the full section reads

ssl.get_default_verify_paths()
Returns a named tuple with paths to OpenSSL’s default cafile and capath. The paths are the same as used by SSLContext.set_default_verify_paths(). The return value is a named tuple DefaultVerifyPaths:

cafile - resolved path to cafile or None if the file doesn’t exist,
capath - resolved path to capath or None if the directory doesn’t exist,
openssl_cafile_env - OpenSSL’s environment key that points to a cafile,
openssl_cafile - hard coded path to a cafile,
openssl_capath_env - OpenSSL’s environment key that points to a capath,
openssl_capath - hard coded path to a capath directory
Availability: LibreSSL ignores the environment vars openssl_cafile_env and openssl_capath_env

Thats a different argument libressl not supporting the full spec and not passing a path back so cant used is hardly a great argument for when it is passed back by the default build

Python certificate handling explicitly states openssl_capath_env and openssl_cafile_env are used by openssl. Thats 90% linux distributions right there.
https://docs.python.org/3/library/ssl.html#certificate-handling

@creslinux
Copy link
Author

creslinux commented Aug 9, 2018

aiohttp is undermining how an OS, openssl, python ssl, and other popular pillars of python libraries manage company CAs.

There's no getting away from that. To ignore where the OS says to take certs from and claim its for security when only encourages people to disable verification doesn't really add up.

As a simple observation, guys n gals in deployment, users deploying etc - really are not in the main going to re-code and and pick apart an application that ignores their OS, they're just going to disable verification.

@creslinux
Copy link
Author

creslinux commented Aug 9, 2018

As a final comment, what this policy means in Linux is:

  • Only a root user may verify the server connecting to is who they claim to be.
  • As only the root user may modify /etc

In windows and OSX user have their own keychains/ceritficate stores.
In Linux they have ENV vars.

@tiran
Copy link

tiran commented Aug 10, 2018

The env vars SSL_CERT_FILE and SSL_CERT_PATH are handled by OpenSSL, not by Python. They override the location of the default CA cert file and CA cert directory for https://docs.python.org/3/library/ssl.html#ssl.SSLContext.set_default_verify_paths . aiohttp has to use SSLContext.load_default_certs, SSLContext.set_default_verify_paths, or ssl.create_default_context in order to load default CA certs from either the default location or SSL_CERT_FILE location.

  • LibreSSL ignores the vars.
  • The files must be readable by the current user. OpenSSL doesn't report any errors if the file is missing or not readable.
  • The env var must be set before the context is created.
  • Some runtime environments unset all env vars.

@creslinux
Copy link
Author

creslinux commented Aug 10, 2018

LibreSSL ignores the vars.

If the var its not passed it wont be used. Same as any var

The major distributions are on OpenSSL by default. So unless an admin has explicitly moved away from default SSL this seems pretty moot. Debian by inference is Ubuntu/Mint etc.

Name Release System OpenSSL ver TLS 1.1 and 1.2 Supported Notes
Debian 10* 1.1.0 Yes Buster (dev)
Debian 9 1.1.0 + 1.0.1 Yes Stretch
Debian 8 1.0.1 Yes Jessie
Debian 7 1.0.1 Yes Wheezy
RedHat/CentOS 8* 1.1.0 Yes (dev)
RedHat/CentOS 7 1.0.2 Yes
RedHat/CentOS 6 1.0.1 Yes

The files must be readable by the current user. OpenSSL doesn't report any errors if the file is missing or not readable.

True for all the paths returned by OpenSSL, including the host certificate store -- if a user is lucky enough their sys admin root user can be bothered to add a certificate, or accepts 1 users company CA should be added globally for ALL users on the host.

That would be a questionable security decision to force unto a multi user host.

Mr Foo from Company Foo MUST trust Company Bar's root CA as user Mrs Bar wants to trust her company Bar's CA? -- ???

The env var must be set before the context is created.

The var must be set to be read? well...

Some runtime environments unset all env vars.

The var must be set to be read? we....

And the flip side it

  • Aiohttp is out of step *most other popular python libraries in this area, inc ssl, requests, twisted etc
  • Only root users may have the privilege of knowing the server they are connecting to is that server
  • For 1 user to trust a CA means ALL users must trust a CA
  • aiohttp knows better and its a good policy to pick-and-choose the ca-paths the OS asks to be used
  • its fine to take the stance all users should pick apart and recode applications that for a puritan pursuit

Im sorry. This simple encourages to not verify servers. Its counter productive in the extreme. To take the stance you should either install a root CA for 1 user globally, must be root user, or must recode the application.

Im shocked - truly SHOCKED Google did not go down this path with Chrome on Linux

@asvetlov
Copy link
Member

asvetlov commented Aug 10, 2018

aiohttp calls create_default_context(): https://github.com/aio-libs/aiohttp/blob/master/aiohttp/connector.py#L761-L772
@tiran does it mean that SSL_CERT_FILE and SSL_CERT_DIR are processed?

@creslinux
Copy link
Author

SSL_CERT_FILE and SSL_CERT_DIR not SSL_CERT_PATH are the ENV vars

@asvetlov
Copy link
Member

Sorry, was my typo.

@creslinux
Copy link
Author

exporting SSL_CERT_DIR =/tmp
results in print("ssl.get_default_verify_paths())
returning the verify paths to include a set capath

DefaultVerifyPaths(
cafile=None, 
capath='/tmp', 
openssl_cafile_env='SSL_CERT_FILE',
openssl_cafile='/usr/lib/ssl/cert.pem', 
openssl_capath_env='SSL_CERT_DIR', 
openssl_capath='/usr/lib/ssl/certs')

similar for SSL_CERT_FILE / cafile

the verify openssl_* paths are being honored but not capath / cafile verify paths.

@tiran
Copy link

tiran commented Aug 10, 2018

aiohttp calls create_default_context(): /aiohttp/connector.py@master#L761-L772
@tiran does it mean that SSL_CERT_FILE and SSL_CERT_DIR are processed?

Yes, ssl.create_default_context() calls SSLContext.set_default_verify_paths, which calls SSL_CTX_set_default_verify_paths internally. https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_set_default_verify_paths.html will load the certificates from file or directory of hashed passwords. aiohttp does everything correctly.

@asvetlov
Copy link
Member

Cool! Thank you!

@creslinux
Copy link
Author

creslinux commented Aug 10, 2018

The link is good, aiohttp does not behave as it describes, it failing to verify servers when a hashed cert is in capath.

Can this be raised as a bug then?

The capath below is not null its shown to be capath=/tmp/certs/ printed from the python ssl module, and is a dir thats contains a hashed certificate that is shown in the same test to work when added to openssl_capath store.

from the link provided by @tiran
https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_set_default_verify_paths.html

screen shot 2018-08-10 at 11 47 37 am

Tested below.

Test script:

cat a.py 
import aiohttp
import asyncio

async def main():
    session = aiohttp.ClientSession()
    async with session.get('https://api.binance.com/api/v1/time') as response:
        print(await response.text())
    await session.close()

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Run test script with no cert in SSL_CERT_DIR or in store (fails as expected)

creslin@creslab:~$ python3 a.py
Traceback (most recent call last):
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 822, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 804, in create_connection
    sock, protocol_factory, ssl, server_hostname)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 830, in _create_connection_transport
    yield from waiter
  File "/usr/lib/python3.6/asyncio/sslproto.py", line 505, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/lib/python3.6/asyncio/sslproto.py", line 201, in feed_ssldata
    self._sslobj.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "a.py", line 12, in <module>
    loop.run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "a.py", line 6, in main
    async with session.get('https://api.binance.com/api/v1/time') as response:
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/client.py", line 843, in __aenter__
    self._resp = await self._coro
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/client.py", line 366, in _request
    timeout=timeout
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 445, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 757, in _create_connection
    req, traces, timeout)
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 879, in _create_direct_connection
    raise last_exc
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 862, in _create_direct_connection
    req=req, client_error=client_error)
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 827, in _wrap_create_connection
    raise ClientConnectorSSLError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host api.binance.com:443 ssl:None [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)]
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f48d70e8ef0>

set up SSL_CERT_DIR and add a hashed certificate

creslin@creslab:~$ mkdir /tmp/certs
creslin@creslab:~$ cp ca.crt /tmp/certs
creslin@creslab:~$ openssl x509 -hash -in /tmp/certs/ca.crt 
9426e611
-----BEGIN CERTIFICATE-----
MIICEzCCAXygAwIBAgIJAKBgdWawyyZnMA0GCSqGSIb3DQEBCwUAMA0xCzAJBgNV
BAMMAmZ0MB4XDTE4MDcyNzEwMDkyN1oXDTI4MDcyNDEwMDkyN1owDTELMAkGA1UE
AwwCZnQwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAK+lfec0yUzTHvyHkcKk
C/xZyZtb80D1pxa1/UpT3LpzAopEqtHSuBosx7yhWBMjcvUCk1fkk9CCyfrdd1Q1
784rba+AMR4XbX0GHShpYchmkBHaoAk9zRMcgHrAhLux/Mp3rR8oOO4JJI0g0OHn
1s674hbS2cmNnfFeC7jy3rUjAgMBAAGjezB5MB0GA1UdDgQWBBT6JFAIzTCwTQem
Gv/ttlGxrJ3jlzA9BgNVHSMENjA0gBT6JFAIzTCwTQemGv/ttlGxrJ3jl6ERpA8w
DTELMAkGA1UEAwwCZnSCCQCgYHVmsMsmZzAMBgNVHRMEBTADAQH/MAsGA1UdDwQE
AwIBBjANBgkqhkiG9w0BAQsFAAOBgQBNpWNFWz68dRDvsVlAYTZKegCJsholBwW4
BaGKvqA8BtZTm/B6in3NmFjNH3AafRv3qceX1oOE0NNsC4aBKt34TovuE/bNehid
mitqeptL14lJpDSjltIh6QjMBEuhgnlhUB9zuPwOfJbdJihNK5H9Mys3/Zpflz0H
creX6zj9GQ==
-----END CERTIFICATE-----
creslin@creslab:~$ cp /tmp/certs/ca.crt /tmp/certs/9426e611
creslin@creslab:~$ ls -al /tmp/certs/9426e611
-rw-rw-r-- 1 creslin creslin 782 Aug 10 14:28 /tmp/certs/9426e611

set, export, SSL_CERT_DIR
print ssl.get_default_verify_paths from within python to show SSL see the capath /tmp/certs

cat s.py 
import ssl
print("\n ssl Paths", ssl.get_default_verify_paths() ,"\n")

creslin@creslab:~$ SSL_CERT_DIR=/tmp/certs/
creslin@creslab:~$ export SSL_CERT_DIR
creslin@creslab:~$ python3 s.py 

 ssl version OpenSSL 1.1.0g  2 Nov 2017

 ssl Paths DefaultVerifyPaths(cafile=None, capath='/tmp/certs/', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/lib/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/lib/ssl/certs') 

Run test script again - still fails <<---- This is where its broken. We've show SSL module sees capath and there is a hashed certificate in here.

creslin@creslab:~$ python3 a.py
Traceback (most recent call last):
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 822, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 804, in create_connection
    sock, protocol_factory, ssl, server_hostname)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 830, in _create_connection_transport
    yield from waiter
  File "/usr/lib/python3.6/asyncio/sslproto.py", line 505, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/lib/python3.6/asyncio/sslproto.py", line 201, in feed_ssldata
    self._sslobj.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "a.py", line 12, in <module>
    loop.run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "a.py", line 6, in main
    async with session.get('https://api.binance.com/api/v1/time') as response:
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/client.py", line 843, in __aenter__
    self._resp = await self._coro
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/client.py", line 366, in _request
    timeout=timeout
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 445, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 757, in _create_connection
    req, traces, timeout)
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 879, in _create_direct_connection
    raise last_exc
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 862, in _create_direct_connection
    req=req, client_error=client_error)
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 827, in _wrap_create_connection
    raise ClientConnectorSSLError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host api.binance.com:443 ssl:None [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)]
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7efc256f9ef0>

Add to OS store

creslin@creslab:~$ sudo mkdir /usr/local/share/ca-certificates/test/
creslin@creslab:~$ sudo cp /tmp/certs/ca.crt /usr/local/share/ca-certificates/test/
creslin@creslab:~$ sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

done.
Updating Mono key store
Mono Certificate Store Sync - version 5.12.0.226
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 133, your new list has 134
Certificate added: CN=ft
1 new root certificates were added to your trust store.
Import process completed.

Importing into BTLS system store:
I already trust 133, your new list has 134
Certificate added: CN=ft
1 new root certificates were added to your trust store.
Import process completed.
Done
done.

Now works.

creslin@creslab:~$ unset SSL_CERT_DIR
creslin@creslab:~$ python3 a.py
{"serverTime":1533900636485}

@asvetlov
Copy link
Member

@creslinux could you try your snippet with explicitly crafted SSLContext?

@creslinux
Copy link
Author

creslinux commented Aug 10, 2018

I could but that would be another test / different thing

The issue/bug im chasing down is aiohttp is not following OS/openssl/python-ssl guidence.

Every link being pointed at by others explicitly states capath when passed by openssl should be used to look for a certificate. There seems to be a paper-review opinion that "everything is correct" but technical testing shows it is not as assumed.

The challenge is users of apps / apps being deployed into environments where "their" company ca certificate is in a location deployed for general use.

@creslinux
Copy link
Author

(plus my 6mnth puppy is telling me its her toilet/walk time -- and ... well, carpet priorities)

@creslinux
Copy link
Author

@tiran wrote

Yes, ssl.create_default_context() calls SSLContext.set_default_verify_paths, which calls >SSL_CTX_set_default_verify_paths internally. >https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_set_default_verify_paths.html will load the >certificates from file or directory of hashed passwords. aiohttp does everything correctly.

As tests show will load the certificates from file or directory of hashed passwords is false, isn't this then a bug?

@asvetlov
Copy link
Member

Reminder: asyncio uses MemoryBIO, requests and others work with wrap_socket.

@creslinux I have no certificates to test myself, I need your help to pin down the problem.

@webknjaz
Copy link
Member

@asvetlov JFYI I'm trying to track SSL testing helpers here: cherrypy/cheroot#95. Maybe we could try to add trustme to the test suite?

@ldbfpiaoran
Copy link

@asvetlov So how do you ignore the ssl certificate error? Is there a parameter like requests verify=False?

@asvetlov
Copy link
Member

RTFM please: http://docs.aiohttp.org/en/stable/client_advanced.html#ssl-control-for-tcp-sockets
r = await session.get('https://example.com', ssl=False)

@ldbfpiaoran
Copy link

@asvetlov
Thank you, but I used your method to get an exception.
TypeError: _request() got an unexpected keyword argument 'ssl'
The code is like this
Async def scan(url):
     Url = url +'/trace'
     # try:
     Async with aiohttp.ClientSession() as session:
         Res = await session.get(url,ssl=False).text()

@asvetlov
Copy link
Member

Perhaps you use an old aiohttp version

@ldbfpiaoran
Copy link

@asvetlov thank you very much

@lock
Copy link

lock bot commented Oct 29, 2019

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.

@lock lock bot added the outdated label Oct 29, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Oct 29, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
invalid This doesn't seem right outdated question StackOverflow
Projects
None yet
Development

No branches or pull requests

6 participants