ClientConnectorSSLError when trying connect to PyPi

[Bobronium]

This happens every time I try to execute dephell command that needs connection with PyPi:

$ dephell deps add --envs dev test -- pytest
WARNING empty root passed 
ERROR ClientConnectorSSLError: Cannot connect to host pypi.org:443 ssl:None [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)] 

Other tools (twine, pip, Pipenv) work without any issue

Any ideas?

[orsinium]

Thank you for the issue. Please, run the command with --traceback --level=DEBUG and provide output.

[Bobronium]

Yep, here it is:

$ dephell deps add --envs dev test --traceback --level=DEBUG -- pytest
WARNING empty root passed 
DEBUG merge... 
ERROR ClientConnectorSSLError: Cannot connect to host pypi.org:443 ssl:None [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)] 
Traceback (most recent call last):
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/aiohttp/connector.py", line 924, in _wrap_create_connection
    await self._loop.create_connection(*args, **kwargs))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 820, in create_connection
    sock, protocol_factory, ssl, server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 846, in _create_connection_transport
    yield from waiter
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/sslproto.py", line 505, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/sslproto.py", line 201, in feed_ssldata
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/dephell/cli.py", line 77, in main
    result = task()
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/dephell/commands/deps_add.py", line 50, in __call__
    resolved = resolver.resolve(level=1, silent=self.config['silent'])
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/dephell/controllers/_resolver.py", line 86, in resolve
    resolved = self._resolve(debug=debug, silent=silent, level=level, spinner=spinner)
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/dephell/controllers/_resolver.py", line 113, in _resolve
    no_conflicts = self._apply_deps(deps, debug=debug)
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/dephell/controllers/_resolver.py", line 190, in _apply_deps
    conflict = self.apply(dep)
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/dephell/controllers/_resolver.py", line 49, in apply
    if not other_dep.compat:
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/dephell/models/dependency.py", line 148, in compat
    for group in self.groups:
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/dephell/models/groups.py", line 169, in __iter__
    self._load_release_deps(release)
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/dephell/models/groups.py", line 148, in _load_release_deps
    release.dependencies = loop.run_until_complete(gathered)[0]
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/dephell/controllers/_repos.py", line 141, in get_dependencies
    return await repo.get_dependencies(name=name, version=version, extra=extra)
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/dephell/repositories/_warehouse/_api.py", line 135, in get_dependencies
    deps = await asyncio.gather(asyncio.ensure_future(task))
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/dephell/repositories/_warehouse/_api.py", line 236, in _get_from_json
    async with session.get(url) as response:
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/aiohttp/client.py", line 1005, in __aenter__
    self._resp = await self._coro
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/aiohttp/client.py", line 476, in _request
    timeout=real_timeout
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/aiohttp/connector.py", line 522, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/aiohttp/connector.py", line 854, in _create_connection
    req, traces, timeout)
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/aiohttp/connector.py", line 992, in _create_direct_connection
    raise last_exc
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/aiohttp/connector.py", line 974, in _create_direct_connection
    req=req, client_error=client_error)
  File "/Users/rocky/.local/share/dephell/venvs/dephell/lib/python3.6/site-packages/aiohttp/connector.py", line 929, in _wrap_create_connection
    raise ClientConnectorSSLError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host pypi.org:443 ssl:None [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)]

[espdev]

See: aio-libs/aiohttp#2822

[Bobronium]

See: aio-libs/aiohttp#2822

So, it looks like on macOS (and other systems maybe?) aiohttp won't work out of the box with default certificate.

Using certify will solve the problem

[orsinium]

Can we fix it on dephell level?

[Bobronium]

Yes, just use:

ssl_context = ssl.create_default_context(cafile=certifi.where())
connector = aiohttp.TCPConnector(ssl=ssl_context)
async with aiohttp.ClientSession(connector=connector) as session:
    ...

instead of:

async with aiohttp.ClientSession() as session:
    ...

[Bobronium]

How about adding function like this and use it project-wide for any request?

async def make_request(url, method='get', decode_method='json', headers=None, **kwargs):
    ssl_context = ssl.create_default_context(cafile=certifi.where())
    connector = TCPConnector(ssl=ssl_context)

    async with ClientSession(connector=connector) as session:
        async with session.request(method, url, headers=headers, **kwargs) as response:
            response.raise_for_status()
            return await getattr(response, decode_method)()

[Bobronium]

Another option is to create this function:

def get_client_session(**kwargs):
    ssl_context = ssl.create_default_context(cafile=certifi.where())
    connector = TCPConnector(ssl=ssl_context)
    return ClientSession(connector=connector, **kwargs)

and then use:

async with get_client_session() as session:
    ...

This will not break any existing logic in project

[orsinium]

Why isn't it a default behavior in aiohttp?

[Bobronium]

Why isn't it a default behavior in aiohttp?

As asvetlov mentioned here:

aiohttp does nothing special with certs but uses built-in Python support.
In turn Python is not shipped with certificates bundle but uses OS one.

requests uses certifi for root certificates source. This approach has own advantages and disadvantages. Several years ago we decided to not rely on custom certificate libraries.

So the issue known, but considered ok

[orsinium]

This approach has own advantages and disadvantages.

Of course, @asvetlov made the decision on the base of some pros and cons of both possibilities. However, these differences aren't clear to me. Will we get in troubles by switching on certify?

[Bobronium]

Will we get in troubles by switching on certify?

As the requests uses certify as well as many other libs (e. g. aiogram that also uses it with aiohttp), I don't think so. But we'll see 😄

[asvetlov]

The main certifi question is: "who and when upgrades this package"?
Is it ok for you to have 3-years old certifi installed?

[orsinium]

What do you mean? Do you mean users never upgrade site-packages or certify developers don't do new bugfixes?

[asvetlov]

certifi devs are great and responsible, users are not.
You as the library author cannot control it.
An OS has a centralized auto-upgrade tool that is executed periodically and keeps it more-or-less up to date.
There is no such thing for python packages, and people are lazy.

Using aiohttp==0.17 is perfectly fine if your project was developed 5+ years ago and "just works". But using 5 years old root CA is a security hole, this is the difference.

[orsinium]

@asvetlov, thank you :)

Ok, in most of the cases connect to pypi, github, and custom repositories. I think, certify works here. Especially because dephell installs in it's own venv and has it's own actual certify version. Also, vendorizing on the way.