certificate is not valid for IP

[lilydjwg]

🐞 Describe the bug

When requesting with IPv6 enabled, I get this weird exception.

💡 To Reproduce

1. python3 -m venv venv and . venv/bin/activate
2. pip install -U aiohttp
3. Run code:

import asyncio
import aiohttp

async def main():
  async with aiohttp.ClientSession() as session:
    r = await session.get('https://pypi.org/')
    print(await r.text())

if __name__ == '__main__':
  asyncio.run(main())

4. See error.

💡 Expected behavior

📋 Logs/tracebacks

Traceback (most recent call last): 
  File "/home/lilydjwg/tmpfs/venv/lib/python3.8/site-packages/aiohttp/connector.py", line 946, in _wrap_create_connection  
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection                                      
    transport, protocol = await self._create_connection_transport(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport                                              await waiter                                                                                                                          File "/usr/lib/python3.8/asyncio/sslproto.py", line 529, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata                                                             
    self._sslobj.do_handshake()
  File "/usr/lib/python3.8/ssl.py", line 944, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid 
for '151.101.128.223'. (_ssl.c:1124)

The above exception was the direct cause of the following exception: 

Traceback (most recent call last):
  File "t.py", line 12, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "t.py", line 8, in main
    r = await session.get('https://pypi.org/')
  File "/home/lilydjwg/tmpfs/venv/lib/python3.8/site-packages/aiohttp/client.py", line 490, in _request
    conn = await self._connector.connect(
  File "/home/lilydjwg/tmpfs/venv/lib/python3.8/site-packages/aiohttp/connector.py", line 528, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/lilydjwg/tmpfs/venv/lib/python3.8/site-packages/aiohttp/connector.py", line 868, in _create_connection
    _, proto = await self._create_direct_connection(
  File "/home/lilydjwg/tmpfs/venv/lib/python3.8/site-packages/aiohttp/connector.py", line 1023, in _create_direct_connection
    raise last_exc
  File "/home/lilydjwg/tmpfs/venv/lib/python3.8/site-packages/aiohttp/connector.py", line 999, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/home/lilydjwg/tmpfs/venv/lib/python3.8/site-packages/aiohttp/connector.py", line 948, in _wrap_create_connection
    raise ClientConnectorCertificateError(
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host pypi.org:443 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '151.101.128.223'. (_ssl.c:1124)")]

📋 Your version of the Python

$ python --version
Python 3.8.6

📋 Your version of the aiohttp/yarl/multidict distributions

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.7.0
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: Nikolay Kim
Author-email: fafhrd91@gmail.com
License: Apache 2
Location: /home/lilydjwg/tmpfs/venv/lib/python3.8/site-packages
Requires: yarl, attrs, async-timeout, chardet, multidict
Required-by: 

$ python -m pip show multidict
Name: multidict
Version: 5.0.0
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /home/lilydjwg/tmpfs/venv/lib/python3.8/site-packages
Requires: 
Required-by: yarl, aiohttp

$ python -m pip show yarl
Name: yarl
Version: 1.6.2
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /home/lilydjwg/tmpfs/venv/lib/python3.8/site-packages
Requires: idna, multidict
Required-by: aiohttp

📋 Additional context

I find this when I'm trying to reproduce another error.

[djmitche]

Seeing the same with python 3.6 and 3.7.

Logs are here and here.

It looks like aiohttp is passing an IP to the SSL verification.

[djmitche]

Repro for python 3.6:

import aiohttp

loop = asyncio.get_event_loop()

async def main():
  async with aiohttp.ClientSession(loop=loop) as session:                                                                                                                                                                                                                                                                                                                                      
    r = await session.get('https://pypi.org/')
    print(await r.text())

if __name__ == '__main__':
  loop.run_until_complete(main())

[djmitche]

In aiohttp/connector.py it looks like hinfo['hostname'] is a name in aiohttp==3.6.3 and an IP in aiohttp==3.7.0.

[djmitche]

In the good case, self._resolver.resolve (a ThreadedResolver) returns

resolver -> [{'hostname': 'pypi.org', 'host': '151.101.192.223', 'port': 443, 'family': <AddressFamily.AF_INET: 2>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICHOST: 4>}, {'hostname': 'pypi.org', 'host': '151.101.128.223', 'port': 443, 'family': <AddressFamily.AF_INET: 2>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICHOST: 4>}, {'hostname': 'pypi.org', 'host': '151.101.64.223', 'po rt': 443, 'family': <AddressFamily.AF_INET: 2>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICHOST: 4>}, {'hostname': 'pypi.org', 'host': '151.101.0.223', 'port': 443, 'family': <AddressFamily.AF_INET: 2>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICHOST: 4>}, {'hostname': 'pypi.org', 'host': '2a04:4e42::223', 'port': 443, 'family': <AddressFamily.AF_INET6: 10>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICHOST: 4>}, {'hostname': 'pypi.org', 'host': '2a04:4e42:600::223', 'port': 443, 'family': <AddressFamily.AF_INET6: 10>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICHOST: 4>}, {'hostname': 'pypi.org', 'host': '2a04:4e42:400::223', 'port': 443, 'family': <AddressFamily.AF_INET6: 10>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICHOST: 4>}, {'hostname': 'pypi.org ', 'host': '2a04:4e42:200::223', 'port': 443, 'family': <AddressFamily.AF_INET6: 10>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICHOST: 4>}]

In the bad case,

[{'hostname': '151.101.192.223', 'host': '151.101.192.223', 'port': 443, 'family': <AddressFamily.AF_INET: 2>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICSERV|AI_NUMERICHOST: 1028>}, {'hostname': '151.101.128.223', 'host': '151.101.128.223', 'port': 443, 'family': <AddressFamily.AF_INET: 2>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICSERV|AI_NUMERICHOST: 1028>}, {'hostname': '15 1.101.64.223', 'host': '151.101.64.223', 'port': 443, 'family': <AddressFamily.AF_INET: 2>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICSERV|AI_NUMERICHOST: 1028>}, {'hostname': '151.101.0.223', 'host': '151.101.0.223', 'port': 443, 'family': <AddressFamily.AF_INET: 2>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICSERV|AI_NUMERICHOST: 1028>}, {'hostname': '2a04:4e42::223', 'host': '2a04:4e42::223', 'port': 443, 'family': <AddressFamily.AF_INET6: 10>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICSERV|AI_NUMERICHOST: 1028>}, {'hostname': '2a04:4e42:600::223', 'host': '2a04:4e42:600::223', 'port': 443, 'family': <AddressFamily.AF_INET6: 10>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICSERV|AI_NUMERICHOST: 1028>}, {'hostname': '2a04:4e42:400::223', 'host': '2a04 :4e42:400::223', 'port': 443, 'family': <AddressFamily.AF_INET6: 10>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICSERV|AI_NUMERICHOST: 1028>}, {'hostname': '2a04:4e42:200::223', 'host': '2a04:4e42:200::223', 'port': 443, 'family': <AddressFamily.AF_INET6: 10>, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICSERV|AI_NUMERICHOST: 1028>}]

[djmitche]

Old resolve:

aiohttp/aiohttp/resolver.py

Lines 27 to 40 in 1940503

	async def resolve(self, host: str, port: int=0,
	family: int=socket.AF_INET) -> List[Dict[str, Any]]:
	infos = await self._loop.getaddrinfo(
	host, port, type=socket.SOCK_STREAM, family=family)
	hosts = []
	for family, _, proto, _, address in infos:
	hosts.append(
	{'hostname': host,
	'host': address[0], 'port': address[1],
	'family': family, 'proto': proto,
	'flags': socket.AI_NUMERICHOST})
	return hosts

New resolve:

aiohttp/aiohttp/resolver.py

Lines 28 to 53 in cac3cd2

	async def resolve(self, host: str, port: int=0,
	family: int=socket.AF_INET) -> List[Dict[str, Any]]:
	infos = await self._loop.getaddrinfo(
	host, port, type=socket.SOCK_STREAM, family=family)
	hosts = []
	for family, _, proto, _, address in infos:
	if family == socket.AF_INET6 and address[3]: # type: ignore
	# This is essential for link-local IPv6 addresses.
	# LL IPv6 is a VERY rare case. Strictly speaking, we should use
	# getnameinfo() unconditionally, but performance makes sense.
	host, _port = socket.getnameinfo(
	address, socket.NI_NUMERICHOST | socket.NI_NUMERICSERV)
	port = int(_port)
	else:
	host, port = address[:2]
	hosts.append({
	'hostname': host,
	'host': host,
	'port': port,
	'family': family,
	'proto': proto,
	'flags': socket.AI_NUMERICHOST | socket.AI_NUMERICSERV,
	})
	return hosts

so it looks like this is a simple matter of variable shadowing. However, it's a very serious error!