GET Requests to link-local IPv6 addresses don't work on Python 3.7+

[mjonuschat]

🐞 Describe the bug
The aiohttp resolver loses information related to linklocal IPv6 addresses on Python 3.7+ due to a changes in the representation returned by socket.getaddrinfo()

💡 To Reproduce
Try to get an URL like http://[fe80::1%eth0]:8080/, it will result in an OSError (Invalid argument) exception.

This seems to be due to the way that scopeid's are handled in resolver.py:

Run socket.getaddrinfo('fe80::1%eth0', 8080, family=socket.AF_INET6, proto=socket.IPPROTO_TCP)[0][4] on python 3.6:

socket.getaddrinfo('fe80::1%eth0', 8080, family=socket.AF_INET6, proto=socket.IPPROTO_TCP)[0][4]
>>> socket.getaddrinfo('fe80::1%eth0', 8080, family=socket.AF_INET6, proto=socket.IPPROTO_TCP)[0][4]
('fe80::1%eth0', 8080, 0, 4)

Run it on python 3.7:

>>> socket.getaddrinfo('fe80::1%eth0', 8080, family=socket.AF_INET6, proto=socket.IPPROTO_TCP)[0][4]
('fe80::1', 8080, 0, 4)y

The address element of the tuple no longer includes the textual representation of the scope id, it's only contained in the matching scope_id element of the tuple - which then is missing when later callings _loop.create_connection().

💡 Expected behavior
The URL is successfully retrieved for link local IPv6 addresses.

📋 Logs/tracebacks

N/A

📋 Your version of the Python

$ python3 --version
Python 3.6.6
$ python3.7 --version
Python 3.7.5

📋 Your version of the aiohttp/yarl/multidict distributions

$ python -m pip show aiohttp
python -m pip show aiohttp
Name: aiohttp
Version: 3.6.2

$ python -m pip show multidict
Name: multidict
Version: 4.7.4

$ python -m pip show yarl
Name: yarl
Version: 1.4.2

📋 Additional context
OS: Centos7 Linux
Proxy Server: No
Related to: client

[fried]

https://bugs.python.org/issue35545#msg332275 more context around asycio create_server and how getaddrinfo is not broken in 3.7+ and the behavior is the new norm. Though create_server does not accept a scopeid :(, you have to pass the ipv6 address with %scope to create_server host field.

[socketpair]

Yep. This was the change I've done in Python. Tuple actually contains information about scope Id (4 - is the interface index of your network interface eth0). So, %eth0 is superfluous. Actually, the first tuple element should be a sequence of 16 bytes, not a string in IPv6 address form, but unfortunately, by compatibility reasons, it uses this form.

Possibly, you should use getnameinfo():

In [16]: socket.getnameinfo(('fe80::841e:8fff:fea4:4f19', 456, 0, 3), socket.AI_NUMERICHOST)
Out[16]: ('fe80::841e:8fff:fea4:4f19%wlp2s0', '456')

[socketpair]

@mjonuschat please check if my patch works for you. But anyway, I highly recommend you to use ipv6 ULA addresses. https://en.wikipedia.org/wiki/Unique_local_address. Unfortunately software is typically not aware of LL addresses.

For example. RFC does not say us whether we should use URLEncode for interface name or not. The same for percent sign in LL addresses.

For example, ] is a valid interface name in Linux.

[socketpair]

saghul/pycares#121. So, do not use pycares in such cases,

[mjonuschat]

@socketpair: Looks good, tested on python 3.7.5 using the following snippet:

import aiohttp
import asyncio

async def test():
   url = r"http://[fe80::1%usb0]:8080/api/info"
   async with aiohttp.ClientSession() as session:
       async with session.get(url, timeout=30) as resp:
           return await resp.json()

asyncio.run(test())