Connecting to IPv6-only hosts fails with a `RuntimeError`

[leenr]

Describe the bug

When aiohttp tries to connect to a host with only IPv6 address (with no IPv4 address), the exception RuntimeError: coroutine raised StopIteration is raised in TCPConnector._create_direct_connection caused by the StopIteration in _DNSCacheTable.next_addrs.

To Reproduce

1. Create aiohttp.ClientSession.
2. Try to request an URL with IPv6-only host, e.g. ipv6.google.com.
   E.g: await ClientSession().get('https://ipv6.google.com')

Expected behavior

await ClientSession().get('https://ipv6.google.com') does not fail and returns a response.

Logs/tracebacks

$ python -m asyncio
asyncio REPL 3.9.7 (default, Oct 10 2021, 15:13:22)
[GCC 11.1.0] on linux
>>> import asyncio
>>> from aiohttp import ClientSession
>>> await ClientSession().get('https://ipv6.google.com')
Traceback (most recent call last):
  File "/home/***/.local/lib/python3.9/site-packages/aiohttp/connector.py", line 894, in _resolve_host
    return self._cached_hosts.next_addrs(key)
  File "/home/***/.local/lib/python3.9/site-packages/aiohttp/connector.py", line 716, in next_addrs
    next(loop)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 445, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "<console>", line 1, in <module>
  File "/home/***/.local/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/home/***/.local/lib/python3.9/site-packages/aiohttp/connector.py", line 543, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/***/.local/lib/python3.9/site-packages/aiohttp/connector.py", line 906, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/home/***/.local/lib/python3.9/site-packages/aiohttp/connector.py", line 1153, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
RuntimeError: coroutine raised StopIteration

Python Version

$ python --version
Python 3.9.7

aiohttp Version

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.8.0
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author:
Author-email:
License: Apache 2
Location: /home/***/.local/lib/python3.9/site-packages
Requires: aiosignal, async-timeout, attrs, charset-normalizer, frozenlist, multidict, yarl
Required-by:

multidict Version

$ python -m pip show multidict
Name: multidict
Version: 5.1.0
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /home/***/.local/lib/python3.9/site-packages
Requires:
Required-by: aiohttp, grpclib, yarl

yarl Version

$ python -m pip show yarl
Name: yarl
Version: 1.6.3
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /home/***/.local/lib/python3.9/site-packages
Requires: idna, multidict
Required-by: aiohttp

OS

Linux

Related component

Client

Additional context

No response

Code of Conduct

• I agree to follow the aio-libs Code of Conduct

[leenr]

Bug does not occur in 3.7.4.post0:

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.7.4.post0
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: Nikolay Kim
Author-email: fafhrd91@gmail.com
License: Apache 2
Location: /home/***/.local/lib/python3.9/site-packages
Requires: async-timeout, attrs, chardet, multidict, typing-extensions, yarl
Required-by:

$ python -m asyncio
asyncio REPL 3.9.7 (default, Oct 10 2021, 15:13:22)
[GCC 11.1.0] on linux
>>> import asyncio
>>> from aiohttp import ClientSession
>>> await ClientSession().get('https://ipv6.google.com')
<ClientResponse(https://ipv6.google.com) [200 OK]>
<CIMultiDictProxy(...)>

[leenr]

Looking through the diff, I think the issue was caused by 2b92ef7.
UPD 13:22 UTC: Yep, reverting it locally does fix the problem.
UPD 13:34 UTC: It's seems like this commit skips an address if it's scope is 0 (global). So connections should work to non-global IPv6 addresses, but should not to global ones.

[webknjaz]

OS

Linux

Could you specify the actual OS/distro name and version, not just the name of the kernel project?

Also, it would be useful to have some sort of a regression test PR with a pytest-based test case. (We can mark it as xfail per https://pganssle-talks.github.io/xfail-lightning until it's fixed but it should exist at least)

[leenr]

Could you specify the actual OS/distro name and version, not just the name of the kernel project?

Arch Linux, rolling, Linux 5.14.14-zen1-1-zen #1 ZEN SMP PREEMPT Wed, 20 Oct 2021 21:35:17 +0000 x86_64 GNU/Linux.

Also, it would be useful to have some sort of a regression test PR with a pytest-based test case.

I don't see it as a straightforward task, since it's seems to me it only reproduces on global scope addresses, and not on a loopback addresses. Sorry, I don't have enough time to do it right now.

Thanks!

[asvetlov]

Reproducer:

import asyncio
import aiohttp


async def main():
    async with aiohttp.ClientSession() as client:
        async with client.get('https://ipv6.google.com') as resp:
            await resp.read()

asyncio.run(main())

Uses external site, sorry.

[Hanaasagi]

Sorry about it. I made a pull request #6204 for this.