test_SocketType_resolve fails

[sorcio]

Apparently on Mac, on some setups, getaddrinfo() will always return IPv4 mapped addresses regardless of whether the AI_V4MAPPED is actually set. This causes failures in this assertion

trio/trio/tests/test_socket.py

Lines 420 to 422 in fd0d877

	sock6.setsockopt(tsocket.IPPROTO_IPV6, tsocket.IPV6_V6ONLY, True)
	with pytest.raises(tsocket.gaierror) as excinfo:
	await s6res(("1.2.3.4", 80))

This is probably a system configuration issue and maybe there is not much we can do about it. But it's causing our tests to fail on some machines. I've seen this happen regularly on my work laptop for some time (macOS 10.13.5, Python 3.6 and 3.7) and a couple other sprinters at EuroPython had the same issue. Good news is that it doesn't affect our Mac CI.

One potential (maybe not ideal) solution is to verify whether the system has this quirk and in that case skip the test.

Details

Calling getaddrinfo() on an IPv4 address, with AF_INET6 and the AI_V4MAPPED flag returns an IPv4-mapped IPv6 address. This is documented and works in all platforms.

>>> getaddrinfo('1.2.3.4', 0, AF_INET6, SOCK_STREAM, 0, AI_V4MAPPED)
[(<AddressFamily.AF_INET6: 30>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::ffff:1.2.3.4', 0, 0, 0))]

If I only want IPv6 addresses I won't set the AI_V4MAPPED and I won't get any result back. For example, this is what happens on Linux:

>>> s.getaddrinfo('1.2.3.4', 0, s.AF_INET6, s.SOCK_STREAM, 0, 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -9] Address family for hostname not supported

But if I try the same on my Mac I still get the IPv4-mapped results, even though I shouldn't!

>>> getaddrinfo('1.2.3.4', 0, AF_INET6, SOCK_STREAM, 0, 0)
[(<AddressFamily.AF_INET6: 30>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::ffff:1.2.3.4', 0, 0, 0))]

[njsmith]

Huh, weird.

Does this only happen if you explicitly request AF_INET6? What happens if you leave the family unspecified, like:

getaddrinfo('1.2.3.4', 0, 0, SOCK_STREAM, 0, 0)

?

[njsmith]

(I'm trying to figure out whether we should just relax the test, or if we should do some fixups in our getaddrinfo to actually get rid of this behavior. Whenever trio calls getaddrinfo internally – outside of tests – it always uses family=0 and trusts that it won't get back V4 mapped addresses, so finding out whether that's currently broken is an important data point.)

[sorcio]

family=0 behaves the same as AF_INET, i.e. only returns v4 addresses and ignores AI_V4MAPPED.

I tried disabling the interface that had a default route configured on (I thought it might be a hack introduced to make v6 operations smoother for some applications) but the result is the same. Must be noted that I can't (or don't know how to) disable ::1

[njsmith]

Okay, that's good to know...

Another question: what do you get for:

getaddrinfo("facebook.com", 0, family=socket.AF_INET6, type=socket.SOCK_STREAM)
getaddrinfo("vorpus.org", 0, family=socket.AF_INET6, type=socket.SOCK_STREAM)

? (That's one site that resolves to both an ipv4 and an ipv6, and one that resolves to just an ipv4.)

Looking at the test more closely, I guess it is actually testing a real thing – when you have a AF_INET6 socket, and do await sock.connect(host, port), then we do

getaddrinfo(host, port, sock.family, sock.type, sock.proto, flags)

where for connect, flags is either 0 or AI_V4MAPPED depending on whether or not sock has IPV6_V6ONLY set. The IPV6_V6ONLY flag controls whether this socket supports V4 mapped addresses. So we're actually doing the right thing, and asking for V4 mapped responses in exactly the cases where we can use them, and not in the cases where we can't. So Apple's behavior here is not actually very helpful :-).

This is a pretty edge case-y kind of thing – it looks like the stdlib's version of this code never passes AI_V4MAPPED, so with a stdlib AF_INET6 socket doing sock.connect("1.2.3.4", port) will always fail, even if the socket does support V4 mapped addresses.

[sorcio]

On an IPv4-only network, I get IPv4-mapped addresses for both hostnames. I will try later with IPv6 internet.

[njsmith]

OK, I thought hard about adding a real workaround, but I don't think it's worth it given that:

• So far this hasn't actually caused bugs in anyone's code, just caused spurious test failures
• In order to hit it, you have to be using some fairly low-level and system-specific APIs: either calling getaddrinfo by hand, or else using sockets directly with IPV6_V6ONLY, and either way you're going to have to deal with system-specific behavior.
• We can still test that our code is doing what it's supposed to, by running the test on e.g. Linux

So here's a patch to disable that test on buggy systems, and then we can close this: #589

If it does start causing actual problems for someone, we can re-open this. Working around it should be fairly straightforward: you can detect an IPv4-mapped address with ipaddress.IPv6Address(...).ipv4_mapped is not None (ref). So either user code or trio could do a little loop over getaddrinfo return values checking this.

[sorcio]

Just as a further datapoint: when I have IPv6 internet both getaddrinfo('facebook.com', 80, AF_INET6) and getaddrinfo('facebook.com', 80, AF_INET6, SOCK_STREAM, 0, AI_V4MAPPED) return pure v6 addresses (not v4-mapped addresses).

[njsmith]

Interesting. Could it be that all our CI systems think they have IPv6 enabled, and the funny behavior only happens when MacOS thinks it has IPv4 only? Seems weird given how allergic most cloud providers are to IPv6, but who knows...

[sorcio]

I don't think that's the reason, because resolving dotted IPv4 address strings (like '1.2.3.4' above) will still do the v4-mapping in all conditions I tested.

[njsmith]

Oh, right. So much for that guess. I guess if we really want to know then the next step is to start digging through https://opensource.apple.com/tarballs/Libinfo/ – but I don't think I'm that curious :-).