New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_SocketType_resolve fails #580

Closed
sorcio opened this Issue Jul 29, 2018 · 10 comments

Comments

Projects
None yet
2 participants
@sorcio
Contributor

sorcio commented Jul 29, 2018

Apparently on Mac, on some setups, getaddrinfo() will always return IPv4 mapped addresses regardless of whether the AI_V4MAPPED is actually set. This causes failures in this assertion

sock6.setsockopt(tsocket.IPPROTO_IPV6, tsocket.IPV6_V6ONLY, True)
with pytest.raises(tsocket.gaierror) as excinfo:
await s6res(("1.2.3.4", 80))

This is probably a system configuration issue and maybe there is not much we can do about it. But it's causing our tests to fail on some machines. I've seen this happen regularly on my work laptop for some time (macOS 10.13.5, Python 3.6 and 3.7) and a couple other sprinters at EuroPython had the same issue. Good news is that it doesn't affect our Mac CI.

One potential (maybe not ideal) solution is to verify whether the system has this quirk and in that case skip the test.

Details

Calling getaddrinfo() on an IPv4 address, with AF_INET6 and the AI_V4MAPPED flag returns an IPv4-mapped IPv6 address. This is documented and works in all platforms.

>>> getaddrinfo('1.2.3.4', 0, AF_INET6, SOCK_STREAM, 0, AI_V4MAPPED)
[(<AddressFamily.AF_INET6: 30>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::ffff:1.2.3.4', 0, 0, 0))]

If I only want IPv6 addresses I won't set the AI_V4MAPPED and I won't get any result back. For example, this is what happens on Linux:

>>> s.getaddrinfo('1.2.3.4', 0, s.AF_INET6, s.SOCK_STREAM, 0, 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -9] Address family for hostname not supported

But if I try the same on my Mac I still get the IPv4-mapped results, even though I shouldn't!

>>> getaddrinfo('1.2.3.4', 0, AF_INET6, SOCK_STREAM, 0, 0)
[(<AddressFamily.AF_INET6: 30>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::ffff:1.2.3.4', 0, 0, 0))]
@njsmith

This comment has been minimized.

Member

njsmith commented Jul 29, 2018

Huh, weird.

Does this only happen if you explicitly request AF_INET6? What happens if you leave the family unspecified, like:

getaddrinfo('1.2.3.4', 0, 0, SOCK_STREAM, 0, 0)

?

@njsmith

This comment has been minimized.

Member

njsmith commented Jul 29, 2018

(I'm trying to figure out whether we should just relax the test, or if we should do some fixups in our getaddrinfo to actually get rid of this behavior. Whenever trio calls getaddrinfo internally – outside of tests – it always uses family=0 and trusts that it won't get back V4 mapped addresses, so finding out whether that's currently broken is an important data point.)

@sorcio

This comment has been minimized.

Contributor

sorcio commented Jul 29, 2018

family=0 behaves the same as AF_INET, i.e. only returns v4 addresses and ignores AI_V4MAPPED.

I tried disabling the interface that had a default route configured on (I thought it might be a hack introduced to make v6 operations smoother for some applications) but the result is the same. Must be noted that I can't (or don't know how to) disable ::1

@njsmith

This comment has been minimized.

Member

njsmith commented Jul 30, 2018

Okay, that's good to know...

Another question: what do you get for:

getaddrinfo("facebook.com", 0, family=socket.AF_INET6, type=socket.SOCK_STREAM)
getaddrinfo("vorpus.org", 0, family=socket.AF_INET6, type=socket.SOCK_STREAM)

? (That's one site that resolves to both an ipv4 and an ipv6, and one that resolves to just an ipv4.)

Looking at the test more closely, I guess it is actually testing a real thing – when you have a AF_INET6 socket, and do await sock.connect(host, port), then we do

getaddrinfo(host, port, sock.family, sock.type, sock.proto, flags)

where for connect, flags is either 0 or AI_V4MAPPED depending on whether or not sock has IPV6_V6ONLY set. The IPV6_V6ONLY flag controls whether this socket supports V4 mapped addresses. So we're actually doing the right thing, and asking for V4 mapped responses in exactly the cases where we can use them, and not in the cases where we can't. So Apple's behavior here is not actually very helpful :-).

This is a pretty edge case-y kind of thing – it looks like the stdlib's version of this code never passes AI_V4MAPPED, so with a stdlib AF_INET6 socket doing sock.connect("1.2.3.4", port) will always fail, even if the socket does support V4 mapped addresses.

@sorcio

This comment has been minimized.

Contributor

sorcio commented Jul 30, 2018

On an IPv4-only network, I get IPv4-mapped addresses for both hostnames. I will try later with IPv6 internet.

@njsmith

This comment has been minimized.

Member

njsmith commented Jul 31, 2018

OK, I thought hard about adding a real workaround, but I don't think it's worth it given that:

  • So far this hasn't actually caused bugs in anyone's code, just caused spurious test failures
  • In order to hit it, you have to be using some fairly low-level and system-specific APIs: either calling getaddrinfo by hand, or else using sockets directly with IPV6_V6ONLY, and either way you're going to have to deal with system-specific behavior.
  • We can still test that our code is doing what it's supposed to, by running the test on e.g. Linux

So here's a patch to disable that test on buggy systems, and then we can close this: #589

If it does start causing actual problems for someone, we can re-open this. Working around it should be fairly straightforward: you can detect an IPv4-mapped address with ipaddress.IPv6Address(...).ipv4_mapped is not None (ref). So either user code or trio could do a little loop over getaddrinfo return values checking this.

njsmith added a commit to njsmith/trio that referenced this issue Jul 31, 2018

@sorcio

This comment has been minimized.

Contributor

sorcio commented Aug 1, 2018

Just as a further datapoint: when I have IPv6 internet both getaddrinfo('facebook.com', 80, AF_INET6) and getaddrinfo('facebook.com', 80, AF_INET6, SOCK_STREAM, 0, AI_V4MAPPED) return pure v6 addresses (not v4-mapped addresses).

@sorcio sorcio closed this in #589 Aug 1, 2018

@njsmith

This comment has been minimized.

Member

njsmith commented Aug 1, 2018

Interesting. Could it be that all our CI systems think they have IPv6 enabled, and the funny behavior only happens when MacOS thinks it has IPv4 only? Seems weird given how allergic most cloud providers are to IPv6, but who knows...

@sorcio

This comment has been minimized.

Contributor

sorcio commented Aug 1, 2018

I don't think that's the reason, because resolving dotted IPv4 address strings (like '1.2.3.4' above) will still do the v4-mapping in all conditions I tested.

@njsmith

This comment has been minimized.

Member

njsmith commented Aug 1, 2018

Oh, right. So much for that guess. I guess if we really want to know then the next step is to start digging through https://opensource.apple.com/tarballs/Libinfo/ – but I don't think I'm that curious :-).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment