IPv4 prioritized over IPv6 for local server with custom DNS server

[janedenone]

I did this

In my dual-stack LAN, a bind server resolves local domains (*.something.internal) for both A and AAAA records, and all LAN webservers listen on IPv4 and IPv6. When accessing the servers, IPv6 gets prioritized (as expected) for all webservers except for my local client (which also acts as a webserver):

# This is my local client and webserver
curl -Iv https://static.something.internal/
* Host static.something.internal:443 was resolved.
* IPv6: 2001:908:45a1:bca0:10e0:6db1:72f1:9c2f
* IPv4: 192.168.78.55
*   Trying 192.168.78.55:443...
* Connected to static.something.internal (192.168.78.55) port 443

# This is another webserver on the LAN
curl -Iv https://nuc.something.internal/
* Host nuc.something.internal:443 was resolved.
* IPv6: 2001:908:45a1:bca0:1e69:7aff:feaf:7455
* IPv4: 192.168.78.42
*   Trying [2001:908:45a1:bca0:1e69:7aff:feaf:7455]:443...
* Connected to nuc.something.internal (2001:908:45a1:bca0:1e69:7aff:feaf:7455) port 443

Firefox and Safari behave exactly the same, while Chrome uses IPv6 addresses for all servers. I used the Network tab of the developer tools in all three browsers to compare and verify the behavior.

If I populate /etc/hosts on the local machine (using 127.0.0.1 and ::1 for its domains) instead of using my DNS server, all applications (including Firefox and Safari) prioritize IPv6:

curl -Iv https://static.something.internal/
* Host static.something.internal:443 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:443...
* Connected to static.something.internal (::1) port 443

I expected the following

I would expect curl to always prioritize IPv6, especially since #11465 was solved in September 2023.

curl/libcurl version

curl 8.9.1 (aarch64-apple-darwin23.4.0) libcurl/8.9.1 OpenSSL/3.3.1 (SecureTransport) zlib/1.2.12 brotli/1.1.0 zstd/1.5.6 libidn2/2.3.7 libssh2/1.11.0 nghttp2/1.61.0 librtmp/2.3
Release-Date: 2024-07-31
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

operating system

macOS 14.6.1

[icing]

Could you give us the log output when running with --trace-config all? It looks like the socket setup fails before the "Trying..." message is printed.

[janedenone]

Sure:

$ /opt/homebrew/Cellar/curl/8.9.1/bin/curl -Iv --trace-config all https://eden.internal/
18:23:40.127947 [0-x] * [READ] client_reset, clear readers
18:23:40.130042 [0-0] * Host eden.internal:443 was resolved.
18:23:40.130066 [0-0] * IPv6: 2001:908:45a1:bca0:10e0:6db1:72f1:9c2f
18:23:40.130083 [0-0] * IPv4: 192.168.78.55
18:23:40.130117 [0-0] * [HTTPS-CONNECT] added
18:23:40.130153 [0-0] * [HTTPS-CONNECT] connect, init
18:23:40.130171 [0-0] * [HTTPS-CONNECT] connect, check h21
18:23:40.130188 [0-0] * [HAPPY-EYEBALLS] created ipv4 (timeout 299998ms)
18:23:40.130206 [0-0] * [HAPPY-EYEBALLS] created ipv6 (timeout 299998ms)
18:23:40.130224 [0-0] * [HAPPY-EYEBALLS] ipv4 starting (timeout=299998ms)
18:23:40.130255 [0-0] *   Trying 192.168.78.55:443...
18:23:40.130279 [0-0] * [TCP] cf_socket_open() -> 0, fd=5
18:23:40.130371 [0-0] * [TCP] local address 192.168.78.55 port 60819...
18:23:40.130391 [0-0] * [HAPPY-EYEBALLS] ipv4 connect -> 0, connected=0
18:23:40.130409 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
18:23:40.130428 [0-0] * [TCP] adjust_pollset, !connected, POLLOUT fd=5
18:23:40.130445 [0-0] * [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
18:23:40.130463 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
18:23:40.130482 [0-0] * [HTTPS-CONNECT] connect, check h21
18:23:40.130501 [0-0] * [TCP] not connected yet
18:23:40.130518 [0-0] * [HAPPY-EYEBALLS] ipv4 connect -> 0, connected=0
18:23:40.130535 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
18:23:40.130553 [0-0] * [TCP] adjust_pollset, !connected, POLLOUT fd=5
18:23:40.130571 [0-0] * [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
18:23:40.130588 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
18:23:40.131469 [0-0] * [HTTPS-CONNECT] connect, check h21
18:23:40.131498 [0-0] * [TCP] connected
18:23:40.131515 [0-0] * [HAPPY-EYEBALLS] ipv4 connect -> 0, connected=1
18:23:40.131533 [0-0] * Connected to eden.internal (192.168.78.55) port 443

For comparison, the IPv6 connection is established just fine when I add the -6 option:

$ /opt/homebrew/Cellar/curl/8.9.1/bin/curl -Iv -6 --trace-config all https://eden.internal/
18:25:16.549006 [0-x] * [READ] client_reset, clear readers
18:25:16.551694 [0-0] * Host eden.internal:443 was resolved.
18:25:16.551713 [0-0] * IPv6: 2001:908:45a1:bca0:10e0:6db1:72f1:9c2f
18:25:16.551725 [0-0] * IPv4: (none)
18:25:16.551752 [0-0] * [HTTPS-CONNECT] added
18:25:16.551778 [0-0] * [HTTPS-CONNECT] connect, init
18:25:16.551790 [0-0] * [HTTPS-CONNECT] connect, check h21
18:25:16.551802 [0-0] * [HAPPY-EYEBALLS] created ipv6 (timeout 299998ms)
18:25:16.551814 [0-0] * [HAPPY-EYEBALLS] ipv6 starting (timeout=299998ms)
18:25:16.551846 [0-0] *   Trying [2001:908:45a1:bca0:10e0:6db1:72f1:9c2f]:443...
18:25:16.551868 [0-0] * [TCP] cf_socket_open() -> 0, fd=5
18:25:16.552042 [0-0] * [TCP] local address 2001:908:45a1:bca0:10e0:6db1:72f1:9c2f port 60924...
18:25:16.552054 [0-0] * [HAPPY-EYEBALLS] ipv6 connect -> 0, connected=0
18:25:16.552066 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
18:25:16.552079 [0-0] * [TCP] adjust_pollset, !connected, POLLOUT fd=5
18:25:16.552091 [0-0] * [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
18:25:16.552103 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
18:25:16.552115 [0-0] * [HTTPS-CONNECT] connect, check h21
18:25:16.552129 [0-0] * [TCP] not connected yet
18:25:16.552141 [0-0] * [HAPPY-EYEBALLS] ipv6 connect -> 0, connected=0
18:25:16.552152 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
18:25:16.552165 [0-0] * [TCP] adjust_pollset, !connected, POLLOUT fd=5
18:25:16.552177 [0-0] * [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
18:25:16.552189 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
18:25:16.545a108 [0-0] * [HTTPS-CONNECT] connect, check h21
18:25:16.545a135 [0-0] * [TCP] connected
18:25:16.545a153 [0-0] * [HAPPY-EYEBALLS] ipv6 connect -> 0, connected=1
18:25:16.545a172 [0-0] * Connected to eden.internal (2001:908:45a1:bca0:10e0:6db1:72f1:9c2f) port 443

[icing]

There seems to be a bug where curl does not try the ipv6 address as it should. Investigating...

Update: ah, now I see it. curl inspect the address list from the resolver and starts with the family that comes first. Usually, this is ipv6, but in your case a ipv4 address comes first. So, it tries this first and connects.

@bagder: you are more familiar with the history and intention of this address family approach. Is this what we want or would we always prefer to start with ipv6, if present, no matter the order in the list?

[janedenone]

Thanks for looking into this! According to the log, IPv6 is resolved first:

18:23:40.127947 [0-x] * [READ] client_reset, clear readers
18:23:40.130042 [0-0] * Host eden.internal:443 was resolved.
18:23:40.130066 [0-0] * IPv6: 2001:908:45a1:bca0:10e0:6db1:72f1:9c2f
18:23:40.130083 [0-0] * IPv4: 192.168.78.55

Am I missing something? Because the DNS lookup tool in Firefox (about:networking#dnslookuptool) does indeed return the "wrong" order (IPv4, then IPv6) for the local client, but the correct one for all other machines in my LAN.

[bagder]

That output always shows IPv6 before IPv4.

The order returned to curl is determined by getaddrinfo() (and /etc/gai.conf)

[janedenone]

Ok, I see the difference at the happy eyeballs stage now.

[bagder]

Is this what we want or would we always prefer to start with ipv6, if present, no matter the order in the list?

I believe maybe that is a trace of trying to respect getaddrinfo(), as that is supposed to return the addresses in a priority order. But thinking about it now, I think that is most likely the wrong thing and we should probably forcibly always go with IPv6 first.

[vityank]

This change breaks systems with IPv6 disabled but ::1 present in hosts file for localhost(Default on RHEL).
I think, at least for Linux, The code should run-time check the sysctl net.ipv6.conf.all.disable_ipv6 and for current outgoing iface
the /proc/sys/net/ipv6/conf/LIN_IFACE_DEV/disable_ipv6 and if IPv6 is disabled use v4,v6 order, otherwise the v6,v4.

All PHPs cURL tests which use server bound to localhost are failing with a process timeout now:

 ** ERROR: process timed out **
================================================================================
--
     %A
     Warning: curl_multi_add_handle(): CURLOPT_STDERR resource has gone away, resetting to stderr in %s on line %d
     %A
005- Ok for CURLOPT_STDERR
005+ * IPv6: ::1
006+ * IPv4: 127.0.0.1
007+ *   Trying [::1]:31322...
008+ * Immediate connect fail for ::1: Cannot assign requested address
009+ *   Trying 127.0.0.1:31322...
010+ * Connection #0 is not open enough, cannot reuse

[icing]

@vityank can you provide a more detailed trace? curl's happy eyeballing should immediately continue to connect with the ipv4 address. (Hint a curl_global_trace("all") at the start should give the details).

[bagder]

This change breaks systems with IPv6 disabled

How is it disabled? curl should not try IPv6 if it is disabled

but ::1 present in hosts file for localhost

curl does not use the hosts file for localhost, it is internally "hard-coded".

[vityank]

@icing,
The case is PHP's Test curl_setopt() with curl_multi function with basic functionality [ext/curl/tests/curl_basic_018.phpt](One of a few cases failing), thus I can't minimize it to direct curl test now, as it's PHP's curl extension affected.
I don't have too much information now, but those tests locks for few minutes then fail due to timeout and this is started with 8.10.0.
Failing test, mentioning IPv6 is:
Variation of bug #48203 with curl_multi_exec (Crash when file pointers passed to curl are closed before calling curl_multi_exec) [ext/curl/tests/bug48203_multi.phpt]

@bagder
We have sysctl net.ipv6.conf.all.disable_ipv6=1.
Nics doesn't have ipv6 addresses assigned as a result.
However /etc/hosts have this row, and it affects system resolver(Our cURL is built w/o c-ares and cURL itlsef has IPv6 support, so it can properly resolve external IPv6 only hosts):
::1 localhost localhost.localdomain

I'm not 100% sure if is's caused by this change or there are some other regression.

[bagder]

@vityank I recommend you create a new issue with all the details. This old issue is closed and fixed. You see something else.

I don't think the IPv6 part is what causes your the problem but rather the "Connection #0 is not open enough, cannot reuse" part. Are you sure this exact commit broke the functionality for you?

[vityank]

@bagder,
Thanks I will. I'm not sure its related to this specific commit. This was the first suspect but tests hang for 10m, so it doesn't look like resolve related... I'll continue investigation and will try to possibly bisect it.