IPv6 not preferred when -Djava.net.preferIPv6Addresses=system is set

[DanielThomas]

We're setting -Djava.net.preferIPv6Addresses=system and we've found Netty doesn't honor the setting, because Netty implements IPv4/IPv6 loopback address/interface detection itself, causing different results from InetAddress.

In recent JDKs, LookupPolicy takes care how addresses are resolved, but the logic is essentially the same in all releases since JDK 9, but it's harder to follow in releases before LookupPolicy was added. In short:

• If -Djava.net.preferIPv4Stack=true is set, IPv4 is forced
• If the system is IPv4 or IPv6 only, the available protocol is used
• Otherwise both IPv4 and v6 are enabled. The preferIPv6Addresses value determines the ordering of addresses:
  • For false IPv4 addresses are sorted first
  • For true IPv6 addresses are sorted first
  • For system addresses are provided in the order getaddrinfo returns them in
• If no preference is specified via either flag, IPv4 addresses are ordered first. Equivalent to preferIPv6Addresses=false

The NetUtil implementation differs to this, and doesn't implement the system value. The real problem is the address reordering caused by the implicit preference:

• netty/resolver-dns/src/main/java/io/netty/resolver/dns/DnsAddressResolveContext.java

  Lines 63 to 67 in 4eda913

  	@Override
  	List<InetAddress> filterResults(List<InetAddress> unfiltered) {
  	Collections.sort(unfiltered, PreferredAddressTypeComparator.comparator(parent.preferredAddressType()));
  	return unfiltered;
  	}

• netty/resolver-dns/src/main/java/io/netty/resolver/dns/PreferredAddressTypeComparator.java

  Lines 30 to 39 in 4eda913

  	static PreferredAddressTypeComparator comparator(InternetProtocolFamily family) {
  	switch (family) {
  	case IPv4:
  	return IPv4;
  	case IPv6:
  	return IPv6;
  	default:
  	throw new IllegalArgumentException();
  	}
  	}

The system value is required for mixed environments, because the combination of true/false for the two settings causes connectivity issues depending on the combination of source/destination being IPv4-only, IPv6-only or dualstack:

We chose system because it avoids the complexity of deciding whether it's safe to set -Djava.net.preferIPv6Addresses=true.

Expected behavior

Netty's selection of IP version, loopback address and name resolution order is identical to the JDK.

Actual behavior

The preferIPv6Addresses=system is not honored, causing addresses to be reordered based on the default preference, making it impossible to safely prefer IPv6 in mixed-protocol environments.

Steps to reproduce

Run with:

-Djava.net.preferIPv6Addresses=system

Note the failure:

io.netty.util.internal.SystemPropertyUtil: Unable to parse the boolean system property 'java.net.preferIPv6Addresses':system - using the default value: false

I don't quite follow why this is implemented explicitly, rather than calling io.netty.util.internal.SocketUtils#loopbackAddress and checking the instance of the returned address, that seems safer, with a check for the system value to determine if address reordering should be disabled.

Netty version

At least 4.1 and later.

JVM version

JDK 9 and later.

OS version

All operating systems.

[franckhlmartin]

It also seems that the class RoundRobinInetAddressResolver re-order the list of IPs with no consideration for the inet family of those addresses.
see #14278