Verify using IPv4 addresess when IPv6 isn't available logic for Armeria's DNS resolvers work

[anuraaga]

I had a report that after upgrading Armeria, requests started to fail with for example com.linecorp.armeria.client.UnprocessedRequestException: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: cloudtrace.googleapis.com/2404:6800:4004:80a:0:0:0:200a:443, scheme=gproto+https, headers=[:method=UNKNOWN, :path=?, :scheme=https, :authority=?], content=DefaultRpcRequest{serviceType=GrpcLogUtil, method=google.devtools.cloudtrace.v2.TraceService/BatchWriteSpans

netty/netty#9048 was added to Netty to fix things like this but there may be a regression after Armeria forked the DNS resolver. Haven't investigated anything though.

/cc @danada

[trustin]

RefreshingAddressResolverGroup follows this logic: https://github.com/line/armeria/blob/master/core/src/main/java/com/linecorp/armeria/client/RefreshingAddressResolverGroup.java#L66-L81

[danada]

Thanks for this.

To add some background, the server is running on Google Cloud and the problems go away after adding -Djava.net.preferIPv4Stack=true to jvm args.

[trustin]

Can we get the ip link and ip addr output (with some values masked)?

[minwoox]

Oops, we don't have that condition here. https://github.com/line/armeria/blob/master/core/src/main/java/com/linecorp/armeria/client/endpoint/dns/DnsAddressEndpointGroup.java#L108

[anuraaga]

@danada What are the before and after versions of Armeria where you see the issue?

[minwoox]

Ah, So This is happened after upgrading Armeria, right?

[danada]

We went from 0.95.0 to 0.97.0. I will try to get ip link and ip addr outputs tomorrow morning!

[danada]

Sorry for the delay, here's the output (don't mind the busybox aliases):

$ iplink
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth0@if18692: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1460 qdisc noqueue 
    link/ether 1a:5e:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff

$ ipaddr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: eth0@if18692: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1460 qdisc noqueue 
    link/ether 1a:5e:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet 10.xxx.xxx.xxx/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::xxxx:xxxx:xxxx:xxxx/64 scope link 
       valid_lft forever preferred_lft forever

Network seems unreachable when pinging

$ ping6 ipv6.google.com
PING ipv6.google.com (2404:6800:4004:809::200e): 56 data bytes
ping6: sendto: Network is unreachable

[anuraaga]

The IPv6 starts with fe80 so it's link local and not a real address (similar to 169.blah on IPv4). Didn't look at the code yet but I suspect both our check in Netty and here are only checking if address is loopback, but not checking for link local / site local which aren't routable to the web

Though in that case wouldn't affect the version to matter. Hrm

[minwoox]

The IPv6 starts with fe80 so it's link local and not a real address (similar to 169.blah on IPv4).

Perhaps netty takes care of that but RefreshingDnsResolver doesn't? 😅
Let me take a look at the code if there's something I missed.

[anuraaga]

Also for reference, the client being referenced in that stack trace, cloudtrace.googleapis.com does not use an endpoint group, just a normal endpoint.

https://github.com/curioswitch/curiostack/blob/master/common/google-cloud/core/src/main/java/org/curioswitch/curiostack/gcloud/core/grpc/GrpcApiClientBuilder.java#L52

[minwoox]

@danada Is this always happening? or It differs every time the application started? (e.g It keeps throwing exceptions and it does not throw an exception at all after reboot so on.)

[danada]

It throws exceptions when trying to write spans to cloudtrace when processing a request. If no requests are processed no errors are thrown. The application has been restarted several times and produced the exception every time.

[minwoox]

Thanks for the information. 😉

[trustin]

Hopefully fixed via #2464. Please reopen if the problem shows up again.