Be able to retry the query via TCP if a query failed because of a timeout

[normanmaurer]

Motivation:

We should allow people to retry the query via TCP if the query failed because of a
timeout when using UDP.

Modifications:

• Move all the retry code for TCP into DnsQueryContext so we can reuse
  the same code for handling truncation and retry.
• Retry on timeout if configured by user
• Add unit tests

Result:

More robust resolver

[idelpivnitskiy]

Can you please elaborate on the motivation in a bit more details? Where this is coming from?
In my experience, DNS timeouts are quite frequent and automatically retrying them via TCP may cause unexpected side-effects, like:

1. People expecting the query to fail within a pre-defined timeout. Retrying it will double the total timeout.
2. Spike of TCP traffic for DNS when timeouts are frequent enough.

I will propose to make this an opt-in feature, but mostly looking for a motivational story behind it.

[hellojukay]

@idelpivnitskiy you may read this https://www.weave.works/blog/racy-conntrack-and-dns-lookup-timeouts

Dns through tcp is real in needed, maybe tcp is not default behavior, but we should make people can enable tcp fallback option and set tcp timeout.

We use redisson watch dns change event, redisson dns watching implement by netty dns resolver, redisson dns udp timeout 5000ms frequently becase of udp dns race. redisson/redisson#5137 and #13705

[idelpivnitskiy]

Thanks for the link and additional context.

My arguments (1) and (2) remain valid. If we merge this PR, it changes the current behavior and may cause unexpected outages under heavy load and scale. So, if we proceed with this change, it should be configurable (preferable to be opt-in by default) to let impacted users experiment in a safe fashion without affecting everyone.

After reading the article, I'm not sure this specific change is a good mitigation for the described problem. DNS timeouts are quite often, happen not only in k8s env, and may be caused by various different reasons. There are cases, when automatic retry over TCP can make everything even worse.
Good mitigation points from the article are:
a. disable parallel lookups
b. use TCP for lookups

With (b) and RFC7766 together, it makes more sense to let users chose TCP as the default protocol for their DNS client and skip UDP completely or let it be a fallback. However, it opens other questions around connection management and DoS protection described in RFC7766.

[hellojukay]

We cannot assume what kind of network our program will run on. In any case, a timeout will occur. We should allow users to configure: when a failure occurs, switch to another network connection method.

[normanmaurer]

@idelpivnitskiy the problem is that there is really no way for people to to do a retry via TCP by themself (we might want to support also create a DnsNameResolver which only does TCP in the future). That said let's just make it configurable to also fallback to TCP on timeout.

[hellojukay]

Will this PR be merged in the futrue.

[normanmaurer]

Will have a look later this weekAm 10.01.2024 um 04:39 schrieb hellojukay ***@***.***>: Will this PR be merged in the futrue. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[idelpivnitskiy]

Thanks for the config flag @normanmaurer!

[normanmaurer]

Will work on a patch for main

[paolosciamm]

Hi,
thanks for this release.
Just a question about this quote:

• Retry on timeout if configured by user

Using the 4.1.105.Final version, how can I configure/enable the TCP fallback in Spring?
Is it needed a particular properties or bean config?

[normanmaurer]

you would configure it via the DnsNameResolverBuilder.

[violetagg]

Hi, thanks for this release. Just a question about this quote:

• Retry on timeout if configured by user

Using the 4.1.105.Final version, how can I configure/enable the TCP fallback in Spring? Is it needed a particular properties or bean config?

If you use WebClient then you can configure the Reactor Netty HttpClient like this:

HttpClient client = HttpClient.create().resolver(spec -> spec.retryTcpOnTimeout(true));

See more here https://projectreactor.io/docs/netty/release/reference/index.html#_host_name_resolution_2

[paolosciamm]

Hi, thanks for this release. Just a question about this quote:

• Retry on timeout if configured by user

Using the 4.1.105.Final version, how can I configure/enable the TCP fallback in Spring? Is it needed a particular properties or bean config?

If you use WebClient then you can configure the Reactor Netty HttpClient like this:

HttpClient client = HttpClient.create().resolver(spec -> spec.retryTcpOnTimeout(true));

See more here https://projectreactor.io/docs/netty/release/reference/index.html#_host_name_resolution_2

Thanks for the reply.
And if I won't use Reactor Netty, would adding the following configuration (according to @normanmaurer answer) be enough?

@Configuration
public class CustomDnsNameResolver {
    @Bean
    public DnsNameResolver customDnsNameResolverBuilder() {
        NioEventLoopGroup group = new NioEventLoopGroup();
        return new DnsNameResolverBuilder(group.next())
                .channelType(NioDatagramChannel.class)
                .resolvedAddressTypes(ResolvedAddressTypes.IPV4_ONLY)
                .nameServerProvider(DnsServerAddressStreamProviders.platformDefault())
                .queryTimeoutMillis(5000)
                .maxQueriesPerResolve(2)
                .build();
    }
}

[violetagg]

@paolosciamm I would say, you need .socketChannelType(NioSocketChannel.class, true)