Increased DNS Timeouts After Upgrading to v1.31.0

[agrawroh]

Description

We have observed an increase in DNS timeouts after upgrading our Dev/Staging environment from v1.30.2 to v1.31.0 (Dev). All other factors, including traffic load, remained unchanged. We suspect this issue might be related to the recent major version upgrade of c-ares [Reference].

DNS Resolver Events:

An uptick in the DNS timeouts after switching to v1.31.0 which goes away once we move back to v1.30.2,

Number of DNS resolutions (didn't change),

cc @alyssawilk @deveshkandpal1224

[nezdolik]

cc @yanavlasov @mattklein123

[alyssawilk]

Huh. Would it be possible to run an experiment with the version downgraded back down? If so we could downgrade Envoy's dependency but I think we'd have to file with cares to fix the underlying problem

[arulthileeban]

This could be related: c-ares/c-ares#542, which is part of the c-ares v.1.20.0 release. AFAIK Envoy uses the defaults for timeouts. Should we bump the defaults internally and/or expose it through an option?

[agrawroh]

This could be related: c-ares/c-ares#542, which is part of the c-ares v.1.20.0 release. AFAIK Envoy uses the defaults for timeouts. Should we bump the defaults internally and/or expose it through an option?

@arulthileeban Thanks for the pointers. Re: DNS Timeouts, I tried doing that here and am happy to re-open this PR and scope it to c-ares only for now.

[alyssawilk]

Sure we can go with that or I can land #35335 which attempts to just restore prior behavior. thoughts?

[agrawroh]

@alyssawilk You can merge your changes and I'll also re-open my PR as I think it's good for users to have these values configurable (we might need to up it more on our end)

[alyssawilk]

SGTM, thanks!

[agrawroh]

@alyssawilk We tried back-porting your changes to set the previous defaults but we are still seeing the timeouts issue. I even tried to increase the timeout to 10s and am still noticing similar spikes. It looks like something else changed either around how we are reporting the timeouts or in the c-ares.

The graph is from ~30 days showing two set of spikes when we tried out rolling the new version without your changes at first and then after including the timeout changes.