DNS failure logs to be logged on info level instead of debug level

[kishor7007]

Title: DNS failure logs to be logged as info instead of debug

Description:

On strict dns / logical dns resolutions if there are errors on the DNS resolution, the failure logs are getting written only when debug log level enabled. To easily identify the DNS failure logs, it is beneficial to log them as info so that users can easily identify the failure logs without out need of debug log level enablement.

Sample Log:

2025-04-16T08:59:11.694103Z debug envoy dns external/envoy/source/extensions/network/dns_resolver/cares/dns_impl.cc:152 dns resolution for myapp.net failed with c-ares status 11 thread=19

[optional Relevant Links:]

Sample images given below

[ramaraochavali]

Can you use stats https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/dns_resolution.html to identify the failures? These logs can spam if there are my dns clusters in mesh

[kishor7007]

All are summarised counters at per Envoy,  without details on which specific FQDN like test.example.comlookup actually failed and respective failure reason.
Moreover, as failures should be addressed promptly rather than lingering in the system, having logs of DNS lookup failures logs enable monitoring systems like ElasticSearch to filter sidecar logs for real-time alarms, analyse historical patterns, trigger alerts, and improve response times efficiently.

[ramaraochavali]

You can also get which cluster DNS resolution failed with update_failure metric on cluster.

If you still want logging, you can use https://www.envoyproxy.io/docs/envoy/v1.34.0/operations/cli.html#cmdoption-component-log-level and enable it just for dns like dns:debug

[kishor7007]

Enabling the dns:debug level prints the unwanted success logs as well, right?

[yanavlasov]

You can use fine grain logger to limit the log messages emitted by Envoy. https://www.envoyproxy.io/docs/envoy/latest/operations/cli#cmdoption-enable-fine-grain-logging

Generally logging DNS errors in debug log file is not a viable approach as even at info level a busy proxy can produce too many log entries. However I'm unsure hot to surface DNS diagnostics in an easy way as Envoy does not have access log for DNS resolutions. Maybe others can suggest something.

[kishor7007]

Since debug logs are not enabled by default, any DNS errors occurring in the system may go unnoticed, potentially masking issues until an external user reports them. Given the minimal volume of DNS errors compared to the current logging activity in an active traffic system, enabling logs for these errors is essential to ensure timely awareness of potential problems.

[kishor7007]

waiting for the response on this.

[yanavlasov]

Changing the level of DNS error messages in dns_impl.cc to info can product a log spam for Envoy deployments that use dynamic forwarding proxy where DNS resolution errors can be common. I do not think this is practical.

If you need to monitor DNS resolution for clusters where error rate is lower, can use fine grain logger to just enable messages from dns_impl.cc ? To better isolate the errors we can downgrade the rest of messages in this file to trace.

[kishor7007]

Thanks @yanavlasov for the response, I agree with you that isolating the error logs from general logs, i.e.

1. Setting the general logs to trace level, providing maximum detail.
2. Setting the error logs specifically to debug level, providing enhanced visibility on problems.

[kishor7007]

@ramaraochavali and @yanavlasov,
Hope, we can proceed with this approach.

[ramaraochavali]

Sure. As long as info level is not touched, I am fine with that.

[kishor7007]

Hi @ramaraochavali and @yanavlasov,
As part of working on this, noticed that apple_dns_impl.cc already has warn log level for the errors.

grep -ir "ENVOY_LOG(warn" .
./apple/apple_dns_impl.cc: ENVOY_LOG(warn, "DNS resolver error ({}) in dnsServiceGetAddrInfo for {}", error, dns_name);
./apple/apple_dns_impl.cc: ENVOY_LOG(warn, "DNS resolver error in dnsServiceRefSockFD for {}", dns_name);
./apple/apple_dns_impl.cc: ENVOY_LOG(warn, "DNS resolver error ({}) in DNSServiceProcessResult", error);

Based on this, I would like to derive the following options.

1. Marking the cares/dns_impl.cc error logs to warn level as well to match consistence on the dns component error logs
2. Keeping the ./apple/apple_dns_impl.cc error logs as is i.e. warn level as this is already in the production and marking cares/dns_impl.cc error logs to debug level.
3. Marking the both ./apple/apple_dns_impl.cc and cares/dns_impl.cc error logs to debug level.

Please suggest a best suitable one to proceed on this.

[yanavlasov]

Apple DNS resolver is not used in production environments. You can either go with 2 or 3.