Resolution providing incomplete answer on CoreDNS 1.6.6

[codyharris-h2o-ai]

Hello all!

I'm trying to diagnose why resolving a domain like "foo.cognitiveservices.azure.com" is giving me a different result in our K8s cluster than outside. Our cluster is an EKS cluster on Kubernetes 1.17 using coredns coredns:v1.6.6-eksbuild.1. I can confirm the proper behaviour in CoreDNS 1.7.1 on DigitalOcean.

Since I cannot reproduce this on a later version of CoreDNS, I assume there is either a different setting or a bug fixed between the two versions.

For example:

$ host foo.cognitiveservices.azure.com
foo.cognitiveservices.azure.com is an alias for eastus.api.cognitive.microsoft.com.
eastus.api.cognitive.microsoft.com is an alias for cognitiveuseprod.trafficmanager.net.
cognitiveuseprod.trafficmanager.net is an alias for cognitiveuseprod.azure-api.net.
cognitiveuseprod.azure-api.net is an alias for apimgmttmsxrvtifzqdtqhxmfmfanqbnjviisx1axz0alfmmew.trafficmanager.net.
apimgmttmsxrvtifzqdtqhxmfmfanqbnjviisx1axz0alfmmew.trafficmanager.net is an alias for cognitiveuseprod-eastus-01.regional.azure-api.net.
cognitiveuseprod-eastus-01.regional.azure-api.net is an alias for apimgmthsongb0y17ikykhnnaz0xc9gtblsk5ll468obeehld0.cloudapp.net.

And without coredns:

$ host h2o-ms-ocr.cognitiveservices.azure.com
foo.cognitiveservices.azure.com is an alias for eastus.api.cognitive.microsoft.com.
eastus.api.cognitive.microsoft.com is an alias for cognitiveuseprod.trafficmanager.net.
cognitiveuseprod.trafficmanager.net is an alias for cognitiveuseprod.azure-api.net.
cognitiveuseprod.azure-api.net is an alias for apimgmttmsxrvtifzqdtqhxmfmfanqbnjviisx1axz0alfmmew.trafficmanager.net.
apimgmttmsxrvtifzqdtqhxmfmfanqbnjviisx1axz0alfmmew.trafficmanager.net is an alias for cognitiveuseprod-eastus-01.regional.azure-api.net.
cognitiveuseprod-eastus-01.regional.azure-api.net is an alias for apimgmthsongb0y17ikykhnnaz0xc9gtblsk5ll468obeehld0.cloudapp.net.
apimgmthsongb0y17ikykhnnaz0xc9gtblsk5ll468obeehld0.cloudapp.net has address 40.121.217.232

I ran a tcpdump to capture the request and the response. The requests are functionally identical (both querying an A record), however the response from CoreDNS appears to be one record short of the answer. Indeed, if I query eastus.api.cognitive.microsoft.com, I get an A record back from CoreDNS 1.6.6; otherwise I get a CNAME record which causes the resolution to fail in tools like curl, python, etc. The response lengths are 422 bytes vs 478 bytes respectively, so the response isn't getting truncated (and the truncated flag is not set)

tl;dr: Is there a setting that we can change to resolve this, or are we looking at upgrading our cluster to a newer version to resolve this?

[chrisohaver]

There was a truncation bug in 1.6.6 where the forward plugin would unset the truncation bit of a truncated response from an upstream before sending the response to the client. It was fixed upstream, but it didn't make it into CoreDNS until 1.8.0. So that doesn't explain why it seems to work in 1.7.1.

[codyharris-h2o-ai]

@chrisohaver - I did read about one of the truncated issues via https://www.netmeister.org/blog/dns-tcpdump.html and related CNAME issues in #2492, however my understanding (which could totally be wrong!) was that this occurs when the packet is larger than 512 bytes, and the "successful" path sees a response packet size 478 bytes. While this is large, I wouldn't expect there to be any sort of truncation, unless it's exiting early because something /thinks/ it'll get truncated? I'm not sure.

[chrisohaver]

Capturing the exchange between CoreDNS and the upstream may shed better light on whats going on.
e.g. Is CoreDNS receiving an incomplete response from the upstream? If so, is there some difference in the query that CoreDNS sends to the upstream that could be causing that?

[codyharris-h2o-ai]

I captured a tcpdump on the node and I'm seeing what you describe: The upstream server is setting the truncated bit, and that is not properly communicated back to the origin client.

Thank you so much for taking the time to talk me through this!!!