DNS requests to systems serviced with dnsmasq fails with timeout

[freddyblue]

Describe the bug

nslookup vc-mgmt-a.site-a.vcf.lab
=>
Server: 10.1.1.1
Address: 10.1.1.1#53

Name: vc-mgmt-a.site-a.vcf.lab
Address: 10.1.1.10
;; communications error to 10.1.1.1#53: timed out
;; communications error to 10.1.1.1#53: timed out
;; communications error to 10.1.1.1#53: timed out
;; Got SERVFAIL reply from "ip upstream dns server"
** server can't find vc-mgmt-a.site-a.vcf.lab: SERVFAIL

Reproduction steps

1. dns configuration of holo-router with ip of upstream dns server during installation
2. ping to external dns server possible from holo-router
3. nslookup of systems being serviced by dnsmasq fails with timeout (see above "describe the bug")
   ...

Expected behavior

nslookup vc-mgmt-a.site-a.vcf.lab works as expected with no timeout errors

Additional context

workaround: kubectl edit cm dnsmasq => setting server="ip upstream dns server" to server="empty" => k get pod => k delete pod dnsmasq-deployment-string fixes the problem, but DNS to external systems not possible anymore
Questions ist, why forwarding for internal system DNS requests are forwarded to external DNS?

[knikhilm]

Are you facing any connectivity issues because of this? Can you check the resolution using dig? I see nslookup returns a response and then times out. It could be because nslookup tries to query both A (IPv4) and AAAA (IPv6) records, and it would fail for IPv6. Does the upstream DNS server that you are using for HoloRouter serve both IPv4 and IPv6 domains?

The details of your workaround are not clear - you are changing the server IP in DNSMASQ configuration to something - but it isn't clear to what. What are you modifying the server parameter to?

One thing you can try is modifying the upstream DNS server to any public IPv4 DNS server and see if that resolves the issue.

[freddyblue]

Are you facing any connectivity issues because of this?

yes, deployment of MD fails, when sddc tries to create certificate for vCenter, because vCenter-FQDN can't be resolved. E.g. ping router-a.site-a.vcf.lab also fails

Can you check the resolution using dig?

dig router-a.site-a.vcf.lab works without problem

I see nslookup returns a response and then times out. It could be because nslookup tries to query both A (IPv4) and AAAA (IPv6) records, and it would fail for IPv6. Does the upstream DNS server that you are using for HoloRouter serve both IPv4 and IPv6 domains?

Upstream DNS server is Microsoft AD, only serves IPv4. I'm not absolutely sure but to ask for AAAA dns entries you have to use nslookup with the option -query=AAAA explicitly.

The details of your workaround are not clear - you are changing the server IP in DNSMASQ configuration to something - but it isn't clear to what. What are you modifying the server parameter to?

removing the upstreams server behind the equals sign from config, only server= instead of server="ip upstream dns server"
apiVersion: v1
data:
dnsmasq.conf: |-
expand-hosts
dhcp-range=set:vlan10,10.1.1.220,10.1.1.239,255.255.255.0,24h
dhcp-option=tag:vlan10,option:router,10.1.1.1
dhcp-option=tag:vlan10,option:dns-server,10.1.1.1
dhcp-option=tag:vlan10,option:ntp-server,10.1.1.1
dhcp-range=set:vlan14,10.1.3.130,10.1.3.200,255.255.255.128,24h
dhcp-option=tag:vlan14,option:router,10.1.3.129
dhcp-option=tag:vlan14,option:dns-server,10.1.3.129
dhcp-option=tag:vlan14,option:ntp-server,10.1.3.129
dhcp-authoritative
expand-hosts
domain=site-a.vcf.lab,10.1.1.0/20
server=
hosts: |-
127.0.0.1 localhost dnsmasq

One thing you can try is modifying the upstream DNS server to any public IPv4 DNS server and see if that resolves the issue.

Test not possible, according to firewall-policy.

[knikhilm]

To understand better if AAAA queries are causing the issue - reconfigure the upstream server in the server= parameter of dnsmasq_configmap, apply it (kubectl apply -f /holodeck-runtime/dnsmasq/dnsmasq_configmap.yaml) and restart the dnsmasq deployment (kubectl delete -f /holodeck-runtime/dnsmasq/dnsmasq_deployment.yaml and kubectl apply -f /holodeck-runtime/dnsmasq/dnsmasq_deployment.yaml).

Then run nslookup -type=A router.site-a.vcf.lab & nslookup -type=AAAA router.site-a.vcf.lab to verify if it times out even when querying just A records

[freddyblue]

To understand better if AAAA queries are causing the issue - reconfigure the upstream server in the server= parameter of dnsmasq_configmap, apply it (kubectl apply -f /holodeck-runtime/dnsmasq/dnsmasq_configmap.yaml) and restart the dnsmasq deployment (kubectl delete -f /holodeck-runtime/dnsmasq/dnsmasq_deployment.yaml and kubectl apply -f /holodeck-runtime/dnsmasq/dnsmasq_deployment.yaml).

Then run nslookup -type=A router.site-a.vcf.lab & nslookup -type=AAAA router.site-a.vcf.lab to verify if it times out even when querying just A records

You are right, AAAA queries are causing the issue. nslookup -type=A router.site-a.vcf.lab is ok, no timeout and nslookup -type=AAAA router.site-a.vcf.lab times out

[freddyblue]

do you have any updates?

[knikhilm]

I think the root cause is that we have disabled IPv6 on HoloRouter and so the DNSMASQ doesn't respond to AAAA queries. But since systemd-resolved is querying AAAA records, it is being sent to upstream server which is not handling AAAA servers well. So it's a mixture of two problems.

One way to resolve this is to configure DNSMASQ to filter the AAAA queries. That should make it track only A queries and not wait for the AAAA queries. To do that,

1. Log into HoloRouter and enter Powershell - pwsh
2. Run the following command -
   $dnsmasq_config = Get-Content -Path /holodeck-runtime/dnsmasq/dnsmasq_configmap.yaml | ConvertFrom-Yaml; $dnsmasq_config.data."dnsmasq.conf" += "`nfilter-AAAA"; $dnsmasq_config | ConvertTo-Yaml | Set-Content -Path /holodeck-runtime/dnsmasq/dnsmasq_configmap.yaml; kubectl apply -f /holodeck-runtime/dnsmasq/dnsmasq_configmap.yaml; kubectl delete -f /holodeck-runtime/dnsmasq/dnsmasq_deployment.yaml; kubectl apply -f /holodeck-runtime/dnsmasq/dnsmasq_deployment.yaml

Once you do that, check DNS resolution if you still experience timeouts or it's waiting for responses to AAAA queries.

[freddyblue]

thanks a lot - that fixed the problem

[knikhilm]

Thank you for the confirmation.

I will add this to the DNSMASQ configmap in the next patch release.

[knikhilm]

This is now fixed with the release of Holodeck 9.0.1 (#57). Thank you @freddyblue for reporting this issue.