DNS not working with some resolve.conf files at startup

[sywesk]

What Happened?

When I start minikube at home or using wifi sharing from my phone, everything goes well. But when I start minikube at work, i cannot build images using eval $(minikube docker-env). Funny thing, after starting the container, I can disconnect from my phone's wifi and switch back to the work wifi, and it continues to work perfectly.

Here's the error I get from docker: ERROR: failed to solve: alpine:latest: failed to do request: Head "https://registry-1.docker.io/v2/library/alpine/manifests/latest": dial tcp: lookup registry-1.docker.io on 0.0.0.0:53: read udp 127.0.0.1:47353->127.0.0.1:53: read: connection refused

Context

Ubuntu: 23.10, using systemd-resolved.

Pro WiFi: We are a small company and using our ISP-provided gateway. We haven't changed anything from the default settings (btw there aren't many settings really).

Diagnosis

resolv.conf

Here's the "usual" /etc/resolv.conf (within the minikube container):

nameserver  192.168.49.1
option ndots:0

When connected to my work wifi:

search box.freepro.com
nameserver 0.0.0.0
options ndots:0

Replacing resolv.conf

However, it seems that the issue is deeper as it seems. As the image is not using systemd-resolved, I tried replacing the /etc/resolv.conf file with the contents of the working one. But it still cannot use the host to do DNS:

root@minikube:/# echo "nameserver 192.168.49.1" > /etc/resolv.conf
root@minikube:/# echo "options ndots:0" >> /etc/resolv.conf
root@minikube:/# dig google.com
;; communications error to 192.168.49.1#53: connection refused
;; communications error to 192.168.49.1#53: connection refused
;; communications error to 192.168.49.1#53: connection refused

; <<>> DiG 9.18.18-0ubuntu0.22.04.1-Ubuntu <<>> google.com
;; global options: +cmd
;; no servers could be reached

Here's my ip a output on the host (i've removed my wlan/lo/tailscale devices):

[...]
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:3e:4d:2c brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
[...]
6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:20:50:27:f2 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:20ff:fe50:27f2/64 scope link 
       valid_lft forever preferred_lft forever
17: br-e6764ceb0c38: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:7d:20:9f:ea brd ff:ff:ff:ff:ff:ff
    inet 192.168.49.1/24 brd 192.168.49.255 scope global br-e6764ceb0c38
       valid_lft forever preferred_lft forever
    inet6 fe80::42:7dff:fe20:9fea/64 scope link 
       valid_lft forever preferred_lft forever
23: vethfef5da4@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-e6764ceb0c38 state UP group default 
    link/ether 3a:34:84:84:06:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::3834:84ff:fe84:6ee/64 scope link 
       valid_lft forever preferred_lft forever

Confirm the host IP

Executing ip a on minikube:

root@minikube:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:16:e3:e2:5b brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
3: bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3a:a7:15:d3:54:6f brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.1/16 brd 10.244.255.255 scope global bridge
       valid_lft forever preferred_lft forever
4: veth098c6937@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master bridge state UP group default 
    link/ether b2:0b:07:e0:02:9c brd ff:ff:ff:ff:ff:ff link-netnsid 1
22: eth0@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:c0:a8:31:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.49.2/24 brd 192.168.49.255 scope global eth0
       valid_lft forever preferred_lft forever

Pod IP address is 192.168.49.2. Connecting to it from with a wrong password:

$ ssh 192.168.49.2
The authenticity of host '192.168.49.2 (192.168.49.2)' can't be established.
ED25519 key fingerprint is SHA256:/GNLGkp0oz8keBudJAQktFdHwAE0H4AT+VPjBotw3Ik.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.49.2' (ED25519) to the list of known hosts.
xxx@192.168.49.2's password: 
Permission denied, please try again.

Looking at the container's journal:

Jan 17 11:04:50 minikube sshd[13743]: error: Could not get shadow information for NOUSER
Jan 17 11:04:50 minikube sshd[13743]: Failed password for invalid user xxx from 192.168.49.1 port 57738 ssh2

So this confirms that the host's ip address is indeed 192.168.49.1.

Host resolv.conf

Here's the host resolv.conf when connected to the pro wifi:

nameserver 192.168.10.254
nameserver 2a05:6e02:10e4:5410::1
search box.freepro.com

And when connected to my phone:

nameserver 192.168.150.67
search .

Conclusion

I have a gut feeling that my resolv.conf, when connected to the pro wifi, cause some issues with the initialization of the DNS when starting the minikube container.

Attach the log file

log.txt

Operating System

Ubuntu

Driver

Docker

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten