new lima breaks DNS resolution with docker

[brokenjacobs]

There is a fix in the new lima for DNS resolution:
lima-vm/lima#281

And it appears that what happens is the iptables rule for the nat redirect hits after the docker nat rule:

iptables -t nat -L -v --line-numbers
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        2    88 DOCKER     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 DOCKER     all  --  any    any     anywhere            !localhost/8          ADDRTYPE match dst-type LOCAL
2       26  1963 DNAT       udp  --  any    any     anywhere             192.168.5.3          udp dpt:domain to:192.168.5.2:52464

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1       14  1073 MASQUERADE  all  --  any    !docker0  172.17.0.0/16        anywhere

Chain DOCKER (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 RETURN     all  --  docker0 any     anywhere             anywhere

You can test this by running dig inside a container pointed at the port in the nat rule, that works. But trying to hit 192.168.5.3.

I'm not sure where to fix this, but it probably just means that the nat rule from lima needs to run before docker installs it's rules. Trouble is I can't figure out where that is happening.

[abiosoft]

The new lima feature is optional and can be disabled https://github.com/lima-vm/lima/blob/master/pkg/limayaml/default.yaml#L197. You can modify colima's source and add that config line to see if that fixes it as a temporary workaround.

I will try to reproduce this and include a fix in the imminent v0.2.0.

[abiosoft]

@brokenjacobs what is the expected behaviour and what is broken? DNS resolutions are working as expected for me.

Can you kindly elaborate as I am not a network expert. Are all DNS resolutions failing for you, or the nat rules are affecting specific scenarios?

Thanks.

[brokenjacobs]

So what I did was modify colima to include that config line, and things broke at that point. Reading the lima docs, when qemu starts the vm, it provides a dns resolver at 192.168.5.3 udp port 53. However, this resolver does not support the native mac "host resolver", it just looks at the resolv.conf in osx. Consequently any dns names resolved via a VPN, or using mdns, will not work inside the lima VM. So when you set useHostResolver in the config, lima does two things.
https://github.com/lima-vm/lima/blob/master/docs/network.md#dns-19216853

1. It sets up the hostagent to run its own resolver, running on whatever port ssh is bound to in the vm, on UDP instead of TCP. This hostagent resolver IS able to resolve names using the osx name resolution.
2. It configures an iptables nat rule that forwards any dns requests to 192.168.5.3 port 53 udp to instead fo to this udp port where the hostagent does the resolution instead.

For whatever reason, this rule is NOT triggering for docker originated traffic, so the dns resolver being used is the QEMU one, not the hostagent resolver.

[abiosoft]

Apparently, a fix for this is in the works. lima-vm/lima#319.

[brokenjacobs]

Looks like an upstream issue then. Thanks for checking! Closing.