[Mac] K3s DNS server

[dee-kryvenko]

This is dependent on #699, also see k3s-io/k3s#4087 for more context.

The use case is that people use VPN clients like AnyConnect that does Split DNS. I understand Rancher VM is different from default Lima Ubuntu VM, but in Ubuntu dns settings from lima.yaml ends up in systemd-resolved. It can be fed to K3s at installation with url -sfL https://get.k3s.io | sh -s - --resolv-conf /run/systemd/resolve/resolv.conf.

I am not sure what the mechanism Rancher VM is using for DNS - it is probably not systemd-resolved, but the point is - once #699 is solved - we need a way to pass DNS configuration down to the coredns somehow.

This seems to be working out of the box in WSL2 - so it's only a Mac issue.

[jandubois]

I am not sure what the mechanism Rancher VM is using for DNS - it is probably not systemd-resolved

This is just configured via /etc/resolv.conf (which is written by udhcpc). It is just using Lima features to pick DNS settings, so it will add either the name servers configured for en0 on the host, or the servers listed under dns in lima.yaml (it will ignore the en0 name servers if the lima.yaml list isn't empty).

This seems to be working out of the box in WSL2 - so it's only a Mac issue.

Have you been testing this with RD 0.5.0, which was just released yesterday? The 0.4.1 release includes an older version of Lima that doesn't support DNS configuration.

[dee-kryvenko]

This is just configured via /etc/resolv.conf

Good to know - this is where coredns is going to look at, so this might be working out of the box after #699 is fixed. I still think RD should make K3s somewhat configurable by the user - but I don't have any other use case for that at the moment.

Have you been testing this with RD 0.5.0, which was just released yesterday? The 0.4.1 release includes an older version of Lima that doesn't support DNS configuration.

I actually tested it with v0.4.1 and it worked. I think RD VM through some WSL2 magic was automatically able to pick up my VPN connection and DNS on it. I'm not sure how but any WSL2 VM works with VPN and split DNS out of the box, so I didn't had to configure anything in the first place.

[jandubois]

This is just configured via /etc/resolv.conf

Good to know - this is where coredns is going to look at, so this might be working out of the box after #699 is fixed. I still think RD should make K3s somewhat configurable by the user - but I don't have any other use case for that at the moment.

I don't know what you mean by after #699 is fixed here. You should be able to set the dns entries in lima.yaml, and it should work right now (with 0.5.0 only). I assume you mean "when there is UI support for setting dns".

Have you been testing this with RD 0.5.0, which was just released yesterday? The 0.4.1 release includes an older version of Lima that doesn't support DNS configuration.

I actually tested it with v0.4.1 and it worked. I think RD VM through some WSL2 magic was automatically able to pick up my VPN connection and DNS on it. I'm not sure how but any WSL2 VM works with VPN and split DNS out of the box, so I didn't had to configure anything in the first place.

I thought this issue was only about macOS (the summary starts with [Mac]). Only the macOS version (and the upcoming Linux version) are using Lima, so all the stuff about lima.yaml is not relevant to Windows, which uses WSL2, as you noted.

So: if you are using RD 0.5.0 on macOS and set the dns entries in the lima.yaml file, then this should already work for you. It will also set the DNS settings on each new start, as it doesn't use cloud-init, but a custom script to process the data.

[dee-kryvenko]

I don't know what you mean by after #699 is fixed here. You should be able to set the dns entries in lima.yaml, and it should work right now (with 0.5.0 only). I assume you mean "when there is UI support for setting dns".

Not exactly - RD does not give me an option to edit initial config and because changing dns on pre-existing config did't had any effect due to lima-vm/lima#273 (which you just fixed but it's not released just yet) and because I can't find anything like /etc/netplan/50-cloud-init.yaml in RD VM - I have no way to test it. I can see with LIMA_HOME=~/Library/Application\ Support/rancher-desktop/lima limactl shell 0 cat /etc/resolv.conf that it does have my private DNS in it, and coredns setup is pretty much default:

data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {
          ttl 60
          reload 15s
          fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
  NodeHosts: |
    192.168.5.15 lima-rancher-desktop

But coredns does not resolve names known by my private DNS. Meaning it either does not use /etc/resolv.conf or something else going on.

I thought this issue was only about macOS

Correct, I was just trying to say that I tested it and similar issue does not exists for Windows.

[jandubois]

Not exactly - RD does not give me an option to edit initial config and because changing dns on pre-existing config did't had any effect due to lima-vm/lima#273 (which you just fixed but it's not released just yet)

This is only true for systemd-based images. The Alpine image used by RD will process the network settings during each start, so /etc/resolv.conf should still be correct.

I suspect the issue is still that coredns will forward the request to /etc/resolv.conf, which means it picks one of the nameservers at random. When it gets a response that the name was not found, it will not retry the next server, but take the result as authoritative.

So if things work would still to some degree depend on luck, if coredns picks the right nameserver, or not.

I'm wondering now if we should actually remove the qemu nameserver from the list whenever dns addresses are passed in explicitly, either from the en0 settings, or via dns in lima.yaml.

[mattfarina]

While I'm not entirely sure how we'll solve this, I would like a solution in the 0.7 release. I don't think we have time prior to that but if we do it would be great to get it in early.

[dee-kryvenko]

You are right @jandubois - I just tested with LIMA_HOME=~/Library/Application\ Support/rancher-desktop/lima limactl shell 0 sudo vi /etc/resolv.conf removed quemu DNS and now it works. For some reason I've been lucky to always hit in 33% I guess...

[jandubois]

I'm now researching a different approach for DNS in Lima: run a DNS server in a goroutine inside the hostagent and forward it to the VM. The host agent DNS would then look up type A requests by doing a local lookup instead of forwarding to a name server.

That way we should always get the name resolution from the host, so this would also deal with conditional forwards correctly by default. So any VPN changes would be visible right away without even restarting the VM.

Inside the guest the only nameserver would be the one forwarded from the host; no alternates (similar to how systemd-resolved configures 127.0.0.53).

Still needs more research, but feels promising.

[jandubois]

Fixed in Lima, will be in next RD release.

[gaktive]

To note, this issue is marked done based on prior troubleshooting efforts but the new #770 picks up where we need to go next.