Cisco AnyConnect VPN with DNS

[dee-kryvenko]

AnyConnect will setup utunN interface as well as DNS resolvers I could see with scutil --dns.

This doesn't seem to be propagated into VMs. Is there a way to make it work, do I miss some configuration or it's just not supported?

I also tried to put some known DNS servers to my ~/.lima/default/lima.yaml:

dns:
- x.x.x.x
- y.y.y.y

And then limactl stop && limactl start - but my VM still doesn't have it in the /etc/resolv.conf neither am I able to use curl to get from a private host meaning it is not being injected to the VM in any other way. What am I missing?

[jandubois]

This doesn't seem to be propagated into VMs. Is there a way to make it work, do I miss some configuration or it's just not supported?

You have to do this manually via the dns settings in lima.yaml. The only name servers that should be propagated automatically are the ones from the host en0 interface.

my VM still doesn't have it in the /etc/resolv.conf

On a systemd setup they will not be added to /etc/resolv.conf, but should still be configured for the resolved service. You can check with resolvectl status; it should show additional servers beyond 192.168.19.53, e.g.

$ resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.19.53
       DNS Servers: 192.168.19.53 8.8.8.8 1.1.1.1

neither am I able to use curl to get from a private host

I suspect the issue is that the VM cannot route to the additional name servers because they too are on the VPN. I don't have an idea right now how to make that work, unfortunately.

[dee-kryvenko]

I figured the latter question out - modifying list of DNS servers on the existing VM is not having any effect (it only works for VM creation), I had to modify /etc/netplan/50-cloud-init.yaml and now it picked it up after a restart. My DNS server are indeed private but somehow VM can perfectly access it as well as other private hosts behind VPN. This seems to be a DNS only issue.

However, that doesn't solve it entirely. Being able to provide DNS servers is not enough - AnyConnect as well as other VPN clients are using conditional resolvers. There are different DNS servers for different search domains. For example my scutil --dns looking the following way (stripped out private details from the output):

resolver #1
  search domain[0] : enterprise.com
  ...
  search domain[19] : enterprise19.com
  nameserver[0] : 8.8.8.8
  nameserver[1] : 8.8.4.4
  if_index : 6 (en0)
  flags    : Request A records
  reach    : 0x00000002 (Reachable)

resolver #2
  domain   : enterprise2.tech
  nameserver[0] : x.x.x.x
  nameserver[1] : y.y.y.y
  flags    : Supplemental, Request A records
  reach    : 0x00000002 (Reachable)
  order    : 1

resolver #17
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #23
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #24
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #25
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #26
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #27
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  nameserver[0] : 8.8.8.8
  nameserver[1] : 8.8.4.4
  if_index : 6 (en0)
  flags    : Scoped, Request A records
  reach    : 0x00000002 (Reachable)

resolver #2
  search domain[0] : enterprise.tech
  ...
  search domain[19] : enterprise19.tech
  nameserver[0] : x.x.x.x
  nameserver[1] : y.y.y.y
  if_index : 17 (utun2)
  flags    : Scoped, Request A records
  reach    : 0x00000002 (Reachable)
  order    : 1

These x.x.x.x and y.y.y.y DNS servers might not know about anything outside the VPN so I can't just use them as catch-all DNS. Adding both public and private DNS servers to /etc/netplan/50-cloud-init.yaml doesn't seem to work, probably because they are expected to be HA replicas instead of actually different authorities that could return different results.

[dee-kryvenko]

Upon further research - split DNS as well as systemd-resolved is not supported by underlying coredns anyway. I've found out that lima.yaml (and /etc/netplan/50-cloud-init.yaml) DNS ends up in the /run/systemd/resolve/resolv.conf so for K3s one could use --resolv-conf /run/systemd/resolve/resolv.conf to make it work with the corporate VPN.

It would be nice to fix Lima to refresh DNS from lima.yaml on restart so the user don't have to know about /etc/netplan/50-cloud-init.yaml but otherwise this ticket can be closed.

[jandubois]

It would be nice to fix Lima to refresh DNS from lima.yaml on restart so the user don't have to know about /etc/netplan/50-cloud-init.yaml but otherwise this ticket can be closed.

I was just about to say that Lima does refresh the DNS on each restart, but when I just tested it, the new servers were not configured.

Unfortunately this seems to be an issue with cloud-init, as we do pass in the new nameservers:

jan@lima-default:/Users/jan$ sudo cat /mnt/lima-cidata/network-config
version: 2
ethernets:
  eth0:
    match:
      macaddress: '52:55:55:e9:a1:b4'
    dhcp4: true
    set-name: eth0
    nameservers:
      addresses:
      - 1.1.1.1
      - 1.0.0.1

This seems to be processed only during first-boot.

[jandubois]

This seems to be processed only during first-boot.

Yes, this is intentional, as far as cloud-init is concerned, but I found a way to work around it, by changing the instance id on every reboot, so it always looks like a fresh install to cloud-init: #270

[jandubois]

Upon further research - split DNS as well as systemd-resolved is not supported by underlying coredns anyway. I've found out that lima.yaml (and /etc/netplan/50-cloud-init.yaml) DNS ends up in the /run/systemd/resolve/resolv.conf so for K3s one could use --resolv-conf /run/systemd/resolve/resolv.conf to make it work with the corporate VPN.

This is the part I still don't understand:

If coredns doesn't understand the split setup where each name server only knows about a subset of domains, how does it help to configure all with systemd. coredns would still forward to a randomly picked server from /run/systemd/resolve/resolv.conf, and you can either get lucky, or you could not.

I think leaving 192.168.5.3 in the list of name servers is a mistake because in case you explicitly supply VPN name servers, the qemu nameserver will not be able to resolve the name, so you have a 33% chance of failure.

But removing this address will still not make the split-dns scenario work. I'm actually not sure if it is even possible to make it work. This is turning into a rabbit hole...

I'm re-opening this issue, because I think we need to explore this further.

[jandubois]

I'm actually not sure if it is even possible to make it work.

On further thoughts, I think it should be possible: we could specify the search domains together with the nameserver addresses, and systemd-resolved should route queries appropriately. And k8s would use 127.0.0.53 from /etc/resolv.conf.

For systems without systemd-resolved we could run dnsmasq and configure it for conditional forwarding.

This is now getting quite a bit more complex, so it may take a while before I can look into it.

[dee-kryvenko]

I just tried it and it works like a charm! Thanks @jandubois.

[jandubois]

I just tried it and it works like a charm!

Thanks for confirming!

I'm happy with the result; I love it when stuff works correctly without requiring any configuration. It is good to be able to override settings, but it is even better if you don't need to.

[nakamorichi]

So what was the solution to this? The only way I'm able to currently get network working in lima behind VPN is to copy /etc/resolv.conf from host to lima instance, but it will be rewritten with every restart..

[jandubois]

@Kitanotori Can you provide some more details on what exactly is not working?

• Which lima version are you using?
• Do you just need the nameserver, or is it the search domains that are missing?
• Does it fail to resolve any names when the VPN is connected, or only names provided by the VPN?
• Where is it failing: from regular shell commands, or when nerdctl is pulling images, or from within k8s?

Lima now provides a DNS server that resolves names on the host, so generally this should work automatically without any configuration. If this is not the case, then we would like to understand how and why it fails, so we can improve the implementation.

Just today I learned that DNS does not work properly when the names are using the *.local domain (which should really only be used for mDNS).

If all you need to do is to specify the DNS server inside /etc/resolv.conf, then you could pass this in via your lima.yaml file:

dns:
- 1.1.1.1
- 8.8.8.8

There is no way to specify a search domain though.

But even if this fixes your problem, please provide the information I requested above to help us improve Lima!

[nakamorichi]

@jandubois

Host /etc/resolv.conf looks like this:

search intra.aaa.bbb.com
nameserver 10.0.0.1
nameserver 10.0.0.2
nameserver 10.0.0.3

lima instance /etc/resolve.conf:

nameserver 127.0.0.53
options <not sure if these are relevant?>
search .

I tried with exact same resolv.conf as host and one with only the nameservers of the host, and both seemed to work. I guess my issue is that the config from ~/.lima/default/lima.yaml is not reflected to lima instance's resolve.conf..?

Just today I learned that DNS does not work properly when the names are using the *.local domain (which should really only be used for mDNS).

We use such domains, but those don't seem to be issue if using host's nameservers.

[nakamorichi]

Also, the nameservers are not reflected to resolve.conf even when specifying the nameservers during lima instance creation. So, how can I get the configuration persist restarts?

[jandubois]

They are configured in systemd-resolved, which is what 127.0.0.53 points to.

$ grep ^useHost ~/.lima/default/lima.yaml
useHostResolver: false

$ grep -A 2 ^dns ~/.lima/default/lima.yaml
dns:
- 1.1.1.1
- 8.8.8.8

$ lima resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 1.1.1.1
       DNS Servers: 1.1.1.1 8.8.8.8 192.168.5.3

Looks like the host resolver (192.168.5.3) is still included in the nameserver list, but as long as the primary server is available it should make no difference.

[nakamorichi]

@jandubois I'm not familiar with systemd-resolved. Does it mean that currently there's no way to persist correct settings at lima instance?

[jandubois]

@Kitanotori As I've shown just above, the DNS servers configured via lima.yaml are used by systemd-resolved as name servers, and the systemd resolver is exposed on 127.0.0.53. That address is stored in /etc/resolv.conf, so applications not linking against glibc (e.g. Go programs compiled without CGO) can still take advantage of the extended resolver features.