systemd service with JoinsNamespaceOf= does not use separate resolv.conf for that netns

[huyz-git]

$ ls -lah /proc/1/ns | grep -E '(mnt|net)'
lrwxrwxrwx 1 root root 0 Jul 28 12:24 mnt -> mnt:[4026531841]
lrwxrwxrwx 1 root root 0 Jul 27 22:07 net -> net:[4026531840]

I created a netns called warp use netns@warp.service, and create /etc/netns/warp/resolv.conf, then:

$ ip netns exec warp ls -lah /proc/self/ns | grep -E '(mnt|net)'
lrwxrwxrwx 1 root root 0 Jul 28 12:30 mnt -> mnt:[4026532461]
lrwxrwxrwx 1 root root 0 Jul 28 12:30 net -> net:[4026532520]

Processes run by ip netns exec warp use separate netns and mount namespace, and it uses the separate resolv.conf

I also run a systemd service (warp-svc) with JoinsNamespaceOf=netns@warp.service, however:

$ ls -lah /proc/$(pidof warp-svc)/ns | grep -E '(mnt|net)'
lrwxrwxrwx 1 root root 0 Jul 28 12:25 mnt -> mnt:[4026531841]
lrwxrwxrwx 1 root root 0 Jul 27 23:06 net -> net:[4026532520]

It only changes the netns, the mount namespace remains the same as pid 1, and therefore this process does not use the separate resolv.conf

[huyz-git]

I found that the mount namespace of the netns is actually created by ip netns exec command, and is different with each execution:

$ ip netns exec warp sleep 100 &
$ ip netns exec warp sleep 100 &
$ ip netns exec warp sleep 100 &
$ ip netns exec warp sleep 100 &
$ ip netns exec warp sleep 100 &
$ lsns | grep sleep
4026532445 mnt         1  8372 root             sleep 100
4026532446 mnt         1  8373 root             sleep 100
4026532447 mnt         1  8374 root             sleep 100
4026532448 mnt         1  8375 root             sleep 100
4026532449 mnt         1  8376 root             sleep 100

So to let systemd service use /etc/netns/<ns>/resolv.conf I need to setup the mount namespace and bind the file inside the systemd unit. However, if I just use PrivateMounts=yes to isolated the mount namespace, it also isolates /var/run/netns and therefore the systemd unit cannot bind to the netns.

[almightiest]

I never had this issue with Ubuntu 20.04 but I recently installed 22.04 and after re-setting all of this up I am encountering this exact issue I believe.

My netns@vpn.service vpn namespace is using the default /etc/resolv.conf even though I have a custom /etc/netns/vpn/resolv.conf file present.

My netns.conf file inside my service directory is:

[Unit]
BindsTo=netns@vpn.service
After=netns@vpn.service
JoinsNamespaceOf=netns@vpn.service

[Service]
PrivateNetwork=yes

A test service file that uses the netns.conf looks like this:

[Unit]
Description=Test VPN dns
After=netns@vpn.service
Requires=netns@vpn.service

[Service]
ExecStart=/usr/bin/nslookup www.google.com
ExecStart=/usr/bin/traceroute www.google.com
Type=oneshot
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

If I use /usr/bin/env chnetns vpn nslookup www.google.com on its own it uses the vpn custom resolv.conf - but if I add that same command to a systemd service it uses the main /etc/resolv.conf which won't work with my setup since these are separate physical networks.

I haven't found the best way to fix this yet - could it be something different in 22.04 (like systemd-resolved) that's causing this to not work?

[almightiest]

I ended up having to do a custom script to create the netns, get it attached to a nic and get it online, and then added a service override netns.conf to each of my units:

[Service]
NetworkNamespacePath=/var/run/netns/vpn
BindReadOnlyPaths=/etc/netns/vpn/resolv.conf:/etc/resolv.conf:norbind