Unit in network namespace uses default namespace resolv.conf instead of the bind mount /etc/netns/<namespace>/resolv.conf #28694

rdbisme · 2023-08-05T19:46:08Z

systemd version the issue has been seen with

254

Used distribution

Archlinux

Linux kernel version used

6.1.39-3-lts

CPU architectures issue was seen on

x86_64

Component

systemd

Expected behaviour you didn't see

Before the latest update, the process (transmission torrent client) started in a dedicated namespace was able to resolve domains successfully

Unexpected behaviour you saw

Now I get errors about transmission not being able to resolve the tracker domains.

Steps to reproduce the problem

I'm not sure 100% is systemd related. Also if I enter the namespace doing sudo ip netns exec vpn su $USER -, domain are correctly resolved (e.g. drill google.com returns correctly).

Transmission Arch unit

[Unit]
Description=Transmission BitTorrent Daemon
Wants=network-online.target
After=network-online.target

[Service]
User=transmission
Type=notify
ExecStart=/usr/bin/transmission-daemon -f --log-level=error
ExecReload=/bin/kill -s HUP $MAINPID
NoNewPrivileges=true
MemoryDenyWriteExecute=true
ProtectSystem=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Drop-in

[Unit]
After=vpn.service
BindsTo=vpn.service

[Service]
NetworkNamespacePath=/run/netns/vpn
Environment="TR_CURL_VERBOSE=1"
ExecStart=
ExecStart=/usr/bin/transmission-daemon -f --log-level=debug

vpn.service

[Unit]
Description=Start Wireguard VPN with a random peer
Wants=network-online.target nss-lookup.target
Requires=wireguard.service
After=network-online.target nss-lookup.target wireguard.service
PartOf=wireguard.service netns@vpn.service
JoinsNamespaceOf=netns@vpn.service

[Service]
Type=oneshot
RemainAfterExit=true

ExecStartPre=-/usr/bin/env ip -n vpn addr flush dev wg0
ExecStartPre=-/usr/bin/env ip -n vpn link set down wg0
ExecStartPre=-/usr/bin/env ip netns exec vpn /usr/local/bin/teardown_wg

# Set wg
ExecStart=/usr/bin/env ip netns exec vpn /usr/local/bin/setup_wg

# Set interface up
ExecStart=/usr/bin/env ip -n vpn link set up dev wg0

# Set default route
ExecStart=/usr/bin/env ip -n vpn route add default dev wg0

[Install]
WantedBy=multi-user.target
WantedBy=network-online.target

wireguard.service

[Unit]
Description=Wireguard interface in a dedicated namespace
Requires=netns@vpn.service
Wants=network-online.target nss-lookup.target
After=network-online.target nss-lookup.target netns@vpn.service
PartOf=netns@vpn.service
JoinsNamespaceOf=netns@vpn.service

[Service]
Type=oneshot
RemainAfterExit=true

ExecStartPre=-/usr/bin/env ip -n vpn link delete wg0

# Create wg0 interface in normal namespace
ExecStart=/usr/bin/env ip link add dev wg0 type wireguard

# Move wg0 interface to the vpn namespace
ExecStart=/usr/bin/env ip link set wg0 netns vpn

# Stop process
ExecStop=/usr/bin/env ip -n vpn link delete wg0

[Install]
WantedBy=multi-user.target
WantedBy=network-online.target

netns@.service

[Unit]
Description=%I Namespace wrapper
After=network-online.target nss-lookup.target
Wants=network-online.target nss-lookup.target

[Service]
Type=oneshot
RemainAfterExit=true
PrivateNetwork=true
PrivateMounts=false

ExecStartPre=-/usr/bin/env ip netns delete %i

# Add vpn dedicated namespace
ExecStart=/usr/bin/env ip netns add %i

# Pause
ExecStart=/usr/bin/env echo %i

# Set loopback device up
ExecStart=/usr/bin/env ip -n %i link set dev lo up

# Stop process
ExecStop=/usr/bin/env ip netns delete %i

[Install]
WantedBy=multi-user.target
WantedBy=network-online.target

Additional program output to the terminal or log subsystem illustrating the issue

# Error from transmission:
Could not resolve host: <tracker URL>

The text was updated successfully, but these errors were encountered:

rdbisme · 2023-08-05T21:33:16Z

After a tcpdump session, it seems that the transmission-daemon is using the default namespace /etc/resolv.conf and not the one mounted in the vpn namespace (/etc/netns/vpn/resolv.conf) (that I can successfully dump doing cat /etc/resolv.conf when inside the vpn namespace).

Why transmission-daemon is seeing the default namespace resolv.conf?

yuwata · 2023-08-05T21:59:25Z

Should be duplicate of #28686. Please use PrivateMounts=no.

rdbisme · 2023-08-05T22:02:15Z

@yuwata Setting PrivateMounts=no on the transmission service drop-in doesn't work. It still queries the main namespace DNS.

yuwata · 2023-08-05T22:57:39Z

No, PrivateMounts= should be set on vpn.service or so that have PrivateNetwork=.

BTW, what's the purpose of PriveteNetwork= in these services? It looks meaningless to me.

rdbisme · 2023-08-05T23:12:35Z

No, PrivateMounts= should be set on vpn.service or so that have PrivateNetwork=.

BTW, what's the purpose of PriveteNetwork= in these services? It looks meaningless to me.

Well, the description of the option is cryptic, but in my head that would have prevented any potential leak on the main network namespace interface.

Anyway, I've:

Set PrivateMounts=false on netns@.service -> ❌ transmission is still querying main namespace DNS (and not finding it obviously)
Set PrivateMounts=false on vpn.service -> ❌
Removing PrivateMounts and PrivateNetwork from netns@.service -> ❌

The configuraton I posted used to work flawlessly with systemd < 254.

yuwata · 2023-08-08T00:34:37Z

Hmm. Strictly speaking, we made several backward incompat changes on v254. But, still I cannot understand the motivation of several settings in the services you use.

PrivateNetwork=yes in netns@.service is mostly meaningless, though that's not a big matter, but you need to disable PrivateMounts= in the service if you want to use PrivateNetwork=yes.
JoinsNamespaceOf= in vpn.service and wireguard.service sound mostly misconfiguration for me. You should use NetworkNamespacePath=/run/netns/vpn instead, like transmission.service, and drop namespace options in ExecStart= or so. That is, e.g. ip -n vpn link delete wg0 -> ip link delete wg0.

Moreover, I do not understand the issue explained in the title:

Unit in network namespace uses default namespace resolv.conf instead of the bind mount /etc/netns//resolv.conf

Who manages the bind mount?? Is it done by vpn command?? If so, please provide script of config used in vpn command.
I recommend to use BindPaths= (or BindReadOnlyPaths= if it is static) to map the file.

rdbisme · 2023-08-09T09:57:49Z

Hello @yuwata, thanks for your feedback.

PrivateNetwork=yes in netns@.service is mostly meaningless, though that's not a big matter, but you need to disable PrivateMounts= in the service if you want to use PrivateNetwork=yes.

Indeed I removed PrivateNetwork=yes. I probably misunderstood the meaning of that option.

Who manages the bind mount?? Is it done by vpn command?? If so, please provide script of config used in vpn command.
I recommend to use BindPaths= (or BindReadOnlyPaths= if it is static) to map the file.

Well, if I create manually the netns, and then I do sudo ip netns exec vpn su $USER - to open a shell in that network namespace, then if I cat /etc/resolv.conf, I correctly see the same resolv.conf as the one in /etc/netns/resolv.conf. Before 254 update, it was indeed the case also for the vpn.service. But now it is clearly getting the /etc/resolv.conf from the default namespace. Shouldn't the fact that the service is set to run in a specific namespace imply the bind mount of /etc/netns/resolv.conf to /etc/resolv.conf? Or do I need now to explicitly specify it using BindPaths?

wireguard.service needs to run on the default namespace in order to create correctly the vpn interface to ensure that processes running in the vpn netns, will indeed only see the wg0 interface (i.e. the vpn one). Reference: https://www.wireguard.com/netns/#the-new-namespace-solution

rdbisme · 2023-08-09T10:00:59Z

wireguard.service needs to run on the default namespace in order to create correctly the vpn interface to ensure that processes running in the vpn netns, will indeed only see the wg0 interface (i.e. the vpn one). Reference: https://www.wireguard.com/netns/#the-new-namespace-solution

More specifically, in order to have a wg0 interface in the vpn network namespace (such that processes spawned in that network namespace can see only that interface, to prevent leaking) that can communicate with the external internet, first you need to create it in the default namespace (where your physical eth0/en0 interface are living), then you need to move it ip link set wg0 netns vpn in the target namespace after creation.

rdbisme · 2023-08-17T21:07:13Z

Following @yuwata suggestions I edited my units:

netns@.service

[Unit]
Description=%I Namespace wrapper
After=network-online.target nss-lookup.target
Wants=network-online.target nss-lookup.target

[Service]
Type=oneshot
RemainAfterExit=true

ExecStartPre=-/usr/bin/env ip netns delete %i

# Add vpn dedicated namespace
ExecStart=/usr/bin/env ip netns add %i

# Pause
ExecStart=/usr/bin/env echo %i

# Set loopback device up
ExecStart=/usr/bin/env ip -n %i link set dev lo up

# Stop process
ExecStop=/usr/bin/env ip netns delete %i

[Install]
WantedBy=multi-user.target
WantedBy=network-online.target

wireguard.service

[Unit]
Description=Wireguard interface in a dedicated namespace
Requires=netns@vpn.service
Wants=network-online.target nss-lookup.target
After=network-online.target nss-lookup.target netns@vpn.service
PartOf=netns@vpn.service

[Service]
Type=oneshot
RemainAfterExit=true

ExecStartPre=-/usr/bin/env ip -n vpn link delete wg0

# Create wg0 interface in normal namespace
ExecStart=/usr/bin/env ip link add dev wg0 type wireguard

# Move wg0 interface to the vpn namespace
ExecStart=/usr/bin/env ip link set wg0 netns vpn

# Stop process
ExecStop=/usr/bin/env ip -n vpn link delete wg0

[Install]
WantedBy=multi-user.target
WantedBy=network-online.target

vpn.service

[Unit]
Description=Start Wireguard VPN with a random peer
Wants=network-online.target nss-lookup.target
Requires=wireguard.service
After=network-online.target nss-lookup.target wireguard.service
PartOf=wireguard.service netns@vpn.service

[Service]
Type=oneshot
RemainAfterExit=true
NetworkNamespacePath=/run/netns/vpn
BindReadOnlyPaths=/etc/netns/vpn/resolv.conf:/etc/resolv.conf:norbin

ExecStartPre=-/usr/bin/env ip addr flush dev wg0
ExecStartPre=-/usr/bin/env ip vpn link set down wg0
ExecStartPre=-/usr/local/bin/teardown_wg

# Set wg
ExecStart=/usr/bin/env /usr/local/bin/setup_wg

# Set interface up
ExecStart=/usr/bin/env ip link set up dev wg0

# Set default route
ExecStart=/usr/bin/env ip route add default dev wg0

[Install]
WantedBy=multi-user.target
WantedBy=network-online.targe

This works! :)

isodude · 2023-10-12T22:46:22Z

@rdbisme it should be norbind right?

Also, what about
BindReadOnlyPaths=/etc/netns/vpn/resolv.conf:/run/systemd/resolve/resolv.conf:norbind ?

I think /etc/nsswitch.conf checks that file if there's hosts resolve in it.

rdbisme · 2023-10-13T09:07:40Z

@rdbisme it should be norbind right?

Also, what about BindReadOnlyPaths=/etc/netns/vpn/resolv.conf:/run/systemd/resolve/resolv.conf:norbind ?

I think /etc/nsswitch.conf checks that file if there's hosts resolve in it.

Mmmmh... In my case using /etc/resolv.conf works. Isn't /run/systemd/resolve/resolve.conf used by systemd-resolved (that I'm not using)?

isodude · 2023-10-13T09:11:37Z

Well, for others I guess :)

…

On Fri, 13 Oct 2023 at 11:07, Ruben Di Battista ***@***.***> wrote: @rdbisme <https://github.com/rdbisme> it should be norbind right? Also, what about BindReadOnlyPaths=/etc/netns/vpn/resolv.conf:/run/systemd/resolve/resolv.conf:norbind ? I think /etc/nsswitch.conf checks that file if there's hosts resolve in it. Mmmmh... In my case using /etc/resolv.conf works. Isn't /run/systemd/resolve/resolve.conf used by systemd-resolved (that I'm not using)? — Reply to this email directly, view it on GitHub <#28694 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAWI4MYFBTJUUAHXN44YYM3X7EAGNAVCNFSM6AAAAAA3FMZWWSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRRGE3TMOJUGM> . You are receiving this because you commented.Message ID: ***@***.***>

rdbisme added the bug 🐛 Programming errors, that need preferential fixing label Aug 5, 2023

github-actions bot added pid1 labels Aug 5, 2023

rdbisme changed the title ~~Unit in network namespace can't resolve DNS anymore~~ Unit in network namespace uses default namespace resolv.conf Aug 5, 2023

rdbisme changed the title ~~Unit in network namespace uses default namespace resolv.conf~~ Unit in network namespace uses default namespace resolv.conf instead of the bind mount /etc/netns/<namespace>/resolv.conf Aug 5, 2023

yuwata closed this as completed Aug 5, 2023

yuwata added duplicate not-a-bug and removed bug 🐛 Programming errors, that need preferential fixing labels Aug 5, 2023

yuwata reopened this Aug 8, 2023

yuwata added reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer and removed reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks labels Aug 8, 2023

rdbisme closed this as completed Aug 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unit in network namespace uses default namespace resolv.conf instead of the bind mount /etc/netns/<namespace>/resolv.conf #28694

Unit in network namespace uses default namespace resolv.conf instead of the bind mount /etc/netns/<namespace>/resolv.conf #28694

rdbisme commented Aug 5, 2023 •

edited

rdbisme commented Aug 5, 2023 •

edited

yuwata commented Aug 5, 2023

rdbisme commented Aug 5, 2023 •

edited

yuwata commented Aug 5, 2023

rdbisme commented Aug 5, 2023 •

edited

yuwata commented Aug 8, 2023 •

edited

rdbisme commented Aug 9, 2023 •

edited

rdbisme commented Aug 9, 2023 •

edited

rdbisme commented Aug 17, 2023 •

edited

isodude commented Oct 12, 2023

rdbisme commented Oct 13, 2023

isodude commented Oct 13, 2023 via email

Unit in network namespace uses default namespace resolv.conf instead of the bind mount /etc/netns/<namespace>/resolv.conf #28694

Unit in network namespace uses default namespace resolv.conf instead of the bind mount /etc/netns/<namespace>/resolv.conf #28694

Comments

rdbisme commented Aug 5, 2023 • edited

systemd version the issue has been seen with

Used distribution

Linux kernel version used

CPU architectures issue was seen on

Component

Expected behaviour you didn't see

Unexpected behaviour you saw

Steps to reproduce the problem

Additional program output to the terminal or log subsystem illustrating the issue

rdbisme commented Aug 5, 2023 • edited

yuwata commented Aug 5, 2023

rdbisme commented Aug 5, 2023 • edited

yuwata commented Aug 5, 2023

rdbisme commented Aug 5, 2023 • edited

yuwata commented Aug 8, 2023 • edited

rdbisme commented Aug 9, 2023 • edited

rdbisme commented Aug 9, 2023 • edited

rdbisme commented Aug 17, 2023 • edited

isodude commented Oct 12, 2023

rdbisme commented Oct 13, 2023

isodude commented Oct 13, 2023 via email

rdbisme commented Aug 5, 2023 •

edited

rdbisme commented Aug 5, 2023 •

edited

rdbisme commented Aug 5, 2023 •

edited

rdbisme commented Aug 5, 2023 •

edited

yuwata commented Aug 8, 2023 •

edited

rdbisme commented Aug 9, 2023 •

edited

rdbisme commented Aug 9, 2023 •

edited

rdbisme commented Aug 17, 2023 •

edited