Configuring lo interface in a netns from systemd unit not working anymore

[rdbisme]

systemd version the issue has been seen with

254

Used distribution

Archlinux

Linux kernel version used

6.1.39-3-lts

CPU architectures issue was seen on

x86_64

Component

systemd

Expected behaviour you didn't see

The unit used to start as expected before the latest update

Unexpected behaviour you saw

The /usr/bin/env ip -n vpn link set dev lo up command fails when running within the systemd unit

Steps to reproduce the problem

This unit has been working for a year. After latest update, it doesn't work anymore.

[Unit]
Description=%I Namespace wrapper
After=network-online.target nss-lookup.target
Wants=network-online.target nss-lookup.target

[Service]
Type=oneshot
RemainAfterExit=true
PrivateNetwork=true

ExecStartPre=-/usr/bin/env ip netns delete %i

# Add vpn dedicated namespace
ExecStart=/usr/bin/env ip netns add %i

# Pause
ExecStart=/usr/bin/env echo %i

# Set loopback device up
ExecStart=/usr/bin/env ip -n %i link set dev lo up

# Stop process
ExecStop=/usr/bin/env ip netns delete %i

[Install]
WantedBy=multi-user.target
WantedBy=network-online.target

1. sudo systemctl start netns@vpn.service

If I run the commands by hand (prepended by sudo), everything works as expected

Additional program output to the terminal or log subsystem illustrating the issue

❯ sudo systemctl status netns@vpn.service
× netns@vpn.service - vpn Namespace wrapper
     Loaded: loaded (/etc/systemd/system/netns@.service; disabled; preset: disabled)
     Active: failed (Result: exit-code) since Sat 2023-08-05 12:32:20 CEST; 1min 59s ago
    Process: 26977 ExecStartPre=/usr/bin/env ip netns delete vpn (code=exited, status=1/FAILURE)
    Process: 26978 ExecStart=/usr/bin/env ip netns add vpn (code=exited, status=0/SUCCESS)
    Process: 26980 ExecStart=/usr/bin/env echo vpn (code=exited, status=0/SUCCESS)
    Process: 26983 ExecStart=/usr/bin/env ip -n vpn link set dev lo up (code=exited, status=255/EXCEPTION)
   Main PID: 26983 (code=exited, status=255/EXCEPTION)
        CPU: 41ms

Aug 05 12:32:20 xxx systemd[1]: Starting vpn Namespace wrapper...
Aug 05 12:32:20 xxx env[26977]: Cannot remove namespace file "/var/run/netns/vpn": No such file or directory
Aug 05 12:32:20 xxx env[26980]: vpn
Aug 05 12:32:20 xxx env[26983]: setting the network namespace "vpn" failed: Invalid argument
Aug 05 12:32:20 xxx systemd[1]: netns@vpn.service: Main process exited, code=exited, status=255/EXCEPTION
Aug 05 12:32:20 xxx systemd[1]: netns@vpn.service: Failed with result 'exit-code'.
Aug 05 12:32:20 xxx systemd[1]: Failed to start vpn Namespace wrapper.


❯ sudo ip netns
Error: Peer netns reference is invalid.
Error: Peer netns reference is invalid.
vpn

[mrc0mmand]

main:

639   getuid()                          = 0
639   openat(AT_FDCWD</>, "/var/run/netns/foo", O_RDONLY|O_CLOEXEC) = 3</run/netns/foo>
639   setns(3</run/netns/foo>, CLONE_NEWNET) = -1 EINVAL (Invalid argument)
639   write(2</dev/pts/1>, "setting the network namespace \"foo\" failed: Invalid argument\n", 61) = 61
639   close(3</run/netns/foo>)          = 0
639   exit_group(-1)                    = ?
639   +++ exited with 255 +++

v253:

478   openat(AT_FDCWD</>, "/var/run/netns/foo", O_RDONLY|O_CLOEXEC <unfinished ...>
478   <... openat resumed>)             = 3</run/netns/foo>
478   setns(3</run/netns/foo>, CLONE_NEWNET <unfinished ...>
478   <... setns resumed>)              = 0
478   close(3</run/netns/foo> <unfinished ...>
478   <... close resumed>)              = 0

And bisect points to c2da3bf. Given the commit description and the respective documentation this behavior seems to be expected, and adding PrivateMounts=no to the unit should fix the error (/cc @yuwata for confirmation).

[rdbisme]

main:

639   getuid()                          = 0
639   openat(AT_FDCWD</>, "/var/run/netns/foo", O_RDONLY|O_CLOEXEC) = 3</run/netns/foo>
639   setns(3</run/netns/foo>, CLONE_NEWNET) = -1 EINVAL (Invalid argument)
639   write(2</dev/pts/1>, "setting the network namespace \"foo\" failed: Invalid argument\n", 61) = 61
639   close(3</run/netns/foo>)          = 0
639   exit_group(-1)                    = ?
639   +++ exited with 255 +++

v253:

478   openat(AT_FDCWD</>, "/var/run/netns/foo", O_RDONLY|O_CLOEXEC <unfinished ...>
478   <... openat resumed>)             = 3</run/netns/foo>
478   setns(3</run/netns/foo>, CLONE_NEWNET <unfinished ...>
478   <... setns resumed>)              = 0
478   close(3</run/netns/foo> <unfinished ...>
478   <... close resumed>)              = 0

And bisect points to c2da3bf. Given the commit description and the respective documentation this behavior seems to be expected, and adding PrivateMounts=no to the unit should fix the error (/cc @yuwata for confirmation).

Adding PrivateMounts=no works. @mrc0mmand could you elaborate about the change? Is there something I can do to make it work with PrivateMounts=true?

[rdbisme]

This is related: #2741 (comment)

[yuwata]

Sorry, but that's expected and intended change. Closing.