systemd-nspawn+macvlan "could not bring up interface: Device or resource busy"

[SjonHortensius]

I use systemd-nspawn with macvlan on a few different machines; and all of them suffer the same issue. Containers (which boot successfully) cannot be restarted; their network-connection will fail 99% of the time. I use the following overload:

# cat /etc/systemd/system/systemd-nspawn\@.service.d/start-network-macvlan.conf 
[Service]
ExecStart=
ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=guest --network-macvlan=br-lan --machine=%I

Sometimes the reboot works (at least with v221); but most of the times it errors with either:

systemd-networkd[29]: mv-br-lan: Cannot configure IPv4 forwarding for interface mv-br-lan: Read-only file system
systemd-networkd[29]: Enumeration completed
systemd-networkd[29]: mv-br-lan: mv-br-lan       : could not bring up interface: Device or resource busy
systemd-networkd[29]: mv-br-lan: mv-br-lan       : could not set route: Network is unreachable
systemd-networkd[29]: mv-br-lan: Configured

or (another machine):

systemd-networkd[23]: mv-enp0s25: Cannot configure IPv4 forwarding for interface mv-enp0s25: Read-only file system
systemd-networkd[23]: mv-enp0s25: Cannot configure IPv6 forwarding for interface: Read-only file system
systemd-networkd[23]: Enumeration completed
systemd-networkd[23]: mv-enp0s25: mv-enp0s25      : could not bring up interface: Device or resource busy

This still happens with systemd-222, is there any way for me to debug this? Right now I have to restart the whole host to get the container to function again. I think something related to cgroups or the vlan hangs after the container is stopped; preventing a successful restart

[poettering]

Hmm, so this only happens during boot?

There's a race between nspawn and networkd: nspawn with the command line above will expect the bridge to be up, but networkd might still setting it up. We can fix that eventually by making nspawn talk to networkd via the bus, and syncing on creating of the bridge.

[SjonHortensius]

No it actually only works on first boot! The containers cannot be restarted after that first boot; they will never have a working network connection.

Also, the other machine does macvlan without a bridge and has the same issue

[poettering]

what do you mean by "after first boot"? "on subsequent boots"? or "during normal runtime"?

[SjonHortensius]

When the host boots, all containers start up fine. If I execute systemctl reboot in the container, or machinectl restart ... on the host or systemctl restart systemd-nspawn@... (or stop; start) on the host; the container won't boot.

So "on subsequent boots" of the container but during normal runtime of the host.

[SjonHortensius]

This seems to be fixed in 224, were there any changes done that might have caused that? I'll be testing to make sure it really is fixed

[arjenschol]

I've tested this issue and it looks like this is still a problem.
And I think this thread discusses the same issue: http://lists.freedesktop.org/archives/systemd-devel/2015-May/031427.html

[f0]

we see the same error, running coreos (systemd v225) and centos 7.2 (systemd 219) on container site with systemd-networkd

this is the log from inside the container after to get this output i must restart systemd-networkd

Jan 20 21:59:13 memcache03.1example.net systemd[1]: Starting Network Service...
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interfac
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.7 object=n/a interface=n/a member=n/a
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interfac
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interfac
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: timestamp of '/etc/systemd/network' changed
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: timestamp of '/usr/lib/systemd/network' changed
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : flags change: +MULTICAST +BROADCAST
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : link 2 added
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : link state is up-to-date
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : found matching network '/etc/systemd/network/eno3-2192.network'
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : Cannot configure IPv4 forwarding for interface mv-eno3-2192: Read-only file system
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : Cannot configure IPv6 forwarding for interface: Read-only file system
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/network1/link/_32 interface=org.fre
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : bringing link up
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/network1/link/_32 interface=org.fre
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : setting addresses
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : saved original MTU: 1500
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: lo              : flags change: +LOOPBACK +UP +LOWER_UP +RUNNING
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/network1/link/_31 interface=org.fre
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: lo              : link 1 added
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: lo              : link state is up-to-date
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: lo              : unmanaged
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/network1/link/_31 interface=org.fre
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: lo              : saved original MTU: 0
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: lo              : Adding address: ::1/128 (valid for ever)
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : Adding address: 10.62.192.26/26 (valid for ever)
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: lo              : Adding address: 127.0.0.1/8 (valid for ever)
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: Enumeration completed
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : mv-eno3-2192    : could not bring up interface: Device or resource busy
Jan 20 21:59:13 memcache03.1example.net systemd[1]: Started Network Service.
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : Updating address: 10.62.192.26/26 (valid for ever)
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : addresses set
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/network1/link/_32 interface=org.fre
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : setting routes
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : mv-eno3-2192    : could not set route: Network is unreachable
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : routes set
Jan 20 21:59:13 memcache03.1example.net systemd-networkd[215]: mv-eno3-2192    : link configured

[joejulian]

I'm seeing the same issue with 231 (Arch) host and a 219 (CentOS 7) guest. First time the machine is started, the link works fine. Power off the machine and start the machine (machinectl poweroff test1; machinectl start test1) and the link will not come up.

# ip -d link show mv-br0
2: mv-br0: <BROADCAST,MULTICAST> mtu 9000 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether aa:13:9f:1c:7a:ef brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 
    macvlan  mode bridge addrgenmode eui64

Restarting systemd-networkd has no affect (either the host nor the guest).

Removing the host's macvlan bridge (br0 in this example) and restarting systemd-networkd to recreate it does allow the guest's link to function again until the guest is restarted.

[gtjoseph]

Same issue with 232 on Fedora 25/kernel 4.8.10. Using macvlan, the first time the container is started after a host reboot the network interface comes up. If the container is shut down and restarted, the network interface does NOT come back up and attempts to bring it up fail.

# ifconfig -a
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 7  bytes 2345 (2.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7  bytes 2345 (2.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

mv-eno1.172: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        inet 172.31.1.22  netmask 255.255.0.0  broadcast 172.31.255.255
        ether 0a:40:d8:77:bc:49  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# ifconfig mv-eno1.172 up
SIOCSIFFLAGS: Device or resource busy

Note that systemd-networkd actually configured the interface, but just can't bring it up. Oddly enough, container will respond to ARP and ICMP requests but since the interface is down, none of the container's daemons will bind to it.

HOWEVER, it's not really a host reboot that's needed to fix it. It's just rmmoding macvlan on the host and restarting the container that allows the container to bring up the interface.

[SjonHortensius]

With 232 the error is slightly different (I've also added --network-veth to the nspawn, not sure if that causes this); attempting to execute reboot from container, the host will log:

systemd-nspawn[661]: Rebooting.
kernel: IPVS: Creating netns size=2176 id=7
systemd-nspawn[661]: Failed to add new veth interfaces (vb-xxx:host0): File exists
systemd-networkd[612]: vb-xxx: Lost carrier
systemd-timesyncd[585]: Network configuration changed, trying to establish connection.
kernel: DEV: port 6(vb-xxx) entered disabled state
kernel: device vb-xxx left promiscuous mode
kernel: DEV: port 6(vb-xxx) entered disabled state

after systemctl restart systemd-nspawn@xxx on host:

systemd[1]: Stopped Container xxx.
systemd[1]: Starting Container xxx...
kernel: IPVS: Creating netns size=2176 id=8
systemd-timesyncd[585]: Network configuration changed, trying to establish connection.
kernel: IPv6: ADDRCONF(NETDEV_UP): vb-xxx: link is not ready
kernel: DEV: port 6(vb-xxx) entered blocking state
kernel: DEV: port 6(vb-xxx) entered disabled state
kernel: device vb-xxx entered promiscuous mode
systemd-nspawn[26546]: Failed to register machine: Machine 'xxx' already exists

after systemctl stop systemd-nspawn@xxx; systemctl start systemd-nspawn@xxx on host:

# stop
systemd[1]: Stopped Container xxx.
# start
systemd[1]: Starting Container xxx...
kernel: IPVS: Creating netns size=2176 id=9
systemd-timesyncd[585]: Network configuration changed, trying to establish connection.
kernel: IPv6: ADDRCONF(NETDEV_UP): vb-xxx: link is not ready
kernel: DEV: port 6(vb-xxx) entered blocking state
kernel: DEV: port 6(vb-xxx) entered disabled state
kernel: device vb-xxx entered promiscuous mode
systemd-machined[765]: New machine xxx.

The stop/start is the only thing that works consistently.

[clhedrick]

I’ve seen the same issue with lxc. It looks like a problem with Macvlan not releasing the Mac and iP address when the container exits. I’m moving to kvm because of this

[falstaff1288]

Found a working fix to counter a similar issue. This is the error that was returned when the interface failed to come up:

Jul 06 09:32:38 archspawn systemd-networkd[115]: mv-enp3s0: Could not bring up interface: Address already in use

The fix is to assign a random MAC address in the .network file that configures the mv-* interface then restart systemd-networkd. Maybe it could as well for the "Device or resource busy".

Example:

[Match]
Name=mv*

[Link]
MACAddress=00:16:3e:03:94:24

[Network]
DHCP=ipv4