networkd - device configuration never completes inside lxc container #9427

ghost · 2018-06-26T11:12:44Z

systemd version the issue has been seen with

238 (in the LXC container)

Used distribution

host ubuntu 4.15.0-23 (being a VPS on a KVM hypervisor with virtio driver)
container archlinux
lxc 3.0.1

Expected behaviour you didn't see

device completes configuration

Unexpected behaviour you saw

device state remains (configuring) indefinitely

Steps to reproduce the problem

install lxc on host
create lxc container with respective configuration settings
start lxc container and login into
check device state in the container

This state remains indefinitely

networkctl status eth0

● 15: eth0
Link File: n/a
Network File: /etc/systemd/network/eth0.network
Type: ether
State: routable (configuring)
HW Address: 00:16:3e:a5:69:70 (Xensource, Inc.)
Address: 172.25.120.109
Gateway: 172.25.120.1
DNS: 172.25.120.1

the eth0 in the container is the sibling of a veth pair created on the host through lxc.net.[i]type = veth. Subsquent /etc/systemd/network/eth0.network is found on the container once booted

[Match]
Name=eth0

[Network]
DHCP=ipv4

The text was updated successfully, but these errors were encountered:

yuwata · 2018-06-26T11:21:53Z

Please provide relevant logs...

ghost · 2018-06-26T11:42:14Z

That is output from journalctl -g eth0

systemd-networkd[45]: Ignoring /etc/systemd/network/eth0.network, because it's not a regular file with suffix .netdev.
systemd-networkd[45]: eth0: Flags change: +UP +LOWER_UP +RUNNING +MULTICAST +BROADCAST
systemd-networkd[45]: eth0: Link 23 added
systemd-networkd[45]: eth0: Link state is up-to-date
systemd-networkd[45]: eth0: found matching network '/etc/systemd/network/eth0.network'
systemd-networkd[45]: eth0: Started LLDP.
systemd-networkd[45]: eth0: Acquiring DHCPv4 lease
systemd-networkd[45]: eth0: Saved original MTU: 9000
systemd-networkd[45]: eth0: DHCPv4 address 172.25.120.109/24 via 172.25.120.1
systemd-networkd[45]: eth0: Updating address: 172.25.120.109/24 (valid for 1h)
systemd-networkd[45]: eth0: DHCP error: could not get routes: No data available

Does not really say why it is not completing the configuration process. Is ther any place to verbose some debugging?

yuwata · 2018-06-27T05:12:22Z

I guess this is already fixed by 223932c.
Could you test with v239 or recent git snapshot?

ghost · 2018-06-27T09:12:25Z

Whilst currently being tested in the distro installed their 239.0-2 package manually and rebooted with this outcome

systemd[58]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
systemd[58]: systemd-networkd.service: Failed at step USER spawning /usr/lib/systemd/systemd-networkd: Permission denied
systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 4.
systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
systemd[63]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
systemd[63]: systemd-networkd.service: Failed at step USER spawning /usr/lib/systemd/systemd-networkd: Permission denied
systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
systemd[1]: systemd-networkd.service: Start request repeated too quickly.
systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
systemd[1]: systemd-networkd.socket: Failed with result 'service-start-limit-hit'.

yuwata · 2018-06-27T12:38:35Z

Hmm, DynamicUser= is not supported on LXC?
Could you disable DynamicUser= and create user systemd-network if not exist.

ghost · 2018-06-27T12:43:15Z

It might be supported but not in the unpriviliged environment. I would have to ask that question the developers of LXC. Would not know how to en/disable DynamicUser=

ghost · 2018-06-27T12:59:44Z

Tried DynamicUser=false and DynamicUser=no in /etc/systemd/system.conf but that does not change a thing

yuwata · 2018-06-27T13:03:57Z

Please set DynamicUser=no on systemd-networkd.service, It is better to create a drop-in file for networkd:

# /etc/systemd/system/systemd-networkd.service.d/override.conf
[Service]
DynamicUser=no

brauner · 2018-06-27T13:12:58Z

The ArchLinux kernel does not support user namespaces by default. This might cause this error. LXC itself does not restrict access to user namespaces.

brauner · 2018-06-27T13:13:26Z

Oh, sorry, I mistook the container for the host.

ghost · 2018-06-27T13:15:15Z

Now that networkd is up and running (with DynamicUser=no) the logs for the iface showing

systemd-networkd[44]: eth0: Flags change: +UP +LOWER_UP +RUNNING +MULTICAST +BROADCAST
systemd-networkd[44]: eth0: Link 35 added
systemd-networkd[44]: eth0: Link state is up-to-date
systemd-networkd[44]: eth0: found matching network '/etc/systemd/network/eth0.network'
systemd-networkd[44]: eth0: Started LLDP.
systemd-networkd[44]: eth0: Acquiring DHCPv4 lease
systemd-networkd[44]: eth0: Saved original MTU: 9000
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: DHCPv4 address 172.25.120.241/24 via 172.25.120.1
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Updating address: 172.25.120.241/24 (valid for 1h)
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: DHCP error: could not get routes: No data available
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied

networkctl status eth0

● 35: eth0
Link File: n/a
Network File: n/a
Type: ether
State: n/a (unmanaged)
HW Address: 00:16:3e:4f:74:72 (Xensource, Inc.)
Address: 172.25.120.241
Gateway: 172.25.120.1

brauner · 2018-06-27T13:22:13Z

DynamicUser=yes works fine for me with systemd 239 in an ArchLinux container:

[root@arch1 ~]# systemctl status systemd-networkd
● systemd-networkd.service - Network Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-06-27 13:20:43 UTC; 1min 4s ago
     Docs: man:systemd-networkd.service(8)
 Main PID: 54 (systemd-network)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   Memory: 1.3M
   CGroup: /system.slice/systemd-networkd.service
           └─54 /usr/lib/systemd/systemd-networkd

Jun 27 13:20:43 arch1 systemd[1]: Starting Network Service...
Jun 27 13:20:43 arch1 systemd-networkd[54]: Enumeration completed
Jun 27 13:20:43 arch1 systemd[1]: Started Network Service.
Jun 27 13:20:43 arch1 systemd-networkd[54]: request_name_destroy_callback n_ref=1
Jun 27 13:20:43 arch1 systemd-networkd[54]: eth0: DHCPv4 address 10.113.222.25/24 via 10.113.222.1
Jun 27 13:20:44 arch1 systemd-networkd[54]: eth0: Gained IPv6LL
Jun 27 13:20:45 arch1 systemd-networkd[54]: eth0: Configured

so I suspect that you somehow can't create user namespace in your VPS.

ghost · 2018-06-27T13:32:14Z

@brauner

so I suspect that you somehow can't create user namespace in your VPS

It does work with no such issue in 238 and creates the lxc.uts.name = just fine.
238 also does not exhibit

Failed to save link data to /run/systemd/netif/links/35: Permission denied

Please don't takes this wrong way but your test ran in an unpriviliged environment, perhaps that was not expressed clearly in the thread?

yuwata · 2018-06-27T13:36:53Z

Ah, please also revert 2af7677. That is, specify user and group systemd-networkd in /usr/lib/tmpfiles.d/systemd.conf. When DynamicUser= is disabled, the directory /run/systemd/netif/links is not chowned, then the service cannot write any files under the directory.

brauner · 2018-06-27T13:37:44Z

@n8v8R, oh that is odd. You mean ran an ArchLinux unprivileged container?

ghost · 2018-06-27T13:42:44Z

@brauner

You mean ran an ArchLinux unprivileged container?

yes.

brauner · 2018-06-27T13:53:21Z

Yes, I did.

yuwata · 2018-06-27T13:59:47Z

Or, I am not sure, but alternatively, please re-enable DynamicUser= and 'static' user and group systemd-network exist.

ghost · 2018-06-27T14:18:36Z

@yuwata please pardon my ignorance but I did not get this part

but 'static' user and group systemd-networkd make exists.

yuwata · 2018-06-27T14:29:33Z

Sorry for my corrupted English sentences. I mean, that creating user and group systemd-network may sufficient to 'fix' recent your 'Permission Denied' issues.

eworm-de · 2018-06-27T14:44:16Z

[...] creating user and group systemd-network may sufficient to 'fix' recent your 'Permission Denied' issues.

No, it does not.

ghost · 2018-06-27T19:24:31Z

moved to another distro with another network parser, hence please feel at liberty to close the issue.

yuwata · 2018-06-28T06:53:09Z

As I already mentioned, the original issue seems already fixed by 223932c. Let's close this.

poettering added bug 🐛 Programming errors, that need preferential fixing network labels Jun 26, 2018

yuwata added needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer and removed bug 🐛 Programming errors, that need preferential fixing labels Jun 27, 2018

yuwata closed this as completed Jun 28, 2018

ghost mentioned this issue Jul 3, 2018

Units with DynamicUser=yes fail in lxc container #9493

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

networkd - device configuration never completes inside lxc container #9427

networkd - device configuration never completes inside lxc container #9427

ghost commented Jun 26, 2018

yuwata commented Jun 26, 2018

ghost commented Jun 26, 2018

yuwata commented Jun 27, 2018

ghost commented Jun 27, 2018

yuwata commented Jun 27, 2018

ghost commented Jun 27, 2018

ghost commented Jun 27, 2018

yuwata commented Jun 27, 2018

brauner commented Jun 27, 2018

brauner commented Jun 27, 2018

ghost commented Jun 27, 2018 •

edited by ghost

brauner commented Jun 27, 2018

ghost commented Jun 27, 2018 •

edited by ghost

yuwata commented Jun 27, 2018 •

edited

brauner commented Jun 27, 2018

ghost commented Jun 27, 2018

brauner commented Jun 27, 2018

yuwata commented Jun 27, 2018 •

edited

ghost commented Jun 27, 2018

yuwata commented Jun 27, 2018

eworm-de commented Jun 27, 2018

ghost commented Jun 27, 2018

yuwata commented Jun 28, 2018

networkd - device configuration never completes inside lxc container #9427

networkd - device configuration never completes inside lxc container #9427

Comments

ghost commented Jun 26, 2018

yuwata commented Jun 26, 2018

ghost commented Jun 26, 2018

yuwata commented Jun 27, 2018

ghost commented Jun 27, 2018

yuwata commented Jun 27, 2018

ghost commented Jun 27, 2018

ghost commented Jun 27, 2018

yuwata commented Jun 27, 2018

brauner commented Jun 27, 2018

brauner commented Jun 27, 2018

ghost commented Jun 27, 2018 • edited by ghost

brauner commented Jun 27, 2018

ghost commented Jun 27, 2018 • edited by ghost

yuwata commented Jun 27, 2018 • edited

brauner commented Jun 27, 2018

ghost commented Jun 27, 2018

brauner commented Jun 27, 2018

yuwata commented Jun 27, 2018 • edited

ghost commented Jun 27, 2018

yuwata commented Jun 27, 2018

eworm-de commented Jun 27, 2018

ghost commented Jun 27, 2018

yuwata commented Jun 28, 2018

ghost commented Jun 27, 2018 •

edited by ghost

ghost commented Jun 27, 2018 •

edited by ghost

yuwata commented Jun 27, 2018 •

edited

yuwata commented Jun 27, 2018 •

edited