Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

networkd - device configuration never completes inside lxc container #9427

Closed
n8v8R opened this Issue Jun 26, 2018 · 23 comments

Comments

5 participants
@n8v8R
Copy link

n8v8R commented Jun 26, 2018

systemd version the issue has been seen with

238 (in the LXC container)

Used distribution

host ubuntu 4.15.0-23 (being a VPS on a KVM hypervisor with virtio driver)
container archlinux
lxc 3.0.1

Expected behaviour you didn't see

device completes configuration

Unexpected behaviour you saw

device state remains (configuring) indefinitely

Steps to reproduce the problem

  1. install lxc on host
  2. create lxc container with respective configuration settings
  3. start lxc container and login into
  4. check device state in the container

This state remains indefinitely

networkctl status eth0

● 15: eth0
Link File: n/a
Network File: /etc/systemd/network/eth0.network
Type: ether
State: routable (configuring)
HW Address: 00:16:3e:a5:69:70 (Xensource, Inc.)
Address: 172.25.120.109
Gateway: 172.25.120.1
DNS: 172.25.120.1


the eth0 in the container is the sibling of a veth pair created on the host through lxc.net.[i]type = veth. Subsquent /etc/systemd/network/eth0.network is found on the container once booted

[Match]
Name=eth0

[Network]
DHCP=ipv4
@yuwata

This comment has been minimized.

Copy link
Member

yuwata commented Jun 26, 2018

Please provide relevant logs...

@n8v8R

This comment has been minimized.

Copy link
Author

n8v8R commented Jun 26, 2018

That is output from journalctl -g eth0

systemd-networkd[45]: Ignoring /etc/systemd/network/eth0.network, because it's not a regular file with suffix .netdev.
systemd-networkd[45]: eth0: Flags change: +UP +LOWER_UP +RUNNING +MULTICAST +BROADCAST
systemd-networkd[45]: eth0: Link 23 added
systemd-networkd[45]: eth0: Link state is up-to-date
systemd-networkd[45]: eth0: found matching network '/etc/systemd/network/eth0.network'
systemd-networkd[45]: eth0: Started LLDP.
systemd-networkd[45]: eth0: Acquiring DHCPv4 lease
systemd-networkd[45]: eth0: Saved original MTU: 9000
systemd-networkd[45]: eth0: DHCPv4 address 172.25.120.109/24 via 172.25.120.1
systemd-networkd[45]: eth0: Updating address: 172.25.120.109/24 (valid for 1h)
systemd-networkd[45]: eth0: DHCP error: could not get routes: No data available


Does not really say why it is not completing the configuration process. Is ther any place to verbose some debugging?

@yuwata

This comment has been minimized.

Copy link
Member

yuwata commented Jun 27, 2018

I guess this is already fixed by 223932c.
Could you test with v239 or recent git snapshot?

@n8v8R

This comment has been minimized.

Copy link
Author

n8v8R commented Jun 27, 2018

Whilst currently being tested in the distro installed their 239.0-2 package manually and rebooted with this outcome

systemd[58]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
systemd[58]: systemd-networkd.service: Failed at step USER spawning /usr/lib/systemd/systemd-networkd: Permission denied
systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 4.
systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
systemd[63]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
systemd[63]: systemd-networkd.service: Failed at step USER spawning /usr/lib/systemd/systemd-networkd: Permission denied
systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
systemd[1]: systemd-networkd.service: Start request repeated too quickly.
systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
systemd[1]: systemd-networkd.socket: Failed with result 'service-start-limit-hit'.

@yuwata

This comment has been minimized.

Copy link
Member

yuwata commented Jun 27, 2018

Hmm, DynamicUser= is not supported on LXC?
Could you disable DynamicUser= and create user systemd-network if not exist.

@n8v8R

This comment has been minimized.

Copy link
Author

n8v8R commented Jun 27, 2018

It might be supported but not in the unpriviliged environment. I would have to ask that question the developers of LXC. Would not know how to en/disable DynamicUser=

@n8v8R

This comment has been minimized.

Copy link
Author

n8v8R commented Jun 27, 2018

Tried DynamicUser=false and DynamicUser=no in /etc/systemd/system.conf but that does not change a thing

@yuwata

This comment has been minimized.

Copy link
Member

yuwata commented Jun 27, 2018

Please set DynamicUser=no on systemd-networkd.service, It is better to create a drop-in file for networkd:

# /etc/systemd/system/systemd-networkd.service.d/override.conf
[Service]
DynamicUser=no
@brauner

This comment has been minimized.

Copy link
Contributor

brauner commented Jun 27, 2018

The ArchLinux kernel does not support user namespaces by default. This might cause this error. LXC itself does not restrict access to user namespaces.

@brauner

This comment has been minimized.

Copy link
Contributor

brauner commented Jun 27, 2018

Oh, sorry, I mistook the container for the host.

@n8v8R

This comment has been minimized.

Copy link
Author

n8v8R commented Jun 27, 2018

Now that networkd is up and running (with DynamicUser=no) the logs for the iface showing

systemd-networkd[44]: eth0: Flags change: +UP +LOWER_UP +RUNNING +MULTICAST +BROADCAST
systemd-networkd[44]: eth0: Link 35 added
systemd-networkd[44]: eth0: Link state is up-to-date
systemd-networkd[44]: eth0: found matching network '/etc/systemd/network/eth0.network'
systemd-networkd[44]: eth0: Started LLDP.
systemd-networkd[44]: eth0: Acquiring DHCPv4 lease
systemd-networkd[44]: eth0: Saved original MTU: 9000
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: DHCPv4 address 172.25.120.241/24 via 172.25.120.1
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Updating address: 172.25.120.241/24 (valid for 1h)
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: DHCP error: could not get routes: No data available
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied


networkctl status eth0

● 35: eth0
Link File: n/a
Network File: n/a
Type: ether
State: n/a (unmanaged)
HW Address: 00:16:3e:4f:74:72 (Xensource, Inc.)
Address: 172.25.120.241
Gateway: 172.25.120.1

@brauner

This comment has been minimized.

Copy link
Contributor

brauner commented Jun 27, 2018

DynamicUser=yes works fine for me with systemd 239 in an ArchLinux container:

[root@arch1 ~]# systemctl status systemd-networkd
● systemd-networkd.service - Network Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-06-27 13:20:43 UTC; 1min 4s ago
     Docs: man:systemd-networkd.service(8)
 Main PID: 54 (systemd-network)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   Memory: 1.3M
   CGroup: /system.slice/systemd-networkd.service
           └─54 /usr/lib/systemd/systemd-networkd

Jun 27 13:20:43 arch1 systemd[1]: Starting Network Service...
Jun 27 13:20:43 arch1 systemd-networkd[54]: Enumeration completed
Jun 27 13:20:43 arch1 systemd[1]: Started Network Service.
Jun 27 13:20:43 arch1 systemd-networkd[54]: request_name_destroy_callback n_ref=1
Jun 27 13:20:43 arch1 systemd-networkd[54]: eth0: DHCPv4 address 10.113.222.25/24 via 10.113.222.1
Jun 27 13:20:44 arch1 systemd-networkd[54]: eth0: Gained IPv6LL
Jun 27 13:20:45 arch1 systemd-networkd[54]: eth0: Configured

so I suspect that you somehow can't create user namespace in your VPS.

@n8v8R

This comment has been minimized.

Copy link
Author

n8v8R commented Jun 27, 2018

@brauner

so I suspect that you somehow can't create user namespace in your VPS

It does work with no such issue in 238 and creates the lxc.uts.name = just fine.
238 also does not exhibit

Failed to save link data to /run/systemd/netif/links/35: Permission denied

Please don't takes this wrong way but your test ran in an unpriviliged environment, perhaps that was not expressed clearly in the thread?

@yuwata

This comment has been minimized.

Copy link
Member

yuwata commented Jun 27, 2018

Ah, please also revert 2af7677. That is, specify user and group systemd-networkd in /usr/lib/tmpfiles.d/systemd.conf. When DynamicUser= is disabled, the directory /run/systemd/netif/links is not chowned, then the service cannot write any files under the directory.

@brauner

This comment has been minimized.

Copy link
Contributor

brauner commented Jun 27, 2018

@n8v8R, oh that is odd. You mean ran an ArchLinux unprivileged container?

@n8v8R

This comment has been minimized.

Copy link
Author

n8v8R commented Jun 27, 2018

@brauner

You mean ran an ArchLinux unprivileged container?

yes.

@brauner

This comment has been minimized.

Copy link
Contributor

brauner commented Jun 27, 2018

Yes, I did.

@yuwata

This comment has been minimized.

Copy link
Member

yuwata commented Jun 27, 2018

Or, I am not sure, but alternatively, please re-enable DynamicUser= and 'static' user and group systemd-network exist.

@n8v8R

This comment has been minimized.

Copy link
Author

n8v8R commented Jun 27, 2018

@yuwata please pardon my ignorance but I did not get this part

but 'static' user and group systemd-networkd make exists.

@yuwata

This comment has been minimized.

Copy link
Member

yuwata commented Jun 27, 2018

Sorry for my corrupted English sentences. I mean, that creating user and group systemd-network may sufficient to 'fix' recent your 'Permission Denied' issues.

@eworm-de

This comment has been minimized.

Copy link
Contributor

eworm-de commented Jun 27, 2018

[...] creating user and group systemd-network may sufficient to 'fix' recent your 'Permission Denied' issues.

No, it does not.

@n8v8R

This comment has been minimized.

Copy link
Author

n8v8R commented Jun 27, 2018

moved to another distro with another network parser, hence please feel at liberty to close the issue.

@yuwata

This comment has been minimized.

Copy link
Member

yuwata commented Jun 28, 2018

As I already mentioned, the original issue seems already fixed by 223932c. Let's close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.