Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

networkd - device configuration never completes inside lxc container #9427

Closed
ghost opened this issue Jun 26, 2018 · 23 comments
Closed

networkd - device configuration never completes inside lxc container #9427

ghost opened this issue Jun 26, 2018 · 23 comments

Comments

@ghost
Copy link

@ghost ghost commented Jun 26, 2018

systemd version the issue has been seen with

238 (in the LXC container)

Used distribution

host ubuntu 4.15.0-23 (being a VPS on a KVM hypervisor with virtio driver)
container archlinux
lxc 3.0.1

Expected behaviour you didn't see

device completes configuration

Unexpected behaviour you saw

device state remains (configuring) indefinitely

Steps to reproduce the problem

  1. install lxc on host
  2. create lxc container with respective configuration settings
  3. start lxc container and login into
  4. check device state in the container

This state remains indefinitely

networkctl status eth0

● 15: eth0
Link File: n/a
Network File: /etc/systemd/network/eth0.network
Type: ether
State: routable (configuring)
HW Address: 00:16:3e:a5:69:70 (Xensource, Inc.)
Address: 172.25.120.109
Gateway: 172.25.120.1
DNS: 172.25.120.1


the eth0 in the container is the sibling of a veth pair created on the host through lxc.net.[i]type = veth. Subsquent /etc/systemd/network/eth0.network is found on the container once booted

[Match]
Name=eth0

[Network]
DHCP=ipv4
@yuwata
Copy link
Member

@yuwata yuwata commented Jun 26, 2018

Please provide relevant logs...

@ghost
Copy link
Author

@ghost ghost commented Jun 26, 2018

That is output from journalctl -g eth0

systemd-networkd[45]: Ignoring /etc/systemd/network/eth0.network, because it's not a regular file with suffix .netdev.
systemd-networkd[45]: eth0: Flags change: +UP +LOWER_UP +RUNNING +MULTICAST +BROADCAST
systemd-networkd[45]: eth0: Link 23 added
systemd-networkd[45]: eth0: Link state is up-to-date
systemd-networkd[45]: eth0: found matching network '/etc/systemd/network/eth0.network'
systemd-networkd[45]: eth0: Started LLDP.
systemd-networkd[45]: eth0: Acquiring DHCPv4 lease
systemd-networkd[45]: eth0: Saved original MTU: 9000
systemd-networkd[45]: eth0: DHCPv4 address 172.25.120.109/24 via 172.25.120.1
systemd-networkd[45]: eth0: Updating address: 172.25.120.109/24 (valid for 1h)
systemd-networkd[45]: eth0: DHCP error: could not get routes: No data available


Does not really say why it is not completing the configuration process. Is ther any place to verbose some debugging?

@yuwata
Copy link
Member

@yuwata yuwata commented Jun 27, 2018

I guess this is already fixed by 223932c.
Could you test with v239 or recent git snapshot?

@ghost
Copy link
Author

@ghost ghost commented Jun 27, 2018

Whilst currently being tested in the distro installed their 239.0-2 package manually and rebooted with this outcome

systemd[58]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
systemd[58]: systemd-networkd.service: Failed at step USER spawning /usr/lib/systemd/systemd-networkd: Permission denied
systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 4.
systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
systemd[63]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
systemd[63]: systemd-networkd.service: Failed at step USER spawning /usr/lib/systemd/systemd-networkd: Permission denied
systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
systemd[1]: systemd-networkd.service: Start request repeated too quickly.
systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
systemd[1]: systemd-networkd.socket: Failed with result 'service-start-limit-hit'.

@yuwata
Copy link
Member

@yuwata yuwata commented Jun 27, 2018

Hmm, DynamicUser= is not supported on LXC?
Could you disable DynamicUser= and create user systemd-network if not exist.

@ghost
Copy link
Author

@ghost ghost commented Jun 27, 2018

It might be supported but not in the unpriviliged environment. I would have to ask that question the developers of LXC. Would not know how to en/disable DynamicUser=

@ghost
Copy link
Author

@ghost ghost commented Jun 27, 2018

Tried DynamicUser=false and DynamicUser=no in /etc/systemd/system.conf but that does not change a thing

@yuwata
Copy link
Member

@yuwata yuwata commented Jun 27, 2018

Please set DynamicUser=no on systemd-networkd.service, It is better to create a drop-in file for networkd:

# /etc/systemd/system/systemd-networkd.service.d/override.conf
[Service]
DynamicUser=no
@brauner
Copy link
Contributor

@brauner brauner commented Jun 27, 2018

The ArchLinux kernel does not support user namespaces by default. This might cause this error. LXC itself does not restrict access to user namespaces.

@brauner
Copy link
Contributor

@brauner brauner commented Jun 27, 2018

Oh, sorry, I mistook the container for the host.

@ghost
Copy link
Author

@ghost ghost commented Jun 27, 2018

Now that networkd is up and running (with DynamicUser=no) the logs for the iface showing

systemd-networkd[44]: eth0: Flags change: +UP +LOWER_UP +RUNNING +MULTICAST +BROADCAST
systemd-networkd[44]: eth0: Link 35 added
systemd-networkd[44]: eth0: Link state is up-to-date
systemd-networkd[44]: eth0: found matching network '/etc/systemd/network/eth0.network'
systemd-networkd[44]: eth0: Started LLDP.
systemd-networkd[44]: eth0: Acquiring DHCPv4 lease
systemd-networkd[44]: eth0: Saved original MTU: 9000
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: DHCPv4 address 172.25.120.241/24 via 172.25.120.1
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Updating address: 172.25.120.241/24 (valid for 1h)
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: DHCP error: could not get routes: No data available
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied
systemd-networkd[44]: eth0: Failed to save link data to /run/systemd/netif/links/35: Permission denied


networkctl status eth0

● 35: eth0
Link File: n/a
Network File: n/a
Type: ether
State: n/a (unmanaged)
HW Address: 00:16:3e:4f:74:72 (Xensource, Inc.)
Address: 172.25.120.241
Gateway: 172.25.120.1

@brauner
Copy link
Contributor

@brauner brauner commented Jun 27, 2018

DynamicUser=yes works fine for me with systemd 239 in an ArchLinux container:

[root@arch1 ~]# systemctl status systemd-networkd
● systemd-networkd.service - Network Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-06-27 13:20:43 UTC; 1min 4s ago
     Docs: man:systemd-networkd.service(8)
 Main PID: 54 (systemd-network)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   Memory: 1.3M
   CGroup: /system.slice/systemd-networkd.service
           └─54 /usr/lib/systemd/systemd-networkd

Jun 27 13:20:43 arch1 systemd[1]: Starting Network Service...
Jun 27 13:20:43 arch1 systemd-networkd[54]: Enumeration completed
Jun 27 13:20:43 arch1 systemd[1]: Started Network Service.
Jun 27 13:20:43 arch1 systemd-networkd[54]: request_name_destroy_callback n_ref=1
Jun 27 13:20:43 arch1 systemd-networkd[54]: eth0: DHCPv4 address 10.113.222.25/24 via 10.113.222.1
Jun 27 13:20:44 arch1 systemd-networkd[54]: eth0: Gained IPv6LL
Jun 27 13:20:45 arch1 systemd-networkd[54]: eth0: Configured

so I suspect that you somehow can't create user namespace in your VPS.

@ghost
Copy link
Author

@ghost ghost commented Jun 27, 2018

@brauner

so I suspect that you somehow can't create user namespace in your VPS

It does work with no such issue in 238 and creates the lxc.uts.name = just fine.
238 also does not exhibit

Failed to save link data to /run/systemd/netif/links/35: Permission denied

Please don't takes this wrong way but your test ran in an unpriviliged environment, perhaps that was not expressed clearly in the thread?

@yuwata
Copy link
Member

@yuwata yuwata commented Jun 27, 2018

Ah, please also revert 2af7677. That is, specify user and group systemd-networkd in /usr/lib/tmpfiles.d/systemd.conf. When DynamicUser= is disabled, the directory /run/systemd/netif/links is not chowned, then the service cannot write any files under the directory.

@brauner
Copy link
Contributor

@brauner brauner commented Jun 27, 2018

@n8v8R, oh that is odd. You mean ran an ArchLinux unprivileged container?

@ghost
Copy link
Author

@ghost ghost commented Jun 27, 2018

@brauner

You mean ran an ArchLinux unprivileged container?

yes.

@brauner
Copy link
Contributor

@brauner brauner commented Jun 27, 2018

Yes, I did.

@yuwata
Copy link
Member

@yuwata yuwata commented Jun 27, 2018

Or, I am not sure, but alternatively, please re-enable DynamicUser= and 'static' user and group systemd-network exist.

@ghost
Copy link
Author

@ghost ghost commented Jun 27, 2018

@yuwata please pardon my ignorance but I did not get this part

but 'static' user and group systemd-networkd make exists.

@yuwata
Copy link
Member

@yuwata yuwata commented Jun 27, 2018

Sorry for my corrupted English sentences. I mean, that creating user and group systemd-network may sufficient to 'fix' recent your 'Permission Denied' issues.

@eworm-de
Copy link
Contributor

@eworm-de eworm-de commented Jun 27, 2018

[...] creating user and group systemd-network may sufficient to 'fix' recent your 'Permission Denied' issues.

No, it does not.

@ghost
Copy link
Author

@ghost ghost commented Jun 27, 2018

moved to another distro with another network parser, hence please feel at liberty to close the issue.

@yuwata
Copy link
Member

@yuwata yuwata commented Jun 28, 2018

As I already mentioned, the original issue seems already fixed by 223932c. Let's close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants