Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd-networkd default "unmanaged" for docker*, br-*, veth* #28626

Closed
bf opened this issue Aug 1, 2023 · 16 comments · Fixed by #28646
Closed

systemd-networkd default "unmanaged" for docker*, br-*, veth* #28626

bf opened this issue Aug 1, 2023 · 16 comments · Fixed by #28646
Labels
bug 🐛 Programming errors, that need preferential fixing network

Comments

@bf
Copy link

bf commented Aug 1, 2023

systemd version the issue has been seen with

253.7-1

Used distribution

Arch

Linux kernel version used

6.4.6-hardened

CPU architectures issue was seen on

x86_64

Component

No response

Expected behaviour you didn't see

Docker containers cannot connect to unternet,

networkctl shows "configuring" for docker0 and br-* devices.

Unexpected behaviour you saw

systemd-networkd applying default route to docker network interfaces, thereby blocking docker containers from reaching internet.

This issue has cost me countless hours. I dont know why systemd-networkd decides to ship some "clever" defaults in /usr/lib/systemd/network but doesn't have a default for docker.

Steps to reproduce the problem

Install systemd-networkd, install docker, start container: docker run busybox ping -c 1 1.1.1.1 -> timeout

networkctl will show that systemd-networkd is busy "configuring" docker0 and all other docker virtual interfaces

Fix:

create /etc/systemd/network/{20-veth.network,21-docker.network,22-docker-br.network}
with contents:

[Match]
Name=veth*
[Link]
Unmanaged=true
[Match]
Name=docker*
[Link]
Unmanaged=true
[Match]
Name=br*
[Link]
Unmanaged=true

I assume this issue has been reported here hundreds of times. I really dont know why it is still default behavior.

Additional program output to the terminal or log subsystem illustrating the issue

Aug 01 18:36:55 hub systemd-networkd[97962]: docker0: Configuring with /run/systemd/network/91-default.network.
Aug 01 18:36:55 hub systemd-networkd[97962]: br-8297310e8a61: Configuring with /run/systemd/network/91-default.network.
@bf bf added the bug 🐛 Programming errors, that need preferential fixing label Aug 1, 2023
@bf
Copy link
Author

bf commented Aug 1, 2023

# cat /run/systemd/network/91-default.network 
# Automatically generated by systemd-network-generator

[Match]
Name=*

[Link]

[Network]
DHCP=ipv4

[DHCP]
# pacman -Fy systemd-network-generator
core/systemd 253.7-1 [installed]
    usr/lib/systemd/systemd-network-generator

@bf
Copy link
Author

bf commented Aug 1, 2023

# cat /etc/default/grub | grep ip
GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=dhcp"

@bf
Copy link
Author

bf commented Aug 1, 2023

mkinitcpio hooks:

HOOKS=(base udev autodetect modconf keyboard keymap block mdadm_udev lvm2 netconf tinyssh encryptssh filesystems fsck)

It's a bug because ip=dhcp tells systemd-networkd to prepare eth0 network connection for encryptssh with mkinitcpio-tinyssh. Once passphrase is manually given via SSH connection, the HDD is decrypted and boot will continue.

Then, later, docker is started. Even later, docker containers are started. Yet systemd-network-generator messes with the docker interfaces.

@bf
Copy link
Author

bf commented Aug 1, 2023

I assume ip=dhcp in CMDLINE only affects the eth* network interfaces which are actually present during boot. But systemd-networkd tries to be clever and creates /run/systemd/network/91-default.network which messes with docker network interfaces, which are not even existing during boot time. I think this is the main reason for me being upset.

@yuwata yuwata added the network label Aug 1, 2023
@bf
Copy link
Author

bf commented Aug 1, 2023

Proposed fix:

  • systemd-network-generator should only create config files for the network devices which are visible at boot.
  • Unmanaged=True for docker*, br-, veth could be shipped with systemd-networkd

@yuwata
Copy link
Member

yuwata commented Aug 1, 2023

@bf Thank you for the report and suggestion.

Proposed fix:

* systemd-network-generator should only create config files for the network devices which are visible at boot.

"visible at boot" is hard to define, but limiting to a class of interfaces sounds good.

* Unmanaged=True for docker*, br-_, veth_ could be shipped with systemd-networkd

That should be done by docker side.

@bf
Copy link
Author

bf commented Aug 1, 2023

@yuwata thanks for quick reply. I've opened https://bugs.archlinux.org/task/79268 - maybe archlinux packagers will add the systemd-networkd config files in their package.

With regards to limiting "systemd-network-generator" to a certain class of interfaces, is this possible? If yes, that'd be the best solution. My basic assumption for systemd-network-generator is to only cover interfaces with their catch-all rule which are present at boot time. The docker interfaces aren't, and therefore shouldn't be covered.

@bf
Copy link
Author

bf commented Aug 2, 2023

The archlinux docker people refuse to fix the known issue [1] because there's a workaround documented in arch wiki [2]
@Toolybird

[1] : https://bugs.archlinux.org/task/79268
[2]: https://wiki.archlinux.org/title/Docker#docker0_Bridge_gets_no_IP_/_no_internet_access_in_containers_when_using_systemd-networkd

@yuwata
Copy link
Member

yuwata commented Aug 2, 2023

OK. Let's do in the way of the first suggestion.

Note, workaround is simple, just disable systemd-network-generator.service (and reboot the system, or remove files under /run/systemd/network).

yuwata added a commit to yuwata/systemd that referenced this issue Aug 2, 2023
…nly physical interfaces

Otherwise, it also matches later created virtual devices, and that
breaks networks generated and managed by container management services,
like docker.

Closes systemd#28626.
@yuwata
Copy link
Member

yuwata commented Aug 2, 2023

Fix is waiting in #28646.

yuwata added a commit to yuwata/systemd that referenced this issue Aug 2, 2023
…nly physical interfaces

Otherwise, it also matches later created virtual devices, and that
breaks networks generated and managed by container management services,
like docker.

Closes systemd#28626.
@bf
Copy link
Author

bf commented Aug 2, 2023

Wow, thank you so much @yuwata! Please let me know if I should test this.

@yuwata
Copy link
Member

yuwata commented Aug 2, 2023

Please let me know if I should test this.

Please, if possible.

@bf
Copy link
Author

bf commented Aug 2, 2023

I compiled your PR:

[root@builder build]# md5sum systemd-network-generator
a5bf7e9033c6c3352192a103b4984c90  systemd-network-generator
[root@builder build]# md5sum /usr/lib/systemd/systemd-network-generator.bak 
d4f25404aaedbc548ba54a0214e2f571  /usr/lib/systemd/systemd-network-generator.bak
[root@builder build]# cp systemd-network-generator /usr/lib/systemd/systemd-network-generator
[root@builder build]# ls -lh /usr/lib/systemd/systemd-network-generator
-rwxr-xr-x 1 root root 107K Aug  2 22:24 /usr/lib/systemd/systemd-network-generator
[root@builder build]# systemctl ^C
[root@builder build]# networkctl 
IDX LINK      TYPE     OPERATIONAL SETUP      
  1 lo        loopback carrier     configured 
  2 enp0s31f6 ether    routable    configured 
  3 docker0   bridge   no-carrier  configuring

3 links listed.
[root@builder build]# systemctl restart systemd-networkd
[root@builder build]# networkctl 
IDX LINK      TYPE     OPERATIONAL SETUP      
  1 lo        loopback carrier     configured 
  2 enp0s31f6 ether    routable    configured 
  3 docker0   bridge   no-carrier  configuring

3 links listed.
[root@builder build]#  reboot

So no change by only restarting the service. Let's see how it looks after reboot:

[root@builder ~]# networkctl
IDX LINK      TYPE     OPERATIONAL SETUP     
  1 lo        loopback carrier     unmanaged
  2 enp0s31f6 ether    routable    configured
  3 docker0   bridge   no-carrier  unmanaged

It seems to have worked!

Please note that also loopback is unmanaged now. I'll start using docker and report back if the other interfaces are unmanaged as well.

@bf
Copy link
Author

bf commented Aug 2, 2023

Seems to also work with other docker devices:
image

@yuwata
Copy link
Member

yuwata commented Aug 2, 2023

@bf Thank you!!

Please note that also loopback is unmanaged now.

Yeah, that's expected and intended.

@bf
Copy link
Author

bf commented Aug 2, 2023

Thanks to you for the quick fix!

peckato1 pushed a commit to peckato1/systemd that referenced this issue Jan 16, 2024
…nly physical interfaces

Otherwise, it also matches later created virtual devices, and that
breaks networks generated and managed by container management services,
like docker.

Closes systemd#28626.

(cherry picked from commit c25aa6c)
nmeyerhans pushed a commit to nmeyerhans/systemd that referenced this issue Jan 21, 2024
…nly physical interfaces

Otherwise, it also matches later created virtual devices, and that
breaks networks generated and managed by container management services,
like docker.

Closes systemd#28626.

(cherry picked from commit c25aa6c)
(cherry picked from commit d39bfd2)
(cherry picked from commit 2c2b89a)
yuwata added a commit to yuwata/systemd that referenced this issue Apr 26, 2024
…nly physical interfaces

Otherwise, it also matches later created virtual devices, and that
breaks networks generated and managed by container management services,
like docker.

Closes systemd#28626.

(cherry picked from commit c25aa6c)
(cherry picked from commit d39bfd2)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug 🐛 Programming errors, that need preferential fixing network
Development

Successfully merging a pull request may close this issue.

2 participants