NIC Management Reverts back to systemd-networkd

[pceiley]

Less than a week ago, NetworkManager stopped managing my NIC and systemd-networkd took over. I follow the steps in the stateless man page that list:

To switch to NetworkManager  for  all  the interfaces, the user should disable and stop systemd-networkd:
          sudo systemctl disable systemd-networkd  sudo  systemctl  stop  systemd-networkd

       Then,  remove  the file /etc/NetworkManager/conf.d/systemd-networkd-un‐managed.conf

       Finally, restart NetworkManager
          sudo systemctl restart NetworkManager

This worked for a while, even after a reboot. However, it has reverted back once again and the file /etc/NetworkManager/conf.d/systemd-networkd-un‐managed.conf has once again been created.

I can see this in the log:

May 15 20:37:10 ricotta systemd-networkd[477]: Enumeration completed
May 15 20:37:10 ricotta audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-networkd comm="systemd" exe="/usr/lib/systemd/systemd" hostn>
May 15 20:37:10 ricotta systemd-networkd[477]: enp3s0: Interface name change detected, enp3s0 has been renamed to eth0.
May 15 20:37:10 ricotta systemd[1]: Started Network Service.
May 15 20:37:10 ricotta systemd-networkd[477]: eth0: Interface name change detected, eth0 has been renamed to enp3s0.

which may provide a clue - the device name might be confusing the issue (device name always ends up as enp3s0).

Frankly, I'm not sure why systemd-networkd is starting at all since I thought I disabled it!

[gvancuts]

Frankly, I'm not sure why systemd-networkd is starting at all since I thought I disabled it!

I'm not familiar with this issue in particular but systemd services can be started by other services that depend on it, even if it's disabled (which I understand really means that auto-starting it at boot is disabled). A more radical way of making sure it never gets started is to use sudo systemctl mask systemd-networkd.

[mrkz]

I managed to reproduce this after I successfully installed clearlinux on a xps13.

• Installed clearlinux
• reboot to installed clearlinux OS on the laptop
• run swupd to update to latest release
• reboot
• Noticed that NetworkManager is no longer managing wired network

The following files were created:

# cat /etc/NetworkManager/conf.d/systemd-networkd-unmanaged.conf
# created by swupd on Tue May 15 2019 09:03:28 AM CDT
[main]
plugins=keyfile

[keyfile]
unmanaged-devices=interface-name:en*;interface-name:eth*;interface-name:host0;interface-name:ve-*;interface-name:vz-*

and

# cat /var/lib/swupd/one-shot-updates/201904-reconfigure-network-manager.sh.done
ls: cannot access '/etc/systemd/network/*.network': No such file or directory
Created symlink /etc/systemd/system/dbus-org.freedesktop.network1.service → /usr/lib/systemd/system/systemd-networkd.service.
Created symlink /etc/systemd/system/multi-user.target.wants/systemd-networkd.service → /usr/lib/systemd/system/systemd-networkd.service.
Created symlink /etc/systemd/system/sockets.target.wants/systemd-networkd.socket → /usr/lib/systemd/system/systemd-networkd.socket.
Created symlink /etc/systemd/system/network-online.target.wants/systemd-networkd-wait-online.service → /usr/lib/systemd/system/systemd-networkd-wait-online.service.
Executed /usr/lib/swupd/one-shot-updates/201904-reconfigure-network-manager.sh on Wed 15 May 2019 09:03:28 AM CDT

[busykai]

@pceiley, unfortunately, you need to repeat the procedure mentioned in man stateless once again. it will not happen to you again.

the reason behind the issue you observed is that, even though we rolled out migration to Network Manager about 3 weeks ago, we had to put a "workaround" in place which kept systemd-networkd running. about a week ago, we finally disabled systemd-networkd and migration script detected the condition and ran. the script assumes you were running systemd-networkd to manage ethernet and "kept" it that way (including enabling it).

[busykai]

@mrkz, the issue you describe is different. the install process is supposed to create .done file without running the migration script. it does not happen, apparently.

[pceiley]

Thanks, @busykai, I'll keep an eye on it and report back. This first occurred on 10 May, after which time I ran the procedure in man stateless to fix. It occurred again on 15 May when I rebooted, but presumably triggered on 14 May as per:

/usr/lib/swupd/one-shot-updates/201904-reconfigure-network-manager.sh: line 39: [: missing `]'
ls: cannot access '/etc/systemd/network/*.network': No such file or directory
Created symlink /etc/systemd/system/dbus-org.freedesktop.network1.service → /usr/lib/systemd/system/systemd-networkd.service.
Created symlink /etc/systemd/system/multi-user.target.wants/systemd-networkd.service → /usr/lib/systemd/system/systemd-networkd.service.
Created symlink /etc/systemd/system/sockets.target.wants/systemd-networkd.socket → /usr/lib/systemd/system/systemd-networkd.socket.
Created symlink /etc/systemd/system/network-online.target.wants/systemd-networkd-wait-online.service → /usr/lib/systemd/system/systemd-networkd-wait-online.service.
Executed /usr/lib/swupd/one-shot-updates/201904-reconfigure-network-manager.sh on Tue May 14 10:20:57 AEST 2019

Given that I only 'fixed' it on the 10th, are you confident that this will not recur?

[busykai]

@pceiley, will definitely appreciate your further input on this. thank you! as long as that .done file is present on the filesystem, it should not happen again: this is how we keep track of the migration state. please let me know if it does happen though.

[pceiley]

@busykai this happened again (actually, it's happened a few times on two boxes I run). Most recently, it occurred today.

$ sudo ls -la /var/lib/swupd/one-shot-updates/201904-reconfigure-network-manager.sh.done
-rw-r--r-- 1 root root 190 May 26 08:16 /var/lib/swupd/one-shot-updates/201904-reconfigure-network-manager.sh.done
$ ls -la /etc/NetworkManager/conf.d/systemd-networkd-unmanaged.conf
-rw-r--r-- 1 root root 204 May 26 08:16 /etc/NetworkManager/conf.d/systemd-networkd-unmanaged.conf

This bug is annoying me, as my IP address changes as a result (falls back to DHCP) and I lose my NFS mounts. I could work around it by setting a fixed address via systemd-networkd, but I shouldn't have to.

[lebensterben]

Same as #755

[busykai]

@pceiley, i'm working on identifying the root cause. it seems like something wipes the state in /var clean. it does not seem to be swupd, i'm looking for the culprit. do you have any hunch as to when the rollback happens?

[busykai]

@lebensterben if you don't mind, I will close #755 in favor of this (older issue).

[lebensterben]

@lebensterben if you don't mind, I will close #755 in favor of this (older issue).

@busykai closed.

_{Sent with GitHawk}

[busykai]

@pceiley, @lebensterben, could you please run the following and post the output here:

stat -c '%n %i' /var/lib /var/lib/swupd /var/lib/swupd/one-shot-updates /var/lib/swupd/one-shot-updates/201904-reconfigure-network-manager.sh.done | while read n i; do echo $n; debugfs -R 'stat <'"$i"'>' /dev/sda3 2>/dev/null | tail -n +7 | head -n 4; echo ; done

it will need debufs which is provided by storage-utils. also, if needed replace /dev/sda3 with the partition you mount on / (as per findmnt /).

i appreciate your willingness to help me track down this issue. i understand how annoying it is and hope to nail it down soon.

[lebensterben]

@busykai

/var/lib
 ctime: 0x5cdfa288:401c25d8 -- Sat May 18 02:13:28 2019
 atime: 0x5cea4386:d2d77d94 -- Sun May 26 03:43:02 2019
 mtime: 0x5cdfa288:401c25d8 -- Sat May 18 02:13:28 2019
crtime: 0x5c711ad6:e5886a60 -- Sat Feb 23 05:05:10 2019

/var/lib/swupd
 ctime: 0x5ce8e27c:20c095c4 -- Sat May 25 02:36:44 2019
 atime: 0x5cea438b:7fde0238 -- Sun May 26 03:43:07 2019
 mtime: 0x5ce8e27c:20c095c4 -- Sat May 25 02:36:44 2019
crtime: 0x5c711ae9:ea11e0d0 -- Sat Feb 23 05:05:29 2019

/var/lib/swupd/one-shot-updates
 ctime: 0x5ce8e5d9:60513be8 -- Sat May 25 02:51:05 2019
 atime: 0x5ce8dff7:22756448 -- Sat May 25 02:25:59 2019
 mtime: 0x5ce8e5d9:60513be8 -- Sat May 25 02:51:05 2019
crtime: 0x5ce72435:602a86bc -- Thu May 23 18:52:37 2019

/var/lib/swupd/one-shot-updates/201904-reconfigure-network-manager.sh.done
 ctime: 0x5ce72435:7562732c -- Thu May 23 18:52:37 2019
 atime: 0x5ce8e48c:12f1b4c0 -- Sat May 25 02:45:32 2019
 mtime: 0x5ce72435:7562732c -- Thu May 23 18:52:37 2019
crtime: 0x5ce72435:60e19ff8 -- Thu May 23 18:52:37 2019

[busykai]

@lebensterben, @pceiley please remove /usr/lib/tmpfiles.d/swupd-client.conf. i will fix the issue properly in a couple days. thank you for your patience.

[pceiley]

Thanks @busykai! Will the file be re-created once the issue is fixed? I'm keen to ensure /var/lib/swupd is kept under control long term.

[busykai]

@pceiley, the fix will remove the culprit altogether (in review right now). Until the fix is available, only running swupd verify --fix will restore it, which is not something run automatically (well, fingers crossed). I’ll let you guys know when the fix is out.

[busykai]

The issue has been fixed in 29680 which is live already. I appreciate your help, @lebensterben, @pceiley, with tracking it down. I'm going to close it. Please feel free to reopen it if for whatever reason (sigh) it resurfaces again.

[pceiley]

Seems to have done the trick. Really appreciate your persistence with this - Thanks, @busykai!

As an aside, I hope my swupd cache doesn't just grow indefinitely now...

[busykai]

Thanks @pceiley! swupd cache will still be cleaned weekly, but with swupd clean which is a sentient, comparing to tmpfiles approach. :) on average, i do not expect the cache to dramatically increase, though the files needed for proper operation of swupd will be kept (resulting in faster updates) and it will not drop to 0. at any rate, i will be keeping an eye on it.

[lebensterben]

@busykai Is this problem coming back? :(

https://community.clearlinux.org/t/software-center-thinks-im-not-connected-to-internet-but-terminal-and-firefox-does/1158