crash after sudo dnf system-upgrade reboot

[rickyzhang82]

Upgrade to fc24 from 4.5.7-200.fc23.x86_64
After download package, execute 'sudo dnf system-upgrade reboot'
I don't see new option in grub as fedup did befoe, which is expected.
System boots but immediately crash and reboot.
How can I grab the log of boot? It is too fast to see error message.
thanks.

[keszybz]

You should be able to find the logs in journalctl output. Either try sudo dnf system-upgrade log or sudo journalctl -b -N, where -N is -1, -2, -3, etc.

[rickyzhang82]

well, I checked man of journalctl. In my personal opinion, the option doesn't work as it states.

• -b -0 option doesn't show last boot.
• --this-boot option doesn't show this boot.

I have to manually dig into log. In any case, I don't see any special ERROR in boot process.
I don't know why it reboots without launching install.

[keszybz]

-b -0 should should current boot, the same as --this-boot. What output do you see?

[rickyzhang82]

I filed bugzilla report - https://bugzilla.redhat.com/show_bug.cgi?id=1350120
Boot log - https://bugzilla.redhat.com/attachment.cgi?id=1172540
/var/lib/dnf/system-upgrade.json - https://bugzilla.redhat.com/attachment.cgi?id=1172541

[rickyzhang82]

I commented out reboot() call in run_reboot(). I did see symlink '/system-update' created in root.
After manually executing 'sudo systemctl reboot', it reboot and saw 'dnf-system-upgrade.service' is started by systemd. But it exist and automatically reboot again.

Symlink got removed it. So, I believe it did call run_upgrade(). But somehow, it didn't run through the rest in run_upgrade()

I have trouble to find dnf-system-upgrade log message from journalctl. Do you know how? Thanks!

[rickyzhang82]

dnf-system-upgrade.service did remove symlink again. So it may not be run_upgrade() remove it. I still don't figure it out how to get the log message from dnf-system-upgrade plugin.

ExecStopPost=/usr/bin/rm -fv /system-update

[rickyzhang82]

I think I found the culprit in the boot log. Is it possible one of upgrade service /usr/libexec/fwupd/fwupd may bring down the whole system upgrade. I tried to disable fwupd-offline-update.service but it doesn't seem to disable it.

Jun 26 11:22:52 gtx dbus[1055]: [system] Activating service name='org.freedesktop.fwupd' (using servicehelper)
Jun 26 11:22:52 gtx org.freedesktop.fwupd[1055]: /usr/libexec/fwupd/fwupd: symbol lookup error: /lib64/libgusb.so.2: undefined symbol: libusb_get_parent
Jun 26 11:22:52 gtx dbus[1055]: [system] Activated service 'org.freedesktop.fwupd' failed: Launch helper exited with unknown return code 127
Jun 26 11:22:52 gtx fwupdmgr[1047]: Failed to connect to fwupd: Error calling StartServiceByName for org.freedesktop.fwupd: GDBus.Error:org.freedesktop.DBus.Error.Spawn.Chil
Jun 26 11:22:52 gtx systemd[1]: fwupd-offline-update.service: Main process exited, code=exited, status=1/FAILURE
Jun 26 11:22:52 gtx systemd[1]: fwupd-offline-update.service: Unit entered failed state.
Jun 26 11:22:52 gtx systemd[1]: fwupd-offline-update.service: Triggering OnFailure= dependencies.
Jun 26 11:22:52 gtx systemd[1]: fwupd-offline-update.service: Failed with result 'exit-code'.

[rickyzhang82]

The issue is resolved. It turns out that I have installed different version of libusb in /usr/local/lib for kinect, which caused fwupd service failed. The failure of fwupd caused system upgrade service terminates. Therefore, dnf-system-upgrade plugin didn't finish.

Although it appears that symlink got delete, it actually wiped out by systmed in dnf-system-upgrade.service post exec.

I'd suggest that add a new internal state in dnf-system-upgrade plugin to indicate that run_upgrade didn't get launch because of interruption by other system update service.