qubes-dom0-update does not initiate dnf install (correct download, but no install), only affecting some Qubes systems

[Aekez]

Qubes OS version:

Qubes 4.0.

Affected component(s):

• dom0
  • qubes-dom0-update process (correctly downloads dom0 updates).
  • To be exact, "dnf install /var/lib/qubes/updates/rpm/*.rpm" logic is not initiated.

---

Steps to reproduce the behavior:

• This only affects some Qubes systems, i.e. other Qubes systems in the same building does not have this issue.
• The exact trigger to this behavior is unknown, but once it triggers its permanent for the Qubes system in question.
• It happens 100% of the time (every time new dom0 updates are available) on the affected Qubes system, but it happens 0% of the time on the other un-affected Qubes systems.
• I've only seen one Qubes system with this problem so far.
• I'm speculating maybe re-installing Qubes on this hardware might fix it, but it might be a good idea to report this anyway, just in case.

Expected behavior:

qubes-dom0-update to automatically verify and install the otherwise correctly downloaded dom0 updates.

Actual behavior:

• qubes-dom0-update downloads the updates correctly, but does not install them. This bug is silent, it is easy for the user to overlook that there were updates if they do not pay attention to the download, and realize the fact that no install happens after downloading.
• It is uncertain if the hash/key verification happens either, but running rpm -K /var/lib/qubes/updates/rpm/*.rpm shows the downloaded files are ok "whole and matching".

General notes:

Manual fix is easy though, but only if the user realizes the problem and the fix

• The manual fix
  • Maybe redundant, but using just to be safe rpm -K /var/lib/qubes/updates/rpm/*.rpm
  • Then sudo dnf install /var/lib/qubes/updates/rpm/*.rpm
  • Everything presumably keeps working normally.
    • Except the affected system means every new dom0 update will have the same problem, therefore the user needs to stay awake when running dom0 updates, and be sure not to overlook any downloaded dom0 updates, and then install them manually.

A potential critical problem

• The fact the user might not realize the downloaded dom0 updates were not installed, might make this issue more common than realized, and this could also potentially (speculation) be a host to some of the other dom0 issues reported.

---

Related issues:

[marmarek]

Can you paste example output from sudo qubes-dom0-update from affected system? Does qubes-dom0-update --clean help?

[marmarek]

Can you paste example output from sudo qubes-dom0-update from affected system? Does qubes-dom0-update --clean help?

[Aekez]

Certainly, as requested I did it with today's updates' from current-testing.
As requested I did without --clean first, followed up and did the exact same steps again but with added --clean option.
I'll wait with manually installing these updates for 48 hours in case more information is needed, or wait longer if requested to do so.

• Log without --clean option
  • Log uploaded here qubes-dom0-update.log
    • sudo qubes-dom0-update
    • qubes-dom0-update final line reports "No updates available".
    • However doing ls /var/lib/qubes/update/rpm shows the updates are indeed available.
    • Further doing rpm -K /var/lib/qubes/update/rpm/*.rpm to check hash/keys, looks ok.
    • Everything above is included in the uploaded log.

• Log with the --clean option
  • Log uploaded here qubes-dom0-update-with--clean.log
    • sudo qubes-dom0-update --clean
    • qubes-dom0-update final line reports "No updates available".
    • However doing ls /var/lib/qubes/update/rpm shows the updates are indeed available.
    • Further doing rpm -K /var/lib/qubes/update/rpm/*.rpm to check hash/keys, looks ok.
    • Everything above is included in the uploaded log.

[Aekez]

Certainly, as requested I did it with today's updates' from current-testing.
As requested I did without --clean first, followed up and did the exact same steps again but with added --clean option.
I'll wait with manually installing these updates for 48 hours in case more information is needed, or wait longer if requested to do so.

• Log without --clean option
  • Log uploaded here qubes-dom0-update.log
    • sudo qubes-dom0-update
    • qubes-dom0-update final line reports "No updates available".
    • However doing ls /var/lib/qubes/update/rpm shows the updates are indeed available.
    • Further doing rpm -K /var/lib/qubes/update/rpm/*.rpm to check hash/keys, looks ok.
    • Everything above is included in the uploaded log.

• Log with the --clean option
  • Log uploaded here qubes-dom0-update-with--clean.log
    • sudo qubes-dom0-update --clean
    • qubes-dom0-update final line reports "No updates available".
    • However doing ls /var/lib/qubes/update/rpm shows the updates are indeed available.
    • Further doing rpm -K /var/lib/qubes/update/rpm/*.rpm to check hash/keys, looks ok.
    • Everything above is included in the uploaded log.

[marmarek]

Are you sure you don't have those versions already installed? For me it looks like it downloads packages which you already have. Check for example rpm -q xen.

[marmarek]

Are you sure you don't have those versions already installed? For me it looks like it downloads packages which you already have. Check for example rpm -q xen.

[Aekez]

Note the user is different because I originally replaced it with "user" due for anonymity, but I thought a screenshot might be better, hence the difference.

This bug happens every time there are new dom0 updates available for at least some weeks now. It never made any difference if I use --clean, --refresh or without, it still won't trigger the dnf install process. The downloading of updates to dom0 always seem to work perfectly though.

[Aekez]

Note the user is different because I originally replaced it with "user" due for anonymity, but I thought a screenshot might be better, hence the difference.

This bug happens every time there are new dom0 updates available for at least some weeks now. It never made any difference if I use --clean, --refresh or without, it still won't trigger the dnf install process. The downloading of updates to dom0 always seem to work perfectly though.

[Aekez]

I just remembered there is one difference for this machine compared to other Qubes machines that might be important to mention, since it has an integrated Ryzen mobile GPU, installing Qubes normally was not possible (no-graphics) due to kernel version the Linux-firmware version included in the Qubes installer.

I used the unix DD command to disc-clone everything on a different Qubes machine. First I updated everything on the working hardware, and installed the newest available kernel-latest and Linux-firmware in the Qubes repo's at the time. Then I moved everything with DD to the Ryzen laptop then made the system work.

So the act of using DD, could maybe change the dom0 integrity? I do not remember if these two overlap in time though, but I thought it might be important to mention just in case my scenario qubes-dom0-update issue is unique, and if that is indeed the case then this can be closed and I can just wait for the newest Qubes installer (I don't mind at all doing manual installs until then).

[Aekez]

I just remembered there is one difference for this machine compared to other Qubes machines that might be important to mention, since it has an integrated Ryzen mobile GPU, installing Qubes normally was not possible (no-graphics) due to kernel version the Linux-firmware version included in the Qubes installer.

I used the unix DD command to disc-clone everything on a different Qubes machine. First I updated everything on the working hardware, and installed the newest available kernel-latest and Linux-firmware in the Qubes repo's at the time. Then I moved everything with DD to the Ryzen laptop then made the system work.

So the act of using DD, could maybe change the dom0 integrity? I do not remember if these two overlap in time though, but I thought it might be important to mention just in case my scenario qubes-dom0-update issue is unique, and if that is indeed the case then this can be closed and I can just wait for the newest Qubes installer (I don't mind at all doing manual installs until then).

[marmarek]

I don't think dd could cause this problem.

Lets see what cause qubes-dom0-update think there are no updates, try this: sudo bash -x /usr/bin/qubes-dom0-update

[marmarek]

I don't think dd could cause this problem.

Lets see what cause qubes-dom0-update think there are no updates, try this: sudo bash -x /usr/bin/qubes-dom0-update

[Aekez]

I don't think dd could cause this problem.

That's reassuring, it seems like it would have been a wide area of uncertainty if it could.

Here's the log when running sudo bash -x /usr/bin/qubes-dom0-update
qubes-dom0-update-with-bash-x.log

[Aekez]

I don't think dd could cause this problem.

That's reassuring, it seems like it would have been a wide area of uncertainty if it could.

Here's the log when running sudo bash -x /usr/bin/qubes-dom0-update
qubes-dom0-update-with-bash-x.log