Strange messages when upgrading dom0 (a first run crashed my bootloader/system; second: see photo)

[Wikinaut]

Qubes OS version:

R4.0

Affected component(s):

While upgrading dom0, I saw for 1..2 minutes these messages, see photo.

During the update (*), I was impatient, pressed ctrl-c several times and fully crashed (destroyed bootloader, so that my computer BIOS could not find a bootable system anymore (tried UEFI and legacy Boot) my system, so that I had to reinstall Qubes 4.0!

The photo was taken after a full reinstallation, as I updated dom0 as one of the next steps after installation. The strange messages were the same as in (*), so it was somehow reproducible.

I took the photo.

Now I waited. After about 1..2 minutes, the upgrade went through. After the upgrade, the upgrade window closed automatially.

There was no further prompt or information!

Question 1: Is the automatic closing correct?

I clicked a second time update dom0. There was this first "downloading window" shown for about 1 minute, and then the window closes.

No confirmation, no message, that dom0 is up-to-date, or that no upgrade is needed or the like.

Question 2: Is this correct behaviour of the dom0 updater (auto-closing if upgrade is not needed?)

---

Steps to reproduce the behavior:

Expected behavior:

No error messages, or a stop of the upgrade.

Actual behavior:

Error messages, which puzzles me a lot.

General notes:

---

Related issues:

[sebastianlutter]

I have the same problem after upgrading dom0 with this error from your screenshot.

I booted a ubuntu live cd, and got access to my file data with:

cryptsetup luksOpen /dev/sda2 qubes
lvmdiskscan
# lists all partitions available including dom0 root, but it is inactive in the LVM
# run the following to activate all
vgchange -ay qubes_dom0
# then I was able to mount and chroot dom0 root (bind mount dev and sys from host system)
mount /dev/qubes_dom0/root ~/qubes-dom0
chroot qubes-dom0
mount /boot/efi
mount -t proc proc /proc

When I try to do a backup (so I can reinstall and recover) I got:

bash-4.3# qvm-backup --passphrase-file - --compress /storage/
Traceback (most recent call last):
  File "/usr/bin/qvm-backup", line 5, in <module>
    sys.exit(main())
  File "/usr/lib/python3.5/site-packages/qubesadmin/tools/qvm_backup.py", line 162, in main
    'dom0', 'admin.backup.Info', profile_name)
  File "/usr/lib/python3.5/site-packages/qubesadmin/app.py", line 466, in qubesd_call
    client_socket.connect(qubesadmin.config.QUBESD_SOCKET)
FileNotFoundError: [Errno 2] No such file or directory

Also tried to reinstall my UEFI entry with:
efibootmgr -v -c -u -L Qubes -l /EFI/qubes/xen.efi -d /dev/sda -p 1 "placeholder /mapbs /noexitboot"
(taken from https://www.qubes-os.org/doc/uefi-troubleshooting/#installation-finished-but-qubes-boot-option-is-missing-and-xencfg-is-empty)

But this also does not help.

How I can backup my data or recover the boot system? I made no backup, so I'm highly motivated to solve this. Any hints?

[sebastianlutter]

I have the same problem after upgrading dom0 with this error from your screenshot.

I booted a ubuntu live cd, and got access to my file data with:

cryptsetup luksOpen /dev/sda2 qubes
lvmdiskscan
# lists all partitions available including dom0 root, but it is inactive in the LVM
# run the following to activate all
vgchange -ay qubes_dom0
# then I was able to mount and chroot dom0 root (bind mount dev and sys from host system)
mount /dev/qubes_dom0/root ~/qubes-dom0
chroot qubes-dom0
mount /boot/efi
mount -t proc proc /proc

When I try to do a backup (so I can reinstall and recover) I got:

bash-4.3# qvm-backup --passphrase-file - --compress /storage/
Traceback (most recent call last):
  File "/usr/bin/qvm-backup", line 5, in <module>
    sys.exit(main())
  File "/usr/lib/python3.5/site-packages/qubesadmin/tools/qvm_backup.py", line 162, in main
    'dom0', 'admin.backup.Info', profile_name)
  File "/usr/lib/python3.5/site-packages/qubesadmin/app.py", line 466, in qubesd_call
    client_socket.connect(qubesadmin.config.QUBESD_SOCKET)
FileNotFoundError: [Errno 2] No such file or directory

Also tried to reinstall my UEFI entry with:
efibootmgr -v -c -u -L Qubes -l /EFI/qubes/xen.efi -d /dev/sda -p 1 "placeholder /mapbs /noexitboot"
(taken from https://www.qubes-os.org/doc/uefi-troubleshooting/#installation-finished-but-qubes-boot-option-is-missing-and-xencfg-is-empty)

But this also does not help.

How I can backup my data or recover the boot system? I made no backup, so I'm highly motivated to solve this. Any hints?

[sebastianlutter]

It seems like this only happens when you press Strg + c when seeing the error message. It seems stuck, but will continue after some minutes (info from starius from freenode irc) of waiting.

Here is his gist:
https://gist.github.com/starius/1a88b8a16bb597e50af40834821bec16

[sebastianlutter]

It seems like this only happens when you press Strg + c when seeing the error message. It seems stuck, but will continue after some minutes (info from starius from freenode irc) of waiting.

Here is his gist:
https://gist.github.com/starius/1a88b8a16bb597e50af40834821bec16

[Wikinaut]

@sebastianlutter

• You are right in saying, that the process continued after some minutes.
• You are not correct in the part, regarding Strg + c: in my case, the message came without having pressed Strg +c (this was after having re-installed Qubes).

Question

When you now again request a dom0-upgrade, please tell me, whether the window "downloading upgrades" appears and fully disappears after a while, without any information that or that not an upgrade is necessary or has been done.

[Wikinaut]

@sebastianlutter

• You are right in saying, that the process continued after some minutes.
• You are not correct in the part, regarding Strg + c: in my case, the message came without having pressed Strg +c (this was after having re-installed Qubes).

Question

When you now again request a dom0-upgrade, please tell me, whether the window "downloading upgrades" appears and fully disappears after a while, without any information that or that not an upgrade is necessary or has been done.

[sebastianlutter]

In the meantime I tried to manually reinstall the kernel package of dom0 in chroot from ubuntu live cd, but does not work:

# download kernel package
wget https://ftp.qubes-os.org/repo/yum/r4.0/current/dom0/fc25/rpm/kernel-4.14.35-1.pvops.qubes.x86_64.rpm

# copy to some folder in dom0 root partition (i.e. /stoarage)
# reinstall the kernel
bash-4.3# dnf reinstall /storage/kernel-4.14.35-1.pvops.qubes.x86_64.rpm 

[sebastianlutter]

In the meantime I tried to manually reinstall the kernel package of dom0 in chroot from ubuntu live cd, but does not work:

# download kernel package
wget https://ftp.qubes-os.org/repo/yum/r4.0/current/dom0/fc25/rpm/kernel-4.14.35-1.pvops.qubes.x86_64.rpm

# copy to some folder in dom0 root partition (i.e. /stoarage)
# reinstall the kernel
bash-4.3# dnf reinstall /storage/kernel-4.14.35-1.pvops.qubes.x86_64.rpm 

[sebastianlutter]

@Wikinaut I still try to rescue the system, and therefore have no running Qubes OS 4.0 at the moment to test it. As far as I remember it is normal that it diappears without further notice when no update is available.

[sebastianlutter]

@Wikinaut I still try to rescue the system, and therefore have no running Qubes OS 4.0 at the moment to test it. As far as I remember it is normal that it diappears without further notice when no update is available.

[Wikinaut]

@sebastianlutter danke

[Wikinaut]

@sebastianlutter danke

[sebastianlutter]

Since I know no way to debug why after bios I see Xen is started and fails after a half second I backup my data now manually from ubuntu live cd.

Here some infos if someone else has a similar problem:

• after you run vgchange -ay qubes_dom0 from previous posting all private and root partitions of your VMs can be accessed as well
• all private partitions can be mounted directly (i.e. mount /dev/qubes_dom0/vm-personal-private /mnt)
• the root partition block devices contain a partition table and can be mounted with offset or accessed using partx/kpartx

[sebastianlutter]

Since I know no way to debug why after bios I see Xen is started and fails after a half second I backup my data now manually from ubuntu live cd.

Here some infos if someone else has a similar problem:

• after you run vgchange -ay qubes_dom0 from previous posting all private and root partitions of your VMs can be accessed as well
• all private partitions can be mounted directly (i.e. mount /dev/qubes_dom0/vm-personal-private /mnt)
• the root partition block devices contain a partition table and can be mounted with offset or accessed using partx/kpartx

[RefinedSoftwareLLC]

@Wikinaut @sebastianlutter
I had similar issue when a kernel install failed because of lack of disk space on the drive with the boot kernels. I was able to fix it with:

Burn Qubes-R4.0-x86_64.iso to a DVD & Boot from it.
Select "Troubleshooting", then "Rescue a Qubes system".
The only options are to be dropped into a shell, even though the fix can be automated.
chroot /mnt/sysimage
vim /boot/efi/EFI/qubes/xen.cfg
Change default=... to have the ... be one of the options listed below in the same file.
Reboot your computer.

[RefinedSoftwareLLC]

@Wikinaut @sebastianlutter
I had similar issue when a kernel install failed because of lack of disk space on the drive with the boot kernels. I was able to fix it with:

Burn Qubes-R4.0-x86_64.iso to a DVD & Boot from it.
Select "Troubleshooting", then "Rescue a Qubes system".
The only options are to be dropped into a shell, even though the fix can be automated.
chroot /mnt/sysimage
vim /boot/efi/EFI/qubes/xen.cfg
Change default=... to have the ... be one of the options listed below in the same file.
Reboot your computer.

[Wikinaut]

@RefinedSoftwareLLC I wanted simply to use Qubes, 4.0, and not to fix these strange issues (3.2 was fine, 4.0 is not).

[Wikinaut]

@RefinedSoftwareLLC I wanted simply to use Qubes, 4.0, and not to fix these strange issues (3.2 was fine, 4.0 is not).

[RefinedSoftwareLLC]

I agree with you, that is why I made #3973 and linked your issue. I hope your system is up again!

[RefinedSoftwareLLC]

I agree with you, that is why I made #3973 and linked your issue. I hope your system is up again!

[Wikinaut]

@RefinedSoftwareLLC yes, up again.

[Wikinaut]

@RefinedSoftwareLLC yes, up again.

[marmarek]

This looks to be a duplicate of #3756

[marmarek]

This looks to be a duplicate of #3756