Installer option to automatically rescue existing install from common boot issues

[RefinedSoftwareLLC]

Installer option to automatically rescue existing install from common boot issues.
This is an industry standard expected by normal users and mandatory for a good user experience (even if Qubes OS is flawless, other things can corrupt the boot sequence).

Qubes OS version:

Qubes-R4.0-x86_64.iso (I burned to DVD and installed onto a blank computer)

Affected component(s):

Entire Qubes OS as it doesn't boot to the screen asking for your encrypted hard drive password.
Qubes-R4.0-x86_64.iso's "Rescue a Qubes system" needs updated with this new feature.

---

Steps to reproduce the behavior:

Install Qubes OS R4.0.
sudo qubes-dom0-update or sudo qubes-dom0-update --enablerepo=qubes-dom0-security-testing

During the step Install kernel-qubes-vm-1000:4.14.41-1.pvops.qubes.x86_64 this error displays:
cp: error writing '/boot/efi/EFI/qubes/initramfs-4.14.41-1.props.qubes.x86_64.img': no space left on device.
Note /boot/efi/EFI/qubes/initramfs-4.14.41-1.props.qubes.x86_64.img may exist, taking up the rest of the free space, but is only a corrupted part of the intended file.

Reboot your computer.

[...] initramfs unpacking failed: read error 
[...] Unable to mount root fs, Kernal Panic
[...] ... 
[...] Kernal Offset disabled
[...] Kernal Panic: Not Syncing

Burn Qubes-R4.0-x86_64.iso to a DVD & Boot from it.
Select "Troubleshooting", then "Rescue a Qubes system".
The only options are to be dropped into a shell, even though the fix can be automated.
chroot /mnt/sysimage
vim /boot/efi/EFI/qubes/xen.cfg
Change default=... to have the ... be one of the options listed below in the same file.
Reboot your computer.

Expected behavior:

1. Qubes OS doesn't boot to the to the screen asking for your encrypted hard drive password.
2. Burn Qubes-R4.0-x86_64.iso to a DVD & Boot from it.
3. Select "Troubleshooting" then "Rescue a Qubes system".
4. The Rescue Environment displays:

======================================================
Rescue

The rescue environment will now attempt to find your Linux installation.  Choose '1' to automatically rescue from common issues and downloading files as needed. This will NOT use whonix/tor.  Choose '2' for an offline automatic rescue.  Choose '3' to have your installation mounted under the directory : /mnt/sysimage. You can then manually make any changes required to your system.  Choose '4' to mount as read-only instead of read-write.
If for some reason this process does not work choose '5' to skip directly to a shell.

 1) Automatic (Download as needed)

 2) Automatic (Offline)

 3) Manual

 4) Manual (Read-only mount)

 5) Manual (Skip to shell)

 6) Quit (Reboot)

Please make a selection from the above:  1
======================================================

Selecting '1' for automatic, will do the following rescue steps:

Note: Before each rescue step makes its change:

• Verify this step's problem exists, otherwise skip this step.
• Verify they will not run out of disk space while making their change.
• If boot up fix requires more disk space, free up disk space (one way is by deleting an unused kernel) but try to not ask for permission (for safe files to delete) unless you detect an anomaly (like a duel boot needs the kernel files your are deleting).

[Admins, feel free to edit this post to update these checkboxes as needed]

• If the mounted drive /boot/efi/ is full, before making a change, free up enough space for that change to not error out.
• Verify /boot/efi/EFI/qubes/xen.cfg the default=... has ... set to one of the options listed below in the same file and one that passes the following tests. (If listed, use non-corrupt rescue kernel, #3624)
• Note: Include in the Qubes OS Install iso a list of hashes (SHA256?) for all kernel files (That Qubes has ever provided in Installer or dom0 update) and maybe their file sizes too.
• Try to download updated hashes list (but continue rescue either way).
• Verify default kernel has all the files needed for it to boot and they match their hashes.
• If a kernel file doesn't match known hashes, verify there is enough free space (freeing space as needed) and then try to redownload all corrupt files.
• If no kernel is usable, install and use a rescue kernel stored on the Qubes Installer iso (See #3624) (Free space as needed)

More rescue steps to add:

• When needed, automatically apply fixes detailed in #3700 (Disable appvms automatically booting).
• When needed, automatically apply fixes detailed in #3915.
• With each individual fix when its needed, automatically apply fixes detailed in https://www.qubes-os.org/doc/uefi-troubleshooting/ (Automate as many individual fixes possible)
• When needed, automatically apply fixes detailed in #3949.
• When needed, automatically apply fixes detailed in #3903.
• When needed, automatically apply fixes detailed in #3790.
• When needed, automatically apply fixes detailed in #3690.
• When needed, automatically apply fixes detailed in #3619 (Detach usb devices when needed).
• When needed, automatically apply fixes detailed in #3897 (when fix is found).
• When needed, automatically apply know fixes to as many types of kernel panics as possible.
• When needed, automatically apply fixes detailed in #3972. (Detect current UTC time from NTP, check if hwclock is UTC or localtime, & update hwclock as needed)

Actual behavior:

During sudo qubes-dom0-update the step Install kernel-qubes-vm-1000:4.14.41-1.pvops.qubes.x86_64 doesn't clean itself up after the error cp: error writing '/boot/efi/EFI/qubes/initramfs-4.14.41-1.props.qubes.x86_64.img': no space left on device.
Reboot entire computer.
Qubes OS doesn't boot to the screen asking for your encrypted hard drive password.
Burn Qubes-R4.0-x86_64.iso to a DVD & Boot from it.
Select "Troubleshooting", then "Rescue a Qubes system".
The only options are to be dropped into a shell, even though the fix can be automated.
Google error & ask community support. Community support implies that it is normal for the Qubes OS recovery DVD to not be able to automatically fix or rollback a dom0 update (from following a Qubes Security Bulletin) nor any common boot issue. A common user then hates Qubes OS and installs a different operating system.

[RefinedSoftwareLLC]

Per: https://groups.google.com/forum/#!topic/qubes-users/JwLKyzg32ao

• When needed, automatically remove rd.qubes.hide_all_usb Kernel Param, only if no PS/2 keyboard is in use but a usb one is, as a working keyboard is needed to enter hard drive password to boot Qubes OS. (Maybe their old PS/2 keyboard broke, they bought a new USB keyboard, now they are running Rescue to fix not being able to boot with the new keyboard.) Remove rd.qubes.hide_all_usb from /etc/default/grub and/or /boot/efi/EFI/qubes/xen.cfg then if needed run sudo grub2-mkconfig -o /boot/grub2/grub.cfg and/or sudo dracut -f /boot/efi/EFI/qubes/initramfs-$(uname -r).img $(uname -r).

[RefinedSoftwareLLC]

Per: https://groups.google.com/forum/#!topic/qubes-users/JwLKyzg32ao

• When needed, automatically remove rd.qubes.hide_all_usb Kernel Param, only if no PS/2 keyboard is in use but a usb one is, as a working keyboard is needed to enter hard drive password to boot Qubes OS. (Maybe their old PS/2 keyboard broke, they bought a new USB keyboard, now they are running Rescue to fix not being able to boot with the new keyboard.) Remove rd.qubes.hide_all_usb from /etc/default/grub and/or /boot/efi/EFI/qubes/xen.cfg then if needed run sudo grub2-mkconfig -o /boot/grub2/grub.cfg and/or sudo dracut -f /boot/efi/EFI/qubes/initramfs-$(uname -r).img $(uname -r).

[RefinedSoftwareLLC]

• Automatic Rescue should save a human readable log of all the changes it made (if it has access to a hard drive with free space?). This allows debugging if something goes wrong, but also for self taught users to see what was needed to fix their system, and power users to manually revert changes as they wish (maybe to fix a non-standard duel boot that broke when getting Qubes OS to boot).

[RefinedSoftwareLLC]

• Automatic Rescue should save a human readable log of all the changes it made (if it has access to a hard drive with free space?). This allows debugging if something goes wrong, but also for self taught users to see what was needed to fix their system, and power users to manually revert changes as they wish (maybe to fix a non-standard duel boot that broke when getting Qubes OS to boot).