[{Self}-SOLVED]: Following Successful Qubes (R3.1-rc2) Installation, Operation, and Initial Rebooting, Qubes CANNOT Boot from an USB HDD, Once the USB HDD Cable Has Been Disconnected

[HardenedArray]

UPDATE: Proposed Qubes External USB HDD EFI Booting Solution:

Following eight (8) days of languishing without a single developer comment I have identified one known-to-work solution to the Qubes R3.1-rc2 external USB HDD EFI booting problem(s) I discussed below:

+++++++++++++++++++

The process, if you cannot get Qubes, following an error-free, completed installation, to boot from an USB HDD for any reason:

1. Download the most recent *.iso of the 'mandatory-to-own', Gentoo-based, SystemRescueCD, from http://www.system-rescue-cd.org/SystemRescueCd_Homepage.
2. If you know about Larry The Cow and his (sometimes pregnant) compilation powers, you're good to go. If not, keep the faith in loads of SmartGuys who have paid the price of admission to 'free' security (and especially toward those super-hard, Gentoo Hardened devs), and repeat, three times: In Larry, When Pregnant, We Trust. ;-3
3. I cannot accurately recount the number of times SystemRescueCD has 'rescued' my encrypted HDDs, but the actual number is, just shy of: innumerable. Highly Recommended Code, especially if you like to test, and FUBAR 'stuff.'
4. Following verification, install, on a USB stick, the SystemRescueCD*.iso, employing the exact USB installation method detailed at: http://www.system-rescue-cd.org/Sysresccd-manual-en_How_to_install_SystemRescueCd_on_an_USB-stick.
5. Do not use 'dd', nor unetbootin, nor pendrivelinux, nor: another installation 'method,' if you expect to have a working copy of SystemRescureCD. In the event you were not paying attention: Re-read Step 4. Additionally, any old USB stick will suffice. The *.iso is less than 500 MiB.
6. With your Qubes external USB HDD and your SystemRescueCD USB stick connected, fire up your PC from a cold shutdown.
7. During the Boot sequence, select to boot from the SystemRescueCD USB stick.
8. Select the default, US keymap, #41.
9. Scroll/arrow down, select: Directly start the graphical environment.
10. Use Accessories, SpaceFM, Devices, to verify which /dev/ udev has assigned to your Qubes USB HDD.
11. From the root terminal, to see the currently available, ordered, boot options, run:

efibootmgr -v

Twelve (Welcome to GitHub's Broken List Numbering System). If you have no experience with efibootmgr, now would be an excellent time to read:

man efibootmgr

Thirteen. Now, assuming your Qubes USB HDD is recognized by udev as /dev/sdc,
run:

efibootmgr -c --disk /dev/sdc -l \\EFI\\qubes\\xen-4.6.0.efi -L Welcome-To-Prison-Hillary:Proper-OpSec-Is-A-Bitch--as-is-GRANDIOSE-Ego--and-We-Told-You-So--YEARS-Ago!

Note A: We write EFI bootcode to devices, not partitions.

Note B: Notice my triple-use of double-backslashes in the command above. There were: NO typographical errors.

Note C: If you are (for some reason) a Fan-Girl of very-likely-to-be-indicted 'I do secure Classified eMail' US Presidential-hopeful-nominee, Hillary 'Benghazi-video=excuse-for-4-American-embassy-murders-by-ridiculously-unskilled-Radical-Muslim-Pussies' Clinton, and you are (unsurprisingly) SJW-angry, remain calm, as it is possible to substitute another BigLie(tm), as in:

efibootmgr -c --disk /dev/sdc -l \\EFI\\qubes\\xen-4.6.0.efi -L Despite-Decades-Of-Weather-Satellite-Temperature-Monitoring-Records-To-The-Contrary:I-Believe-In-Global-Climate-Warming:Because-All-My-Friends-Do--and--I-Could-Lose-My-Job-If-I-Challenge-Politically-Correct-Thought

Note D: Assuming /dev/sdb is your target HDD, this will (also) work:

efibootmgr -c --disk /dev/sdb -l \\EFI\\qubes\\xen.efi -L I-Am-Not-A-Pussy-Therefore-I-Have-No-Problem-De-Heading-Muslim-Pussies-Who-Intolerantly-Be-Head-Others

In the event you don't have access to a proper Dictionary, mine advises: De-Heading rhymes with decapitation. Seriously.

Note E: If you are really angry, or lost, you could just ManUp, and RTFM, as 'suggested' in Twelve.

Fourteen. After running the command, it is likely you will see efibootmgr output akin to:

GUID Partition Table Header signature is wrong: ################# : ################
(multiple times)

Fifteen. Output like that can be ignored. To verify all is/will be: well, run (again):

efibootmgr -v

Sixteen. What you want to see is whatever -L (label) you created above is now the first listed item in the Boot Order. Read the efibootmgr -v output, carefully, and adjust, if necessary.

Seventeen. With the correct Boot Order set, select, Log Out, Shutdown.

Eighteen. When your system is hard down, remove the SystemRescueCD USB stick.

Nineteen. Power On.

Twenty. Interrupt the Boot process, for example, using the 'esc' key on an Asus, during Boot, and select the EFI option you created with '-L' above.

BlackJack!!! Welcome (back) to Qubes-R3.1-rc2, enter your LUKS passphrase, and you're on your way /home!

++++++++++++++++

I hope these details provide the Qubes devs with some clues to correct the current USB EFI-booting deficiencies within Qubes-R3.1-rc2.

As noted below, I do not suffer from these EFI booting anomalies when booting other USB HDD EFI+LUKS-enabled operating systems.

If I were a Qubes developer, I might think about this problem, this way:

1. I have coded an OS which requires both compatible hardware and to be run on bare metal, thereby effectively eliminating at least 95% of the global computing base of potential users.
2. Given the remaining 5% of potential users, I need to do everything to maximize the base of potential testers, so a hopefully large percentage of them will become users, testers and/or developers in the future.
3. Given 1 and 2, I had better prioritize release of a well-tested and working EFI booting solution so I can maximize the number of Qubes testers, knowing that a large percentage of my potential user base are running laptops, and are only willing to 'try/test' Qubes on an external USB HDD.
4. If Qubes testers cannot (or find it difficult to) boot Qubes from their USB HDDs, not only will they not use Qubes, they will tell their friends!
5. Given these points, it is obvious this issue needs to be addressed.

Cheers, and Qubes: most definitely, is a good 'thing' that can achieve greatness given proper focus.

Furthermore, from a very long time user/proponent of: Gentoo Hardened, i2p and Whonix, I recommend:

Do not give up, nor, in, ever. ;-)

HardenedArray

---

Hi Joanna,

First and foremost, you, and your team, have done superlative work getting Qubes to where it is currently. I know additional improvements are being worked on, and I would like to become a frequent user of Qubes in the future. I've read most of the excellent documentation, and I believe in the Qubes model as a thoughtful way to minimize risk.

However, at the moment, I cannot accomplish that due to the fact that the external USB HDD I've installed Qubes on cannot be booted after, and despite, successfully: installing Qubes, running Qubes, rebooting Qubes, shutting down Qubes, and starting Qubes from a cold boot. All of these things work, as expected, and without error.

The problem, and it is very significant in my mind, occurs the very first time I disconnect, or even accidentally jostle loose, the USB HDD cable from my laptop.

Following USB cable reconnection, all subsequent attempts to boot Qubes from that external USB HDD fail, despite my AMI BIOS recognizing the presence of the Qubes EFI HDD.

Allow me to go into a bit more detail in the hope we can find a solution which will also help a significant percentage of the potential tester/evaluator/user/contributor base moving forward.

I have a laptop with AMI BIOS (Aptio Setup Utility version 2.15.1228). The laptop is an ASUS, and can boot Fed 23 in EFI mode both when Fed 23 is residing as the system HDD in the laptop, AND when that same Fed 23 HDD is removed from the laptop and connected via USB. Fed 23 boots up without issue, in EFI mode, when connected to an USB SATA HDD interface.

With my Fed 23 HDD, it does not matter how many times I disconnect/reconnect the USB cable following a shutdown. It boots in EFI mode flawlessly, every time. In other words, in my mind, the Fed 23 HDD has NVRAM boot persistence, or at least some easy way for my BIOS to find the correct Fed 23 *.efi file. Additionally, my Fed 23 HDD has an almost identical partition structure to Qubes, including an EFI-system partition, a /boot partition and a partition dedicated to LUKS encrypted root and swap.

You should also note that I have successfully booted (one of my) encrypted Gentoo Hardened systems, many times, in EFI mode, using the exact same external USB HDD which I now have Qubes installed on. Again, it never mattered if I had disconnected the USB cable between Gentoo Hardened boots, and reboots, or swapped out new HDDs with dissimilar OSes. EFI to LUKS to kernel, via USB HDD.

I may be wrong, but it seems to me, therefore, this is not an unresolvable hardware, or BIOS/EFI, issue on my end. Perhaps, more positively, my system may represent an excellent test case.

All these comments pertain only to my installation of Qubes-R3.1-rc2.

Following gpg verification, I installed the rc2 *.iso from an USB stick to my external USB HDD. I had removed my laptop's system HDD prior to this (latest) Qubes rc2 installation.

1. Installation completes, as expected, and without error.
2. Initial setup and VM configuration completes, as expected, and without error.
3. Initial login is successful, and all VMs appear to be working correctly.
4. I can reboot Qubes, without issue.
5. I can shutdown Qubes, and fire up Qubes from a cold start, without issue.
6. After Qubes has booted, I can see dmesg reports that my EFI version = EFI v2.31 by American Megatrends.
7. I think when Qubes correctly boots it is loading xen-4.6.0.efi

However, the critical point is both point 4 and 5 above are only true, if I have not removed, or loosened the Qubes HDD USB cable connection to my laptop. As you might imagine, a laptop is of very limited practical value if an external USB HDD must be connected to it at all times, and God forbid, anyone ever touches, or makes accidental contact with the USB cable! ;-)

Worse still, an EFI enabled HDD which cannot boot the OS which resides on it is not worth much to anyone.

Once my Qubes HDD USB cable has been disconnected, accidentally, or intentionally, it is currently impossible for me to get Qubes to boot again. Believe me, I've tried! What I see after re-connecting my USB HDD (with no other drives of any type connected) to my laptop, and powering up, is an unwelcome trip to my BIOS settings screen. Of course, sometimes the USB HDD is not initially recognized, but a few reboots takes care of that.

With my Qubes USB HDD being recognized by my BIOS as something like [UEFI: my-HDD-model-number], and as the only item in the boot list, I save the BIOS settings, and reboot, only to be returned to the BIOS settings screen almost immediately (within a few seconds, at best).

My AMI BIOS Boot screen also offers: 'Add New Boot Option,' which requires an EFI file in this (backslash) format:

fs0:\path\filename.efi

I see two Qubes *.efi files on /dev/sda1: xen.efi and xen-4.6.0.efi. I could not find any *.efi files on /dev/sda2.

Using my BIOS 'Add New Boot Option', I have tried:

fs0:\boot\efi\EFI\qubes\xen-4.6.0.efi

and

fs0:\EFI\qubes\xen-4.6.0.efi

and

fs0:\boot\efi\EFI\qubes\xen.efi

and

fs0:\EFI\qubes\xen.efi

All four of those efforts failed, identically, with an almost immediate return to the BIOS settings screen, following system power up.

Each time after creating the boot option, moving that option to the top of the boot list (i.e. above the existing [UEFI: my-HDD-model-number], saving, and rebooting.

I have also tried creating all 4 new boot options with the fs0: statements above, saving, rebooting, and then manually selecting each one of the five boot options, using the 'esc' key during boot. Each one of those efforts, also fails, and I am immediately returned to my BIOS settings screen after each attempt.

Also of note:

1. I have Secure Boot set to: Disabled.
2. I have Legacy CSM set to: Disabled
3. I have Fast Boot set to: Disabled.
4. I have Legacy USB support set to: Enabled. If I change that to: Disabled, the Qubes HDD is not recognized by BIOS.
5. I have XHCI Pre-Boot Mode set to: Auto. This controls whether USB 3.0 is available, or not, during boot and does not appear to interfere with booting.

I have tried booting my Qubes USB HDD from both of my USB 3.0 ports, and from my USB 2.0 port, on my laptop. Absolutely nothing, or any new approach I take, seems to help with booting my Qubes USB HDD, whatsoever.

As another potential solution, I should also mention that one of my older encrypted Gentoo Hardened installations, installed on an USB HDD, can only be booted when Legacy CSM mode is enabled in BIOS. This should not surprise anyone, as Gentoo only very recently, added support for EFI booting, and literally, years after most other distributions began supporting EFI booting. Furthermore, my laptop system HDD must be removed to allow LUKS to find crypt root on /dev/sda3 (as opposed to /dev/sdb3), when booting this Gentoo Hardened USB HDD. This particular installation uses Grub2 to launch Gentoo Hardened, and this USB HDD definitely has 'boot persistence.' USB HDD cable disconnections in no way impact my ability to easily boot this Gentoo Hardened USB HDD, at any time.

Can Qubes safely provide an *.iso which supports Legacy CSM booting? If so, I would be happy to test it.

I'm out of ideas, other than to note that I've seen rEFInd successfully boot other OSes which Grub2, the kernel, or efibootmgr could not. Roderick Smith wrote and maintains rEFInd, in addition to his other highly useful code: gdisk, and the many Linux books and articles that he has authored.

I have no idea whether rEFInd can fix this serious inability to boot Qubes issue, but I do know that it's very unlikely there is anyone on the planet who knows more about EFI booting than Rod. If anyone wants to read about rEFInd, and EFI boot issues, his site is at: http://www.rodsbooks.com. Also note that rEFInd is available as a RPM package.

I do wish all the best for Qubes, but as I know you realize, I have to be able to boot.

I welcome your suggestions to test, and resolve, this issue. Thank you for your time, and amazing work.

Cheers,

HardenedArray

[andrewdavidwong]

Closing since self-solved.

[andrewdavidwong]

Closing since self-solved.