Support for EFI boot

[marmarek]

Reported by marmarek on 4 Feb 2014 02:53 UTC
EFI is more and more popular, soon there will be some machines with EFI-only boot, so we need badly support for it.
Adding such support may be non-trivial, because Xen 4.1 do not support direct EFI boot (AFAIR it is supported in >=4.2 or even >=4.3). But perhaps some workaround like grub-efi can be used.

Migrated-From: https://wiki.qubes-os.org/ticket/794

[marmarek]

Modified by joanna on 14 Mar 2014 12:51 UTC

[marmarek]

Modified by joanna on 20 Apr 2014 17:07 UTC

[marmarek]

Comment by anonymous on 22 Oct 2014 19:50 UTC
Please mention this problem on system requirements page.
As of now, having a laptop that meets those reqs but lacks UEFI legacy mode (like asus s300c) as the only system may lead to a long trial-and-error quest without any possible resolution.

[marmarek]

Comment by anonymous on 3 Nov 2014 13:56 UTC
Important as most of the new laptops and desktops use EFI

[marmarek]

There is progress here, some preliminary code here: https://github.com/marmarek/qubes-installer-qubes-os/tree/efi

Currently the problem is with ISO9660 boot format ("El Torito"). It support boot images up to 32MB, our have 56MB (which includes 40MB of initrd.img).

How it works

1. ISO9660 with El Torito extension contains "boot catalog" which points at images used to boot. Our image contains three entries here:
   1. for BIOS (isolinux/isolinux.bin),
   2. for UEFI (images/efiboot.img, which is fat32 image of EFI System partition)
   3. for Macs (images/macboot.img, similar to efiboot.img but with some Mac specific modifications)
2. Additionally our image have partition table added by isohybrid, for having the same image for USB sticks. This partition table is prepared such a way that each partition points at appropriate boot image (which exception for BIOS, which just uses syslinux-provided code for MBR). Details on osdev wiki and syslinux wiki
3. EFI boot image contains shim.efi as BOOTX64.efi (not really useful until we got SecureBoot working), which loads grubx64.efi. It presents the user simple menu ("install" and "check media & install"), then loads xen.efi or xen-check.efi, which are just copies of the same file.
4. xen.efi (or xen-check.efi) reads its configuration file (xen.cfg or xen-check.cfg), which contains names of kernel and initrd to load. Then loads those images using EFI services.

The problem

"El Torito" boot catalog structure have 16-bit field for image size (expressed in 512-bytes sectors). This means maximum image size is 32MB. If bigger is used, the field will be silently truncated and only part of the image would be loaded by UEFI firmware. I've found that out looking at hex dump of boot catalog... Then isohybrid would also use that truncated size for partition size, so even when this structure isn't directly used, the image would also be truncated.
Fedora images doesn't have this problem, because kernel and initrd is loaded by grub, so do not need to be in EFI partition (grub has ISO9660 filesystem driver). But xen.efi cannot be loaded as multiboot binary yet and the only way to provide kernel and initrd is config file with its names, which will be loaded using EFI service - so the images must live on EFI partition.

Possible solutions

The most universal solution would be having smaller efiboot.img. I'm currently investigating this option, namely what we can rip of initrd.img. When this will be done, both DVD and USB version would be bootable under EFI.

But if we fails at it, we can still have EFI compatible USB version. We simply need fixing partition size to match efiboot.img size. But this way, image written on DVD would not work under EFI (even worse - it will looks like EFI compatible one, but will crash during startup).

Further steps/problems

1. When provided correct partition table, Xen loads correctly, but Linux kernel hangs (last message was something about EFI Variables Facility). I haven't investigated this, maybe the kernel image still was invalid. Or some kernel configuration option missing.
2. The same problem applies to Live edition, with the same possible solutions.

[marmarek]

Some progress has been made:

1. I've managed to reduce efiboot.img to 31MB. Initrd is really limited there (no network support, dropped all SCSI drivers, dropped many unusual filesystem drivers etc).
2. Found that apparently when USB stick contains ISO9660, UEFI firmware uses that information instead of MBR or GPT partition tables there (both present). And then "Simple Filesystem Protocol Interface" doesn't work. Breaking Eltorito Volume Descriptor "fixes" that problem (but probably makes the image unbootable when used on real DVD). Maybe its a bug in just my BIOS...
3. Kernel still hangs at efivars initialization. But this time I'm pretty sure its my buggy hardware, because I've seen exactly the same kernel working on another platform.

[Daerdemandt]

Maybe its a bug in just my BIOS...

this time I'm pretty sure its my buggy hardware

Will other people trying out exactly the same image on their hardware to see what changes be of any assistance?

[marmarek]

I think so :)
Here is the link:
http://ftp.qubes-os.org/~marmarek/Qubes-20150901-x86_64-DVD.iso
sha256sum: e166d9537d720bc2d7d60ff33d208bb84c2fd0ea71aa13551511ffb4e5ec236f

The image should be treated as untrusted!

Test instructions 1:

1. Write the image to some USB device using dd
2. Boot the image in UEFI mode (it contains both BIOS and UEFI boot
   code)
3. You should get GRUB menu with just two options, on black background
4. Switch to GRUB cmdline (c), enter chainload / and press TAB twice

Possible results:

• you will see list of files on the ISO9660 filesystem - a bunch of directories, one of them is EFI (capital letters), or
• you will see files on FAT32 filesystem (efiboot.img) - just one directory "efi" (lower letters).

Tell me which one do you see and continue to "Test instructions 2".

Test instructions 2 (continue from the point above):

1. Return to the menu (ESC), try to launch setup (the option without
   checking the media).
2. If that doesn't start and return to the menu, go to "Test
   instructions 3", otherwise wait for system to load

Possible results:

• it will hang at "EFI Variables Facility v0.08 2004-May-17"
• it will go through it and start initramfs (a bunch of green "[ ok ]")

If you get here, you can try to install the system somewhere (do not override your primary system!). I haven't got that far, installation will probably fail. In such case, note the exact failure.

Test instructions 3:

1. Break the ISO9660 boot descriptor (make backup of the image first):

   echo -en '\x03' | dd of=Qubes-20150901-x86_64-DVD.iso seek=16 bs=2048 conv=notrunc
   

2. Write the image to some USB device using dd

3. Continue "Test instructions 1" starting from point 2.

Test instructions 4:

Perform test instructions 1-2 using DVD instead of USB stick.

[Daerdemandt]

The image should be treated as untrusted!

Untrusted as in 'never boot from untrusted device' or is it just poor wording?

[marmarek]

On Mon, Sep 14, 2015 at 04:25:29PM -0700, Daerdemandt wrote:

The image should be treated as untrusted!

Untrusted as in 'never boot from untrusted device' or is it just poor wording?

Untrusted as it may steal your data, infect everything it touches and
kill your cat.
Build environment used to produce that image is far from clean and may
be compromised.
So at least I would disconnect disks with valuable data first.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

Regarding that ISO9660/grub/xen.efi problem loading vmlinuz and initrd - using refind instead of grub seems to solve the problem. No need to have separate images for DVD and USB :)
Pity that Fedora has no package for it, so we need to maintain one in our repo...

[marmarek]

Regarding hang at efivars initialization, this happens also on Thinpad T430. Relevant xen-devel thread:

This is a firmware bug.

+1 (and I'm surprised how common this is)

The bug is present in the reference implementation code, which means it
is present in a lot of real firmware. We have kit from 3 different
vendors which are affected, including latest available firmware.

There are also patches attached, I'll check whether are available in any stable Xen version, or is the backport feasible.

[marmarek]

Regarding rEFInd/Grub - finally managed to get that working with Grub. When
'root' variable points at EFI partition on the image (not the whole device), it
simply works.

New images, this time built in more trustworthy environment:
http://ftp.qubes-os.org/~marmarek/Qubes-20150930-x86_64-DVD.iso
http://ftp.qubes-os.org/~marmarek/Qubes-20150930-x86_64-DVD.iso.asc
SHA256: 77845d9265579c30947641500f4a4b587fe77b3cdc58f79171cc2275c2000d3d

And bonus - EFI-enabled LIVE image:
http://ftp.qubes-os.org/~marmarek/Qubes-20150930-x86_64-LIVE.iso
http://ftp.qubes-os.org/~marmarek/Qubes-20150930-x86_64-LIVE.iso.asc
SHA256: d73417c1911bf2ab904f287f2a1be1dc65d59f085d341b2745449bf5d00b0dab

Any feedback welcome :)

[rootkovska]

Problems with the latest EFI Live image:

1. Plymouth runs in text mode only on the machine I tested this
2. Deefault appmenus don't have Firefox (but they do have "Help", heh ;)
3. Networking doesn't work on the machine I tested (and it worked with the previous iteration), the reason is: no firmware for iwlwifi-6000

So, I would say a few regressions from the previous (August) image.

[marmarek]

On Sat, Oct 03, 2015 at 01:18:42AM -0700, Joanna Rutkowska wrote:

Problems with the latest EFI Live image:

1. Plymouth runs in text mode only on the machine I tested this

Also in legacy mode? In EFI mode initramfs is really limited (32MB
limit...), so it may be hard to include plymouth theme (I'll try).

1. Deefault appmenus don't have Firefox (but they do have "Help", heh ;)
2. Networking doesn't work on the machine I tested (and it worked with the previous iteration), the reason is: no firmware for iwlwifi-6000

Strange, template is unchanged... I'll look into it.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[dwangoac]

Antiplex, I'd recommend the rEFInd image from above with -- efi=attr=uc, except...

Marek, is there any update on the possibility of a newer rEFInd version? The problem is the older rEFInd version fails at efibootmgr -c -w -L Qubes -d /dev/sda -p 1 -l \EFI\qubes\shim.efi because efibootmgr is "cowardly refusing to create a boot option". Attempting to create a valid /boot/efi and even a separate /boot and re-running the installer fails because "You have not created a bootloader stage1 target device" and https://bugzilla.redhat.com/show_bug.cgi?id=1010495 is still an open issue. Installing to a different system and copying the entire installation over seems to be getting me the furthest, but I'm stuck because Qubes NetVM doesn't start, throwing a Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory error (presumably as a result of a failed root partition clone, I'm still investigating).

I started work on taking the rEFInd EFI directories from the last test image and merging in the R3.1 release but it wouldn't boot at all. Thoughts? Linux hasn't been this difficult to install for more than a decade. :)

[dwangoac]

50+ hour mark, it's 2:00 AM PST and I'm giving in for the night, but I'm very close to getting this working on my MacBook Pro 10,1 "Early 2013". Steps - install rEFInd from OS X so the Mac boots to it by default and set refind.conf to scan from all media types, i.e. the default option. Insert stock R3.1 USB key but boot "normally" to reach rEFInd, select the EFI option on the USB key, hit F2 to edit it, and add -- efi=attr=uc. Once reaching graphical Anaconda, ctrl+alt+F2 and edit /usr/lib64/python2.7/site-packages/pyanaconda/bootloader.py and change the raise near line 1696 to a pass (or, just replace all of them :). Then, restart Anaconda with:

systemctl stop anaconda.service
anaconda-cleanup
systemctl start --no-block anaconda.service

Proceed with the installation and you will have everything except a valid xen.cfg, which you will have to build yourself. This is the part I'm getting stuck at, and others have gotten stuck here before as well; what goes in xen.cfg when using an LVM for root= parameters, and/or how do you determine what should go there? The problem is /dev/mapper is empty and booting stops at a dracut shell. I'll fall back to standard partitions next but it would be good to figure this bit out.

[marmarek]

Example xen.cfg: #794 (comment)

Regarding empty /dev/mapper - is your disk even discovered there? Check /dev/sd* (or more convenient: fdisk -l - if it is present in dracut env...).

[dwangoac]

HI Marek - I should also quickly note that the MacBook Pro keyboard does not work in dracut, I have to use an external keyboard but at least that works. There is no fdisk in dracut but I do see /dev/sda4 with TYPE="LVM2_member" from blkid. If I attempt to mount it it fails saying unknown filesystem type 'LMV2_member'. My xen.cfg directly matches above, specifically kernel=vmlinuz-4.1.13-8.pvops.qubes.x86_64 root=/dev/mapper/core3-root vconsole.font=latarcyrheb-sun16 rd.lvm.lv=core3/root but because /dev/mapper is empty I don't know where the name core3-root would come from. I see no path forward with LVM as much as it seems like it would e a better choice so I will re-perform the steps from my last comment as well as your change in commit 2556a42 and see what I get. Thanks!

[marmarek]

For LVM - run lvm lvs to list volumes. For full LVM init, call lvm vgchange -ay (and if /dev/mapper is still empty, call lvm vgmknodes).

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[dwangoac]

Marek, unfortunately none of those tools exist in dracut. As I indicated I opted to try again with standard partitioning but it's almost the same problem, now it's unknown filesystem type 'crypto_LUKS'. I'm beginning to believe I have the wrong initramfs. Should that have been generated during installation and placed somewhere?

[marmarek]

If there is no lvm, nor cryptsetup in initramfs, it indeed looks
like a broken initramfs. You can regenerate it by booting recovery
option of the installer (or simply doesn't immediately reboot the system
after installation and switch to tty2). Then execute chroot /mnt/sysroot dracut -f (check the mount for actual directory - AFAIR
it's /mnt/sysroot, but I may be wrong).

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[dwangoac]

OK, this is likely the last update from me in this thread, and thankfully it's for a good reason - I finally have everything booting, with LVM, and even with LUKS! The problem was definitely an incorrect initramfs; after the install completed I failed to copy the file to the correct /EFI/qubes location and thus I was using a bad copy. I had additional problems because the suggested xen.cfg configuration did not work as I had no core3-root (and have no idea how something would be named like that), but simply adding root=/dev/dm-1 with no rd.lvm.ld specified worked fine and I was able to see the post-install welcome configuration wizard for the first time.

My luck changed after that as I have a 14e4:4331 BCM4331 wireless adapter that hard-freezes the system as soon as netvm is started with that device included. I've tried the /etc/systemd/system/qubes-pre-netvm.service permissive mode change as identified in the issue described at #1261 and I've even (painfully) copied in firmware and re-run the b43-fwcutter tool but unfortunately any attempt at normal networking hard locks the system. Because a MacBook Pro of this era has no network port this means I have no networking at all unless I violate dom0 with a USB network adapter. These problems probably belong in the other thread but I wanted to extend my thanks to Marek for the help along the way, and I hope my notes in this history help someone else in the future.

[wardwouts]

Adding -- efi=attr=uc to the boot options in refind lets me boot into the installer on my lenovo yoga 2 pro. After installation adding efi=attr=uc to the options lines in /EFI/qubes/xen.cfg on the first partition results in a bootable system.

Things seem to work fine after that. (Well, KDE doesn't like HiDPI displays, but I can work around that.)

Thanks dwangoac & marmarek

[antiplex]

hmmm... still unsure if Boot00XX* Qubes HD(...)File(\EFI\qubes\xen.efi)p.l.a.c.e.h.o.l.d.e.r. ./.m.a.p.b.s. ./.n.o.e.x.i.t.b.o.o.t. (as posted about 5 days ago #794 (comment)) is the way it is supposed to be.
for those who successfully applied the fix described in https://www.qubes-os.org/doc/uefi-troubleshooting/ , how do the additional options added to the entries end look like? do the show dots between each character like shown above?

using rEFInd was repeatedly recommended to be used in order to fix such EFI-related problems but i'm lacking some sort of understanding about how to apply rEFInd. can anyone help me out with a short step-by-step description of what to boot/load/copy/... when?

cheers, anti

[ghost]

I think these images from marmarek contain Qubes with rEFInd instead of GRUB to overcome the issues.

Next images (main difference: updated xen to support /noexitboot):
http://ftp.qubes-os.org/~marmarek/Qubes-20160208-x86_64-refind.iso
http://ftp.qubes-os.org/~marmarek/Qubes-20160208-x86_64-refind.iso.asc

I personally made a USB 3.0 stick from the ISO using dd and tried to do UEFI boot, but it failed for the same reason as the official 3.1 does. Hangs on "EFI Variables Facility..." line. See my bugreport referenced by marmarek 3 days ago. I did not try "/mapbs /noexitboot", because it was not possible. I tried it on official 3.1 GRUB2 boot, but with no change.

Marmarek says its a vendor, or Xen bug, but it does not affect Fedora 23 respin. I can do UEFI boot on it. I think there is no hope for my HP Probook 4730s to do UEFI boot for Qubes 3.1, I will have to get by using Legacy BIOS.

[antiplex]

@slazer Thanks for pointing me to these ISO's, i must somehow have overseen those ;(
now i also understand why dwangoac asked for some updated rEFInd images a few posts above as the .iso files you linked must pre-date qubes v3.1
so i'll give those a shot and see how it turns out (see below).

I've come across the hang on "EFI Variables Facility..." line on the standard Qubes-release as well but was able to get around through "/mapbs /noexitboot" but got stuck after editing the efi-boot-entries as my system won't boot.
if my devices firmare would let me i'd happily use legacy bios but somehow my fujitsu h720 provides some CSM (compatibility support module) that is supposed to provide some legacy bios services but my issues remain as it seems not really possible to disable uefi...

well... now i tried to use marmarek's rEFInd version but failed to see how i could modify the boot-menu entries (to try different options/combinations of -- efi=attr=uc, -- efi=attr=uc) as possible in the standard qubes version by pressing 'e' in grub.
how would i do that on marmareks build?

[ParadiseBurn]

I waneed to thank everyone for input here! I know tail end was mostly regarding Mac's but had some MAJOR hiccups first time installing Qubes 3.1 and this thread lead me down some good roads!

I did want to chime in with my own general observations regarding some of the EFI and freezing. I mean not sure of EFI specific, but I decided to work my way forward, installed R2. Started upgrading templates etc, working my forward to kind of "shade tree" upgrade / troubleshoot (maybe a stretch being first time Linux user, upgrading borderline Legacy/UEFU aged windows 8 machine).

So I made a snap judgement to try newest stable Qubes Kernel 3.19 via console and was running semi solid loading in then BANG! Freeze on boot, lockup after initial extended login. I rolled back to early kernel (worked fine) the kernel (3.19.018) I researched and found some similar issues in Fedora Git hub and was apparently pretty common. There were numerous issues with 3.19 causing lock up and freeze (usb related) right after initial login in or during OS initial loading after user login.

I'm going to work my way forward version by version until I do more research, but you might try a different kernel version? I was seeing that 3.19.030 had some luck, but it seems that in general 3.19 had some pretty unhappy customers.

[ideologysec]

You can try the other kernel that Qubes should ship with; edit your xen.cfg to have dom0 use the 4.1.13-pvops kernel instead of 3.19.

[ParadiseBurn]

No good on either, can't even load OS. The only kernel I've got
functionality on so far is 3.12.40, I'm working my way forward
On Mar 28, 2016 1:36 PM, "Aktariel" notifications@github.com wrote:

You can try the other kernel that Qubes should ship with; edit your
xen.cfg to have dom0 use the 4.1.13-pvops kernel instead of 3.19.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#794 (comment)

[ParadiseBurn]

I did get it to install 3.0, working on updates and kernel now
On Mar 28, 2016 3:29 PM, "Tim Collins" timcollinsal1@gmail.com wrote:

No good on either, can't even load OS. The only kernel I've got
functionality on so far is 3.12.40, I'm working my way forward
On Mar 28, 2016 1:36 PM, "Aktariel" notifications@github.com wrote:

You can try the other kernel that Qubes should ship with; edit your
xen.cfg to have dom0 use the 4.1.13-pvops kernel instead of 3.19.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#794 (comment)

[ghost]

Please, where could I download any of these?

• an ISO including all pull requests referred to as Fedora 23 support branch
• an ISO including commit mm_2556a42c
• an ISO containing an updated rEFInd images, more recent than 20160208
• any Qubes 3.1 respin ISO

I would like to test any of these, as it might have solved my issue. I am able to compile stuff, though some advice, links or tutorials might come in handy.

[marmarek]

There are no new images yet. You can build one yourself - instructions here:
https://www.qubes-os.org/doc/qubes-builder/
https://github.com/qubesos/qubes-builder

Also watch Fedora 23 dom0 - there will be links for test images when available.

[ghost]

My comprehensive comment got wiped after Vivaldi crash, so I will be short. Just compiled the master branch of marmarek, but the boot still hangs on "EFI Variables Facility..." line like before. The UEFI troubleshooting nor rEFInd did not help.

As soon as the fedora-23 branch at installer-qubes-os gets merged with master, I can recompile the master branch with DIST_DOM0 ?= fc23 and see if UEFI boot works on my HP ProBook 4730s laptop. I have great expectations, because the UEFI bug prevented me from installing Fedora 23 ISO, but was quickly fixed in one of the Fedora 23 ISO respins.

I wonder what I can expect from fc23 in dom0, instead of fc20 like in Qubes3.1. Given the fact it is a master branch, I should be prepared for instability, right?

[marmarek]

On Sun, Apr 10, 2016 at 05:24:20PM -0700, Martin Páleník wrote:

My comprehensive comment got wiped after Vivaldi crash, so Ill be short. Just compiled the master branch of marmarek, but the boot still hangs on "EFI Variables Facility..." line like before. The UEFI troubleshooting nor rEFInd did not help.

As soon as the fedora-23 branch at installer-qubes-os gets merged with master, I can recompile the master branch with DIST_DOM0 ?= fc23 and see if UEFI boot works on my HP ProBook 4730s laptop. I have great expectations, because the UEFI bug prevented me from installing Fedora 23 ISO, but was quickly fixed in one of the Fedora 23 ISO respoins.

I wouldn't expect much from that. The hang on "EFI Variables
Facility..." on Lenovo machines is totally unrelated to distribution
there. Generally at that point only Xen+kernel is running, nothing else.
Maybe there is some patch in kernel used in Fedora 23?

I wonder what I can expect from fc23 in dom0, instead of fc20 like in Qubes3.1. Given the fact it is a master branch, I should be prepared for instability, right?

Yes ;)

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[succeedmr123]

After tl; dr other 6/8 of all these discussions,

all I can say is that I confirm that release Qubes .iso dd -ed to USB and ran in strict UEFI mode without CSM crap their pants on my HP UEFI GPT notebook too (Intel i5/HM77express+AMD powered main GPU. BIOS is H20 Insyde (anyway, in CSM it can start the installation).

Just wanna say something is not yet mature here. Not nearly.

Cheers to the dear devs. I will contrib as much as I gather insight.

M.

[clayfreeman]

For those attempting to install via USB on a MacBook, you can remove the ESP flag on your USB with GParted, mount the prior ESP, and rename the EFI folder to something else. Now you can use sudo bless -device /dev/diskX -legacy -setBoot -nextonly from within OS X to boot the USB device in legacy mode.

[0keh]

Installed 3.2 on Dell Lattitude E6320. I had the same experience as ManoftheSea. I stucked at the penguins with efi and esrt error messages:
efi: EFI_MEMMAP is not enabled.
esrt: ESRT header is not in the memory map.
Nothing from qubes-os.org/doc/uefi-troubleshooting worked, but after a lot of tries and thanks to marmarek in his post (#794 (comment)) I used efi=no-rs.

The usb device for qubes installation loaded and checked the files.
Error messages beneath the
efi: EFI_MEMMAP is not enabled.
esrt: ESRT header is not in the memory map.
are:
dracud-pre-trigger[402]: cat: /tmp/dd_disk: No such file or directory. dracut-initqueue[511]: mount: /dev/sdb is write protected while booting the installation device. Installation was a success.
Everything is automatically and correct configured in xen.cfg. I partitioned the ssd on my own, but with recommended /boot/efi in /boot.
Maybe because of Admin-Password in Bios it could not be written as uefi-bootparameter, but manually set the uefi boot option in bios it is working now.
A tiny bug I think I found while installing. (Have to open another issue)

Great project guys (and girls).