How to make dual (BIOS & UEFI) USB or ISO

[jeffshead]

I'm trying to make a single USB or ISO image that can be used to support both EFI and legacy BIOS computers.

The following commands work to build the image but the keyboard stops working if I use the image and boot efi:

# for systems with BIOS
make bin/acm--ncm--ecm--axge.usb ... # your options may come here

# copy MBR and boot loader image on /dev/sdb disk beginning.
dd if=bin/acm--ncm--ecm--axge.usb of=/dev/sdb oflag=sync

# for systems with UEFI
make bin-x86_64-efi/acm--ncm--ecm--axge.efi ... # your options may come here

# create ESP partition, FAT32 filesystem and copy boot loader so UEFI firmware can find it
# I left 5Mb offset as MBR and /dev/sdb4 partition occupies 4095 sectors from beginning of disk (~2MB)
# partition size - choose any fits your needs. In my case ending point is 311M
parted -s /dev/sdb mkpart primary fat32 10MiB 311MiB
parted -s /dev/sdb set 1 esp on
partprobe /dev/sdb

mkfs.fat -F32 /dev/sdb1

mount /dev/sdb1 /mnt
mkdir -p /mnt/efi/boot/
cp bin-x86_64-efi/acm--ncm--ecm--axge.efi /mnt/efi/boot/bootx64.efi
umount /mnt

The above code came from this post.

According to this post, uncommenting //#define USB_HCD_USBIO /* Very slow EFI USB host controller */ in /src/config/usb.h should resolve the issue but I get the following error when I try to make the efi file:

make bin-x86_64-efi/acm--ncm--ecm--axge.efi

ld: bin-x86_64-efi/blib.a(usbio.o):/home/jeff/ipxe/src/drivers/usb/usbio.c:1725: multiple definition of `usbio_driver'; bin-x86_64-efi/blib.a(efi_usb.o):/home/jeff/ipxe/src/interface/efi/efi_usb.c:1360: first defined here
make: *** [Makefile.housekeeping:1233: bin-x86_64-efi/acm--ncm--ecm--axge.efi.tmp] Error 1
rm bin-x86_64-efi/version.acm--ncm--ecm--axge.efi.o

What am I doing wrong?

[NiKiZe]

To generate dual boot images use util/genfsimg as described in #1155 (comment)

What you seem to be trying to generate is a USB NIC only boot image? To get access to USB interfaces, iPXE has its own USB drivers, which does disconnect devices from BIOS/firmware drivers, iPXE should however have its own USB keyboard driver.

Are you using latest git master? Do you have any other local modifications?

What is your end goal, which NIC are you trying to boot from? (USB Vendor and device id or PCIID)

[jeffshead]

USB NIC only

Yes, I'm trying to PXE boot some computers with USB NICs.

Are you using latest git master?

Yes

Do you have any other local modifications?

No

What is your end goal, which NIC are you trying to boot from?

Plugable, Dell, Kanex, Acer brands. Most have the Realtek RTL8153 chipset. The others have the Asix AX88179.

The computers that I'm trying to PXE boot do not have built-in Ethernet NICs. Their firmwares do not support PXE booting with any of these USB NICs. As a workaround, I'm trying to create a bootable iPXE USB thumb drive. As I stated previously, I successfully created an iPXE USB thumb drive that supports both BIOS and EFI but once booted, the keyboard no longer works for EFI PXE boots. I don't encounter this issue with BIOS PXE boots. Because of this issue, I tried uncommenting that line in the usb.h file as I described previously, but doing so causes errors during the build.

The USB thumb drive doesn't have to be for USB NICs only, but it does need to support USB NICs.

[mcb30]

The computers that I'm trying to PXE boot do not have built-in Ethernet NICs. Their firmwares do not support PXE booting with any of these USB NICs. As a workaround, I'm trying to create a bootable iPXE USB thumb drive. As I stated previously, I successfully created an iPXE USB thumb drive that supports both BIOS and EFI but once booted, the keyboard no longer works for EFI PXE boots.

Please clarify "keyboard no longer works". Do you mean that the keyboard no longer works within the iPXE shell (or menu), or that the keyboard is no longer working within some program that you have used iPXE to boot?

[jeffshead]

The keyboard stops working as soon as I get to the iPXE shell.

[mcb30]

The keyboard stops working as soon as I get to the iPXE shell.

Thanks.

Could you please try building with DEBUG=usb,usbhid,usbkbd? You can do this without the use of parted, partprobe, etc using:

make bin/acm--ncm--ecm--axge.lkrn bin-x86_64-efi/acm--ncm--ecm--axge.efi DEBUG=usb,usbhid,usbkbd

./util/genfsimg -o boot.iso bin/acm--ncm--ecm--axge.lkrn bin-x86_64-efi/acm--ncm--ecm--axge.efi

to generate the hybrid USB/ISO image boot.iso.

[jeffshead]

@mcb30
I was able to use the commands that you provided to create the image but when I boot from it, there is only an option to boot from BIOS, not efi.

Also, when the USB drive is attached to a Windows computer, it uses two drive letters and shows up as a USB drive and a DVD.

I tried twice and got the same results both times.

[mcb30]

The image built with those commands will be a hybrid ISO/USB image (as you originally requested). It's possible that your BIOS does not handle this well. Try building just a USB image instead by changing -o boot.iso to -o boot.img.

[jeffshead]

Thank you for your time!

What's the best way to write the image to the USB drive? I don't know what I'm doing wrong. I've tried dd, isohybrid and Rufus. I get varying results but none (other than my original post) produce a USB thumb drive that shows the correct options under both the legacy and UEFI boot menus when using the .iso or .img file.

Using -o boot.img created a USB drive where the USB drive shows up as a Diskette Drive under the legacy menu:

However, if I select Diskette Drive, it does perform the legacy boot and the keyboard does work.

Below are screen caps depicting the debug info when legacy booting (keyboard works):

The image below depicts the debug info when I do the UEFI boot:

The UEFI boot is successful but the keyboard does not work.

[mcb30]

Thank you for the well described and relevant screenshots!

What's the best way to write the image to the USB drive? I don't know what I'm doing wrong. I've tried dd, isohybrid and Rufus. I get varying results but none (other than my original post) produce a USB thumb drive that shows the correct options under both the legacy and UEFI boot menus when using the .iso or .img file.

Using dd is correct.

Using -o boot.img created a USB drive where the USB drive shows up as a Diskette Drive under the legacy menu:

That makes some sense: genfsimg uses mformat to create a FAT filesystem within the unpartitioned image, and your BIOS is choosing to interpret any unpartitioned disk as "Diskette".

The UEFI boot is successful but the keyboard does not work.

In the BIOS build you have USB_KEYBOARD still enabled. In the UEFI build I think you have a local customisation to disable USB_KEYBOARD since the screenshot shows using driver usbio rather than using driver kbd, i.e. the USB kbd driver is not present and the device is therefore being offered for consumption by existing UEFI drivers via the usbio interface.

Unfortunately the UEFI standard's EFI_USB_IO_PROTOCOL is so badly designed that several OEMs chose to use an alternative, completely proprietary, and undocumented interface instead. You can't really blame them, given the design flaws:

ipxe/src/drivers/usb/usbio.c

Lines 43 to 49 in 59f27d6

	* The EFI_USB_IO_PROTOCOL is an almost unbelievably poorly designed
	* abstraction of a USB device. It would be just about forgivable for
	* an API to support only synchronous operation for bulk OUT
	* endpoints. It is imbecilic to support only synchronous operation
	* for bulk IN endpoints. This apparently intern-designed API
	* throttles a typical NIC down to 1.5% of its maximum throughput.
	* That isn't a typo. It really is that slow.

This has the side effect that the OEM's USB drivers (such as the keyboard driver) are unable to bind to the standard EFI_USB_IO_PROTOCOL interfaces exposed by iPXE. It's therefore not that surprising that your keyboard doesn't work in this setup.

Could you please retry your UEFI test case without disabling USB_KEYBOARD in config/usb.h?

[jeffshead]

I think you have a local customisation to disable USB_KEYBOARD since the screenshot shows using driver usbio rather than using driver kbd

I havn't customized any files nor have I added any files. For this latest test, I redownloaded the source files for a clean start.

Could you please retry your UEFI test case without disabling USB_KEYBOARD in config/usb.h?

That line is commented out by default. I did as you suggested and uncommented that line in config/usb.h before running any other commands.

I get the same results. Keybord still doesn't work. Below is a screenshot of UEFI boot:

[mcb30]

I think you have a local customisation to disable USB_KEYBOARD since the screenshot shows using driver usbio rather than using driver kbd

I havn't customized any files nor have I added any files. For this latest test, I redownloaded the source files for a clean start.

My mistake, sorry. I lost track of what our current default UEFI configuration is.

For BIOS, we include USB_KEYBOARD support by default (set in config/defaults/pcbios.h). For UEFI, we currently assume that the OEM's keyboard driver will continue to function with the EFI_USB_IO_PROTOCOL interface that we expose.

Could you try putting an explicit

#define USB_KEYBOARD

in config/usb.h and see if that then works?

[jeffshead]

in config/usb.h and see if that then works?

Keyboard still doesn't work. Below is screencap with explicit:

[jeffshead]

Next step here is probably to look into which script it is running, and call that directly from your new build.

Unfortunately, it looks like iVentoy is closed source so I guess this is as far as we can go. I'll see if I can figure out how to move away from iVentoy. I started with it because it made PXE booting so simple to set up and it allowed me to use my current Windows DNS server with a single URL to boot both legacy and UEFI.

Thank you @mcb30 and @NiKiZe for your time.

[NiKiZe]

They are using iPXE, the license does not allow closed source. It does allow not to publish sources if it is totally unmodified version of the code found here. But if that is the case you will be able to create a new version of iPXE that matches.

The question then become, is any embedded script part of the source or not.

[jeffshead]

On their Github page, they list the open source portion which has a folder (10 mo old) named ipxe-bd13697. I keyword searched version but I wasn't able to determine the iPXE version. I will attempt to make a USB drive using the version on their Github page.

[jeffshead]

I will attempt to make a USB drive using the version on their Github page

Failed using same make commands. Bunch of errors. Mostly "...architecture is not supported", error: operator '==' has no left operand errors.

terminal_output.txt

[NiKiZe]

On their Github page, they list the open source portion which has a folder (10 mo old) named ipxe-bd13697. I keyword searched version but I wasn't able to determine the iPXE version. I will attempt to make a USB drive using the version on their Github page.

Let's assume it's based on
bd13697

They do have a build.sh script, but nothing embedded there.

[jeffshead]

I used bd13697 to create the USB thumb drive with EMBED=shell.ipxe and with no other modifications. The keyboard fails to work when booting UEFI.

Below is the unedited config/usb.h:

#include <config/defaults.h>

/*
 * USB host controllers (all enabled by default)
 *
 */
//#undef        USB_HCD_XHCI    /* xHCI USB host controller */
//#undef        USB_HCD_EHCI    /* EHCI USB host controller */
//#undef        USB_HCD_UHCI    /* UHCI USB host controller */
//#define       USB_HCD_USBIO   /* Very slow EFI USB host controller */

/*
 * USB peripherals
 *
 */
//#undef        USB_KEYBOARD    /* USB keyboards */
//#undef        USB_BLOCK       /* USB block devices */

/*
 * USB external interfaces

Next, I re-created the USB drive with EMBED=shell.ipxe and with the #undef USB_KEYBOARD /* USB keyboards */ line uncommented in config/usb.h. I got the same results.

Next, I added #define USB_KEYBOARD to config/usb.h and the keyboard works in the shell:

If I enter autoboot, the network version boots but of course the keyboard no longer works.

[NiKiZe]

What I would do is use that commit, apply their tree, check for changes, and maybe commit them, then rebase on current master.

When you create the usb build you will automatically get everything enabled, so no need to make any local usb configuration changes.

[mcb30]

@jeffshead From reading your posts, I think you have:

• Commit bd13697 works only if you add #define USB_KEYBOARD
• Current master works with or without #define USB_KEYBOARD

It would be interesting to know which commit fixed the problem. If you are interested, you can try following the instructions at https://ipxe.org/howto/bisect to identify this specific commit.

(This won't do anything to change the binary included in iVentoy: it would be purely for curiosity.)