[issue]: Ventoy can't run as a standalone EFI payload: 2 small fixes could enable that usecase + supporting a better 4k / 8k alignment

[csdvrx]

Official FAQ

• I have checked the official FAQ.

Ventoy Version

1.0.63

What about latest release

Yes. I have tried the latest release, but the bug still exist.

BIOS Mode

UEFI Mode

Partition Style

GPT

Disk Capacity

118G

Disk Manufacturer

No response

Image file checksum (if applicable)

No response

Image file download link (if applicable)

No response

What happened?

Ventoy has quickly become one of my favorite tool to rescue systems, replacing the handcrafting of EFI payloads from say Ubuntu live images and Windows rescue thumbdrives.

However, using Ventoy.efi as an EFI payload on the EFI partition is not supported, even if it it worked before.

I did a quick analysis and concluded it's likely to be this way because you want Ventoy users to have a seamless experience and catch misconfigurations for them: having the first partition start at 2048 is a nice "canary in the mine": if it doesn't, it means it's likely there's no boot payload from 34 to 2047, which means the drive may not be bootable everywhere... so it's not supported. But it still seems like a feature still desired by users: https://forums.ventoy.net/showthread.php?tid=695

So what about making these checks implicit?

The 2 changes would be: 1) check for a special partition from 34 to 2047 and 2) check for a Ventoy.efi payload on the EFI partition, regardless of where this partition may be (thumb drive or hard drive)

I would like to suggest this as an alternative way that would 1) still perfectly support the current usecases while 2) also enabling Ventoy to be used as a standlone EFI payload and 3) also support better alignment for the Ventoy data partition, which is already desirable for large drives (ex: some Samsung drives should be aligned to 8k, not just 4k) and could be even more desirable in the future while 4) preventing uses from accidentally messing with the VTOYEFI partition by marking it as EFI, meaning extra permissions would be required to access it on Windows

Maybe these changes could be integrated in Ventoy as it all seems for the better?

Alternatively, maybe you could use them in a way that would prevent taking any risk for Ventoy itself, for example to create a separate and different EFI payload (say, Ventoz.efi?), that could be used by those who want to keep Ventoy features on their hard drive?

My analysis:

• The core issue seems to be you hardcode 2048 as the start of the first partition: if the verification fails, it cause an error that was non blocking before (just a 10 seconds delay) but that now causes an exit and a reboot:
  https://github.com/ventoy/Ventoy/blob/836e1aa11e78ef170f0017631b2014b241a47f72/EDK2/edk2_mod/edk2-edk2-stable201911/MdeModulePkg/Application/Ventoy/Ventoy.c around like 580 that triggers ventoy_warn_invalid_device()

• You could do that better by creating a 3rd GPT partition from 34 to 2047, type ef02 in gdisk, to "protect" the post MBR gap, which is something already recommended in some install guides cf https://wiki.archlinux.org/title/GRUB

• In fdisk, I noticed you have the MBR partition 1 be EE ("protective") : I would also suggest moving to an hybrid GPT/MBR partition table instead, using type EE for 1->2047 as a way to protect both the boot record and the 34->2047 partition, then following this first protective partition by the regular partitions: this would also make the Ventoy volumes accessible by older OS not supporting GPT partition tables, since after this protective partition you would have regular MBR partitions using the same offsets as the GPT partitions (example below)

• In gdisk, I see you use the GPT type 0700 for the VTOYEFI partition: maybe you could move it to EF00 and check if it contains the Ventoy.EFI payload

• Such changes would be a better way to identify the drive as "ventoyed": they more accurate and unique that just checking if the data partition starts at 2048, while also supporting better aligned Ventoy data partitions (4k, 8k...)

• They would also allow people to install Ventoy on their hard drive to boot Ventoy automatically when in CSM mode, or through a menu entry in UEFI mode, when adding Ventoy.efi after having "ventoyed" the hard drive by:

1. recreating a similar partition in gdisk type ef02 / guid 21686148-6449-6E6F-744E-656564454649 from 34 to 2047, which is often unused space on modern OS,

2. doing a cat or a dd to populate this new partition of their system drive with the boot record from their Ventoy drive,

3. for UEFI boot: by adding the Ventoy.efi payload to the EFI boot menu (using the bios or efibootmgr),

4. for automatic CSM boot with support for older OS: by making an hybrid MBR/GPT partition table with fdisk or gdisk (ok, it's something slightly hard, but if they can't do that, maybe they shouldn't try to use Ventoy this way...).

Now, on a practical basis, if we use my 118G drive as an example, it currently is:

• in GPT: gdisk p

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048       180289495   86.0 GiB    0700  Ventoy
   2       180289496       180355031   32.0 MiB    0700  VTOYEFI
   3       180355072       247463902   32.0 GiB    8300  Linux filesystem

• in MBR: fdisk M p

Device     Boot Start       End   Sectors  Size Id Type
/dev/sda1           1 247463935 247463935  118G ee GPT

It would become:

• in GPT: gdisk p

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048       180289495   86.0 GiB    0700  Ventoy
   2       180289496       180355031   32.0 MiB    EF00  VTOYEFI
   3       180355072       247463902   32.0 GiB    8300  Linux filesystem
   4              34            2047   1007.0 KiB  EF02  BIOS boot partition

• Create the hybrid MBR/GPT in gdisk with: r h 1 2 3 y enter n ef n enter y o

Number  Boot  Start Sector   End Sector   Status      Code
   1                     1         2047   primary     0xEE
   2                  2048    180289495   primary     0x07
   3             180289496    180355031   primary     0xEF
   4      *      180355072    247463902   primary     0x83

• in MBR as seen by fdisk M p

/dev/sda1               1      2047      2047 1023.5K ee GPT
/dev/sda2            2048 180289495 180287448     86G  7 HPFS/NTFS/exFAT
/dev/sda3       180289496 180355031     65536     32M ef EFI (FAT-12/16/32)
/dev/sda4  *    180355072 247463902  67108831     32G 83 Linux

Now, for the Ventoy code: the check would simply become: "is there an out of order GPT partition type ef02 from 34 to 2047 + an hybrid GPT/MBR partition table with a protective EE partition from 1 to 2047:

So you would go from something like:

            pMBR = (MBR_HEAD *)pBuffer;
            if (pMBR->PartTbl[0].FsFlag != 0xEE)
            {
                if (pMBR->PartTbl[0].StartSectorId != 2048 ||
                    pMBR->PartTbl[1].SectorCount != 65536 ||
                    pMBR->PartTbl[1].StartSectorId != pMBR->PartTbl[0].StartSectorId + pMBR->PartTbl[0].SectorCount)
                {
                    debug("Failed to check disk part table");
                    ventoy_warn_invalid_device();
                }
            }

To something like:

            pMBR = (MBR_HEAD *)pBuffer;
            if (pMBR->PartTbl[0].FsFlag != 0xEE)
            {
                // here PartTbl is assumed to be GPT partitions: the "ventoyed" check is that
                //  the first one in sector order (which is the last one by number)
                //  contains the necessary boot record; 
                if (
                    // first the "ventoyed" check
                    // you can also check if it is gdisk ef02 aka GUID 21686148-6449-6E6F-744E-656564454649
                    (pMBR->PartTbl[-1].StartSectorId != 34 ||
                    pMBR->PartTbl[-1].StopSectorId != 2047)              
                    // followed by checking either:
                    // case A: for a Ventoy drive
                    !(
                    pMBR->PartTbl[1].FsFlag == 0xEF &&
                    pMBR->PartTbl[1].SectorCount == 65536 &&
                    pMBR->PartTbl[1].StartSectorId != pMBR->PartTbl[0].StartSectorId + pMBR->PartTbl[0].SectorCount)
                    ) ||
                    // case B: or for an EFI partition, checking if the VTOYEFI partition
                    // size 32M has been replaced by checking for a regular EFI partition
                    // where any size is acceptable as long as it contains the Ventoy payload
                    // which could implicit if we are executing this code, but which could also check for a given file
                    !(
                    pMBR->PartTbl[0].FsFlag == 0xEF
                    // && could also check if \EFI\Boot\Ventoy.efi exists
                    )
                {
                    debug("Failed to check disk hybrid part table");
                    ventoy_warn_invalid_device();
                }
            }

The only slight difficulty would be to handle the optional "reserved space" where extra partition can go, but if you make this special 34->2047 partition the last GPT partition by convention (hence -1) or use some given number (say 7E to be the 126th partition), it becomes a moot point: either people would accidentally erase this partition when creating their own, triggering the boot failure, or they would not touch it and create their own partitions (from 2 to 126) without damaging it.

The good thing is, if they did accidentally erase this special partition, they could always fix that by recreating it, which would make the drive bootable again: the ventoy_warn_invalid_device() could point to a link explaining what happened with instructions on how to fix that.

If may sound complicated, so I could understand if you preferred to create a separate Ventoz.efi payload for hard drive use, but it seems like a nice simple hack that would be both more accurate and super helpful: I often make my EFI partitions from 4G or 8G and I fill them with ISOs of all the distributions I use or try, along with the EFI payloads.

This would allow me to also stick Ventoy on this hard drive partition to avoid handcrafting EFI payloads from ISOs, opening new use cases for Ventoy.

[csdvrx]

I'm preparing a binary for other people who apparently want that feature like https://forums.ventoy.net/showthread.php?tid=695

BTW oops I just realized pMBR->PartTbl is the protective MBR ; if anyone want to quickly test this usecase using the current ventoy, I think you can simply fool the first part of the test by creating an hybrid MBR GPT with the protective partition EE first, from 2048 to wherever the beginning of the second partition is, in my case 56631295, for example with fdisk M x b 1 2048 to "push" the beginning to 2048 ; for the 2nd part of the test (partition 2 should start immediately after) I see no such function in fdisk so I'm resorting to a manual hexedit of the MBR partition table at offset 0x1BE (google, there are many guides explaining how to do that)

The only small issue is the partition overlap: I wonder if the grub2 from Ventoy will use the partions declared in the protective MBR (0xEE from 2048 to 56631295 : bad, as it only starts at 8192) or the GPT partition (0700 from 8192 to 56631295 : good exfat) as there seem to be many small changes applied to grub2...

Well, I'll try and see :)

[Pacorretaco]

Hello,
I will come from a slightly different angle about the first partition being forcefully at 1MB and why it would be good to go over this strict restriction.

Many modern flash devices are coming by design with a specific offset in their main partition from the factory.
Supposedly this helps with bad block management, for increased longevity and performance.
The official SD Formatter tool provided by the global SD association is also actively leaving this unallocated space at the beginning of the cards when they are formatted. They openly recommend everyone to use this tool for best results.
https://www.sdcard.org/press/thoughtleadership/the-sd-memory-card-formatterhow-this-handy-tool-solves-your-memory-card-formatting-needs/

For example: 128GB SDXC card comes from the factory with 16MB unallocated space at the beginning of the flash.
256GB SDXC has a 32MB offset.

The problem is that Ventoy will refuse to work as long as this offset is kept. It strictly looks for the main partition to be found specifically at 1MB.
So it wants to repartition in its conventional way, making the offset disappear.
This may appear to work well enough, and many users will not even notice what happened.

But it is conflicting with the manufacturer design and could potentially be affecting the long term reliablility and performance of the devices.

So hopefully there could be a way to overcome these restrictions or at least give a way around this 1MB offset requirement, so that both considerations may be satisfied.

In any case, I would like to hear some thoughts on this.

Thank you

[csdvrx]

@Pacorretaco it is a very valid concern: SD cards try to optimize alignment by putting their default partitions in the optimal position.

I don't see any reason why @ventoy should stick to the rigid "partition at 1M" logic instead of looking at the partition table for specific clues + maybe adding a simple heuristic like checking the existence of VENTOY.EFI.

[RaXorX]

@csdvrx Were you able to make any progress on this regards?
I've so long wanting to use ventoy but all this forcedness puts me away from this project. It is simple, it is good. Doesn't mean it has to remain simple for everyone and can't be in an expert mode, which apperently the dev does really puts his heart to not implement.

All he has to do is simply give a warning to not give support on certain use cases but nope, some devs are in a really bad habit to instead lock everything out just because they are fed up of giving support to others. If you don't want to take that headache, you can also say the same instead of hard locking everything up.

[csdvrx]

@RaXorX I haven't, because compiling Ventoy depends on a specific version of Fedora I think, and last time I checked it was an old version and I have not enough free space for old stuff :)

However, you should be able to apply my proposed patch: the partition starting at 2048 causes the most problems IIRC.

Just compile and test: you shouldn't get the error message anymore, just like in the previous versions that allowed custom layouts.

You'd have to partition the drive as indicated above (use the unclaimed space from 34 to 2047), say with gdisk - but I think that's all you need.

It's been a while since I looked, but you should be able to your freshly compiled .EFI file on the EFI partition and boot to it, for example using your BIOS UEFI menu or Linux efibootmgr: it should present you Ventoy usual menu.

Let me know if you need any help. The changes required are not complicated. What's complicated is making sure they play nice with everything else. But a dedicated partition right into the unclaimed space before 2048 looks like the simplest way to achieve this.

[RaXorX]

@csdvrx Hi. I used to stray away from compiling ventoy myself in the past because of the requirements. I think it's CentOS not fedora. But gladly, this time around I saw that they have recently updated their repo to include GitHub actions, which work nicely. I have already compiled and patched ventoy for myself (just removing the unverified source or whatever the message it shows if ventoy is installed without using their installer alongside other boot menus)

Here's the repo if you want to take on a look and I'd be happy to see your patches there. Otherwise I have not much clue what I need to change and how sadly.
https://github.com/raXorX/ventoyPatched

You can check the patched branch. Sadly these patches are only to use ventoy alongside other boot menus and doesn't remove the other restrictions.

[beren12]

I would love to see better standalone support as well. I want larger EFI partition so I can have rEFInd be the default and then pick to boot ventoy or not. For now I suppose I can make a rEFInd iso and ventoy can chainboot that but it's less than ideal and really not needed. I made a ventoy usb drive and enlarged the efi partition but then ventoy told me it wasn't official and refused to boot.

Another thing that would help with this is instead of writing a disk image to the partition, let the user override the partition size and call mkfs.vfat and copy the files to the partition.