UEFI:NTFS - Unable to start driver: [1] or Could not start NTFS partition service: [14]

[zanechua]

Checklist

• I looked at https://github.com/pbatard/rufus/wiki/FAQ to see if my question has already been answered.
• I performed a search in the issue tracker for similar issues using keywords relevant to my problem, such as the error message I got from the log.
• I clicked the "Log" button in Rufus (last small blue button before START) and copy/pasted the log into the line that says <FULL LOG> below.
• The log I am copying is the FULL log, starting with the line Rufus version: x.y.z - I have NOT removed any part of it.

Additionally (if applicable):

• I ran a bad blocks check, by clicking Show advanced format options then Check device for bad blocks, and confirmed that my USB is not defective.
• I also tried one or more of the following:
  • Using a different USB drive.
  • Plugging the USB into a different port.
  • Running Rufus on a different computer.
• If using an image, I clicked on the (✓) button to compute the MD5, SHA1 and SHA256 checksums, which are therefore present in the log I copied. I confirmed, by performing an internet search, that these values match the ones from the official image.

Issue description

Motherboard: S2600CP4
Bios Revision: R02.06.E006
Rufus version: 3.3

Here's the log from Rufus:
https://gist.github.com/zanechua/36c2416c9d8035623a0efb51a176d7af

Here are the errors I am getting on boot:

EDIT:

So it seems like the last version of Rufus v2 works.

Rufus version: 2.18

Here's the log from Rufus:
https://gist.github.com/zanechua/acf8c67c7026814796f7da6414dd6e3e

You are mentioning that you went through a BIOS update, so I wonder if that update is preventing EFI executables with IMAGE_SUBSYSTEM_EFI_BOOT_SERVICE_DRIVER from running.

I don't think it's the bios update though since 2.18 works. I haven't formatted this computer in awhile so I might have been using 2.18 and this is the first time I am using 3.x on this computer.

[pbatard]

Thanks for opening a new issue.

As I mentioned in the previous one, I suspect that the problem comes from the EfiFs NTFS driver having switched from using the IMAGE_SUBSYSTEM_EFI_RUNTIME_DRIVER PE type to IMAGE_SUBSYSTEM_EFI_BOOT_SERVICE_DRIVER, which I suspect some UEFI Firmwares might not like. The problem however is that I had to change that attribute because it was creating issues with other UEFI firmwares, and IMAGE_SUBSYSTEM_EFI_BOOT_SERVICE_DRIVER is the proper type to use for what UEFI:NTFS is doing.

You are mentioning that you went through a BIOS update, so I wonder if that update is preventing EFI executables with IMAGE_SUBSYSTEM_EFI_BOOT_SERVICE_DRIVER from running.

Here's what you can do to validate that this is indeed the issue, using the USB you created with Rufus 3.3:

• Download CFF Explorer from here. If you don't know which version you should pick, I'd recommend the CFF Explorer (x86 Version, stand-alone, Zip Archive) as this is smaller and it doesn't need to be installed.

• Plug in the Flash Drive created by Rufus. Since you're running on Windows 10, you should be able to access both the partitions Rufus created on that drive, including the UEFI:NTFS. However, Windows may not mount it automatically, and Disk Management seems to have trouble displaying small partition to do the following with the GUI, in which case you will have to assign a letter manually using diskpart. In case you cannot see the UEFI_NTFS partition, you should:

  • Open an elevated command prompt

  • Run diskpart

  • Run list vol. This will list all the volumes available, and you should see something like Volume 7 UEFI_NTFS FAT Removable 512 KB Healthy in your list. Take a note of the volume number for the UEFI_NTFS partition.

  • Run sel vol # where # is the number above (e.g. sel vol 7)

  • Run assign letter=X where X is any drive letter that is not already in use. After that, you should be able to access the UEFI:NTFS partition from Windows Explorer. A screenshot of the procedure is also provided below:

    

• With both CFF Explorer and the UEFI:NTFS partiton available, launch CFF Explorer

• Open EFI\Rufus\ntfs_x64.efi on the UEFI:NTFS partition

• On the left pane navigate to Nt Headers → Optional Header. This should allow you to view the Subsystem property and see that it is set to EFI Boot Driver:

  

• Click on EFI Boot Driver and change it to EFI Runtime Driver and then click the Save button (select Yes when it asks you to overwrite the file)

• Try to boot the USB again and see if that works better.

Of course, this is just one possibility, and I may be off mark (especially it's possible that I had already changed the subsystem type before 2.18, in which case the above probably won't help).

The other thing that I'd like you to try, to confirm that this has to do with the EFI driver, is to pick the EFI\Rufus\ntfs_x64.efi from the 2.18 drive and copy it to the 3.3 drive (replacing the existing one) and see how that works. Alternatively, you can try each of the ntfs_x64.efi from https://efi.akeo.ie/downloads/, starting with EfiFs 1.0 (and then selecting the x64 repository) to see which one works and which one doesn't, as this may help me pinpoint what the issue might be.

[zanechua]

Thanks for the very very detailed guide on how to debug this specifically.

I changed the Subsystem header of the v3.3 ntfs_x64.efi file to EFI Runtime Driver like you said but got a different error this time.

However copying the file from 2.18 to the drive created by 3.3 allows the drive to boot perfectly fine.

The ntfs_x64.efi file from the 2.18 drive already has the Subsystem header set to EFI Boot Driver

[pbatard]

Thanks for the test. I probably should have checked the driver from 2.18 before I asked you to run that test.

As I mentioned earlier, can I ask you to please try all of the drivers ntfs_x64.efi from https://efi.akeo.ie/downloads/, starting from efifs-1.0/ and let me know the last version that works with your 3.3 flash drive?

[zanechua]

Oh. I misssed that part sorry. Will get to it.

Tested the following:
v1 (same hash as v2.18): boots fine
v1.1: boots fine
v1.2: errors out
v1.3: errors out

Boot Images:

So it seems like somewhere between v1.1 -> v1.2 is where a breaking change happened.

[pbatard]

Thanks for the update, but next time please don't edit a comment to add new data, as I didn't see your tests with the various version until now! It's much better to create a new entry, as I do get a notification.

There were a lot of changes between 1.1 and 1.2, and since you are the only person with a machine that can replicate the problem, this is probably going to take a lot of back and forth to troubleshoot, so it may take a while.

I guess the first thing I'm going to test, since 1.2 changed the compiler being used, it to try with a different compiler. I can not revert to the old one (Clang/C2) since support for it is being dropped by Microsoft in the latest version of Visual Studio, which is part of the reason why I switched compiler for the binary in the first place. Since version 1.2, the NTFS driver binary is compiled with the EDK2, so a good test will be to try non EDK2 versions.

If you look in https://rufus.ie/testing/ you'll see that I uploaded 2 drivers: ntfs_x64_gcc.efi and ntfs_x64_msvc.efi. These are the latest versions of the NTFS driver, but compiled with GCC and MSVC respectively. Can I please ask you to test each one (by removing the _gcc or _msvc part and replacing your existing ntfs_x64.efi) and see how it goes?

[zanechua]

Sure.

Can I verify that the hash of the ntfs_x64.efi file in the v3.3 of rufus is 872FC3150C9DC39470CF0E03732642789CF8F82F? and it's 47KB?

Both testing drivers work totally fine and boot up as they should. Only the one included with rufus v3.3 doesn't.

[pbatard]

Many thanks for the new tests!

Yes, the sha1sum of ntfs_x64.efi from Rufus 3.3 is 872fc3150c9dc39470cf0e03732642789cf8f82f. And it's 46.3 KB in size. Basically, you can find the file included in Rufus if you extract it from the current uefi-ntfs.img we have in our source tree at https://github.com/pbatard/rufus/tree/master/res/uefi (using something like 7-zip for instance).

It's quite interesting that the test MSVC version works, whereas the original one doesn't, as the EDK2 version used by Rufus uses MSVC for compilation behind the scenes.

So, just to make absolutely sure that this is not an EDK2 issue that has since been fixed, can I please ask you to try one more test? I have just recompiled a new EDK2 driver, which you can find as ntfs_x64_edk2.efi under https://rufus.ie/testing/. Can you run the same test with that one as you did with the others, and confirm that it fails?

If it does, I guess I'll have to figure out the differences between the way the EDK2 invokes MSVC and the way the regular Visual Studio solution does (besides the use of gnu-efi, which I would assert should be the main difference), so that I can try to produce a driver that works.

Else, I guess I'm going to have to switch to compiling all the drivers with Visual Studio + gnu-efi instead of EDK2, which is fine, but which will increase the size of the binaries and therefore the size of Rufus (the reason I'm using EDK2 + MSVC is because it produces the smallest binaries). And it still won't tell me exactly what it is between your platform and these drivers that is causing this weird incompatibility...

[zanechua]

I can confirm that the EDK2 driver fails.

If you need more information let me know and I'll do my best to provide it.

[pbatard]

Thanks.

I think right now I need to process all this good data you provided, and come up with some new things to try, to zero the issue further (which probably won't happen before next week at this stage).

At any rate, your testing and your patience with this issue are much appreciated!

[Mattiwatti]

I can't reproduce this myself (of course...), but EDK2 does make quite a few drastic changes to the standard MSVC build environment in the interest of saving space. I'm going to go ahead and assume EDK2 itself is not at fault (meaning the headers as opposed to the gnu-efi ones) as I've used both for years with no issues. But comparing ntfs_x64_edk2.efi vsntfs_x64_msvc.efi/ntfs_x64_gcc.efi, there are a number of differences the DXE core loader could be stumbling over in the EDK2 PE files compared to the MSVC/GCC ones:

• File and section alignment. These are set to 0x20 by default by EDK2, which is very very aggressive. Try the default MSVC /ALIGN:0x1000 /FILEALIGN:0x200 first and see if that helps. This is IMO the most likely culprit.
• No image base. Probably not the cause since the image should be relocated, but link with /BASE:0x180000000 just in case.
• Subsystem version not set (should be 1.0): link with /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER,1.0 to correct this.
• Checksum not set: it could be that OP's BIOS is being pedantic about this (but if so, it seems strange to me that it's not complaining about the file being unsigned). You can link with /RELEASE to set a checksum, though this will most likely be wrong because EDK2 runs some post-build steps that alter the image. Use something like CFF Explorer to fix this as the very last step.
• SizeOfStackReserve, SizeOfStackCommit, SizeOfHeapReserve and SizeOfHeapCommit have all been zeroed. I have no idea if this matters for OP's BIOS (odds are it doesn't) but I would just set these to 0x100000, 0x1000, 0x100000 and 0x1000 respectively with a hex editor to be on the safe side.
• The debug directory contents have been wiped by EDK, but the PE debug directory still exists and points to empty data. Zero 8 bytes at offset 0x168 to get rid of this entry.
• The exception directory has been removed. I highly doubt this matters as there is no SEH in EFI, but on the other hand, the MS boot loaders do keep this directory around and they probably thought about this. I dunno.

To get most of the above in order you can put the following in the .inf file (obviously applies only to that package - for 'affects everything' scope edit tools_def.txt instead):

[BuildOptions]
  MSFT:*_*_*_DLINK_FLAGS = /ALIGN:0x1000 /FILEALIGN:0x200 /BASE:0x180000000 /SECTION:.pdata,!D /MERGE:.rdata=.text /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER,1.0 /RELEASE
  MSFT:*_*_X64_GENFW_FLAGS = --keepexceptiontable --keepzeropending --keepoptionalheader

[pbatard]

@Mattiwatti, many thanks for the extended analysis. This should indeed be exceedingly helpful, and I'll try to produce and upload drivers that apply your suggestions, to see if we can validate your hypotheses.