Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UEFI:NTFS - Unable to start driver: [1] or Could not start NTFS partition service: [14] #1213

Closed
9 of 10 tasks
zanechua opened this issue Sep 21, 2018 · 74 comments
Closed
9 of 10 tasks
Assignees
Milestone

Comments

@zanechua
Copy link

zanechua commented Sep 21, 2018

Checklist

  • I looked at https://github.com/pbatard/rufus/wiki/FAQ to see if my question has already been answered.
  • I performed a search in the issue tracker for similar issues using keywords relevant to my problem, such as the error message I got from the log.
  • I clicked the "Log" button in Rufus (last small blue button before START) and copy/pasted the log into the line that says <FULL LOG> below.
  • The log I am copying is the FULL log, starting with the line Rufus version: x.y.z - I have NOT removed any part of it.

Additionally (if applicable):

  • I ran a bad blocks check, by clicking Show advanced format options then Check device for bad blocks, and confirmed that my USB is not defective.
  • I also tried one or more of the following:
    • Using a different USB drive.
    • Plugging the USB into a different port.
    • Running Rufus on a different computer.
  • If using an image, I clicked on the (✓) button to compute the MD5, SHA1 and SHA256 checksums, which are therefore present in the log I copied. I confirmed, by performing an internet search, that these values match the ones from the official image.

Issue description

Motherboard: S2600CP4
Bios Revision: R02.06.E006
Rufus version: 3.3

Here's the log from Rufus:
https://gist.github.com/zanechua/36c2416c9d8035623a0efb51a176d7af

Here are the errors I am getting on boot:
error_1

error_2

EDIT:

So it seems like the last version of Rufus v2 works.

Rufus version: 2.18

Here's the log from Rufus:
https://gist.github.com/zanechua/acf8c67c7026814796f7da6414dd6e3e

success

You are mentioning that you went through a BIOS update, so I wonder if that update is preventing EFI executables with IMAGE_SUBSYSTEM_EFI_BOOT_SERVICE_DRIVER from running.

I don't think it's the bios update though since 2.18 works. I haven't formatted this computer in awhile so I might have been using 2.18 and this is the first time I am using 3.x on this computer.

@pbatard
Copy link
Owner

pbatard commented Sep 21, 2018

Thanks for opening a new issue.

As I mentioned in the previous one, I suspect that the problem comes from the EfiFs NTFS driver having switched from using the IMAGE_SUBSYSTEM_EFI_RUNTIME_DRIVER PE type to IMAGE_SUBSYSTEM_EFI_BOOT_SERVICE_DRIVER, which I suspect some UEFI Firmwares might not like. The problem however is that I had to change that attribute because it was creating issues with other UEFI firmwares, and IMAGE_SUBSYSTEM_EFI_BOOT_SERVICE_DRIVER is the proper type to use for what UEFI:NTFS is doing.

You are mentioning that you went through a BIOS update, so I wonder if that update is preventing EFI executables with IMAGE_SUBSYSTEM_EFI_BOOT_SERVICE_DRIVER from running.

Here's what you can do to validate that this is indeed the issue, using the USB you created with Rufus 3.3:

  • Download CFF Explorer from here. If you don't know which version you should pick, I'd recommend the CFF Explorer (x86 Version, stand-alone, Zip Archive) as this is smaller and it doesn't need to be installed.

  • Plug in the Flash Drive created by Rufus. Since you're running on Windows 10, you should be able to access both the partitions Rufus created on that drive, including the UEFI:NTFS. However, Windows may not mount it automatically, and Disk Management seems to have trouble displaying small partition to do the following with the GUI, in which case you will have to assign a letter manually using diskpart. In case you cannot see the UEFI_NTFS partition, you should:

    • Open an elevated command prompt

    • Run diskpart

    • Run list vol. This will list all the volumes available, and you should see something like Volume 7 UEFI_NTFS FAT Removable 512 KB Healthy in your list. Take a note of the volume number for the UEFI_NTFS partition.

    • Run sel vol # where # is the number above (e.g. sel vol 7)

    • Run assign letter=X where X is any drive letter that is not already in use. After that, you should be able to access the UEFI:NTFS partition from Windows Explorer. A screenshot of the procedure is also provided below:

      image1

  • With both CFF Explorer and the UEFI:NTFS partiton available, launch CFF Explorer

  • Open EFI\Rufus\ntfs_x64.efi on the UEFI:NTFS partition

  • On the left pane navigate to Nt HeadersOptional Header. This should allow you to view the Subsystem property and see that it is set to EFI Boot Driver:

    image2

  • Click on EFI Boot Driver and change it to EFI Runtime Driver and then click the Save button (select Yes when it asks you to overwrite the file)

  • Try to boot the USB again and see if that works better.

Of course, this is just one possibility, and I may be off mark (especially it's possible that I had already changed the subsystem type before 2.18, in which case the above probably won't help).

The other thing that I'd like you to try, to confirm that this has to do with the EFI driver, is to pick the EFI\Rufus\ntfs_x64.efi from the 2.18 drive and copy it to the 3.3 drive (replacing the existing one) and see how that works. Alternatively, you can try each of the ntfs_x64.efi from https://efi.akeo.ie/downloads/, starting with EfiFs 1.0 (and then selecting the x64 repository) to see which one works and which one doesn't, as this may help me pinpoint what the issue might be.

@pbatard pbatard self-assigned this Sep 21, 2018
@zanechua
Copy link
Author

zanechua commented Sep 22, 2018

Thanks for the very very detailed guide on how to debug this specifically.

I changed the Subsystem header of the v3.3 ntfs_x64.efi file to EFI Runtime Driver like you said but got a different error this time.

different_error

However copying the file from 2.18 to the drive created by 3.3 allows the drive to boot perfectly fine.

The ntfs_x64.efi file from the 2.18 drive already has the Subsystem header set to EFI Boot Driver

cffv218

@pbatard
Copy link
Owner

pbatard commented Sep 22, 2018

Thanks for the test. I probably should have checked the driver from 2.18 before I asked you to run that test.

As I mentioned earlier, can I ask you to please try all of the drivers ntfs_x64.efi from https://efi.akeo.ie/downloads/, starting from efifs-1.0/ and let me know the last version that works with your 3.3 flash drive?

@zanechua
Copy link
Author

zanechua commented Sep 23, 2018

Oh. I misssed that part sorry. Will get to it.

Tested the following:
v1 (same hash as v2.18): boots fine
v1.1: boots fine
v1.2: errors out
v1.3: errors out

Boot Images:
success_v11
error_v12

So it seems like somewhere between v1.1 -> v1.2 is where a breaking change happened.

@pbatard
Copy link
Owner

pbatard commented Sep 25, 2018

Thanks for the update, but next time please don't edit a comment to add new data, as I didn't see your tests with the various version until now! It's much better to create a new entry, as I do get a notification.

There were a lot of changes between 1.1 and 1.2, and since you are the only person with a machine that can replicate the problem, this is probably going to take a lot of back and forth to troubleshoot, so it may take a while.

I guess the first thing I'm going to test, since 1.2 changed the compiler being used, it to try with a different compiler. I can not revert to the old one (Clang/C2) since support for it is being dropped by Microsoft in the latest version of Visual Studio, which is part of the reason why I switched compiler for the binary in the first place. Since version 1.2, the NTFS driver binary is compiled with the EDK2, so a good test will be to try non EDK2 versions.

If you look in https://rufus.ie/testing/ you'll see that I uploaded 2 drivers: ntfs_x64_gcc.efi and ntfs_x64_msvc.efi. These are the latest versions of the NTFS driver, but compiled with GCC and MSVC respectively. Can I please ask you to test each one (by removing the _gcc or _msvc part and replacing your existing ntfs_x64.efi) and see how it goes?

@zanechua
Copy link
Author

zanechua commented Sep 25, 2018

Sure.

Can I verify that the hash of the ntfs_x64.efi file in the v3.3 of rufus is 872FC3150C9DC39470CF0E03732642789CF8F82F? and it's 47KB?

Both testing drivers work totally fine and boot up as they should. Only the one included with rufus v3.3 doesn't.

success_v13

@pbatard
Copy link
Owner

pbatard commented Sep 25, 2018

Many thanks for the new tests!

Yes, the sha1sum of ntfs_x64.efi from Rufus 3.3 is 872fc3150c9dc39470cf0e03732642789cf8f82f. And it's 46.3 KB in size. Basically, you can find the file included in Rufus if you extract it from the current uefi-ntfs.img we have in our source tree at https://github.com/pbatard/rufus/tree/master/res/uefi (using something like 7-zip for instance).

It's quite interesting that the test MSVC version works, whereas the original one doesn't, as the EDK2 version used by Rufus uses MSVC for compilation behind the scenes.

So, just to make absolutely sure that this is not an EDK2 issue that has since been fixed, can I please ask you to try one more test? I have just recompiled a new EDK2 driver, which you can find as ntfs_x64_edk2.efi under https://rufus.ie/testing/. Can you run the same test with that one as you did with the others, and confirm that it fails?

If it does, I guess I'll have to figure out the differences between the way the EDK2 invokes MSVC and the way the regular Visual Studio solution does (besides the use of gnu-efi, which I would assert should be the main difference), so that I can try to produce a driver that works.

Else, I guess I'm going to have to switch to compiling all the drivers with Visual Studio + gnu-efi instead of EDK2, which is fine, but which will increase the size of the binaries and therefore the size of Rufus (the reason I'm using EDK2 + MSVC is because it produces the smallest binaries). And it still won't tell me exactly what it is between your platform and these drivers that is causing this weird incompatibility...

@zanechua
Copy link
Author

I can confirm that the EDK2 driver fails.

If you need more information let me know and I'll do my best to provide it.

@pbatard
Copy link
Owner

pbatard commented Sep 25, 2018

Thanks.

I think right now I need to process all this good data you provided, and come up with some new things to try, to zero the issue further (which probably won't happen before next week at this stage).

At any rate, your testing and your patience with this issue are much appreciated!

@Mattiwatti
Copy link
Contributor

I can't reproduce this myself (of course...), but EDK2 does make quite a few drastic changes to the standard MSVC build environment in the interest of saving space. I'm going to go ahead and assume EDK2 itself is not at fault (meaning the headers as opposed to the gnu-efi ones) as I've used both for years with no issues. But comparing ntfs_x64_edk2.efi vsntfs_x64_msvc.efi/ntfs_x64_gcc.efi, there are a number of differences the DXE core loader could be stumbling over in the EDK2 PE files compared to the MSVC/GCC ones:

  • File and section alignment. These are set to 0x20 by default by EDK2, which is very very aggressive. Try the default MSVC /ALIGN:0x1000 /FILEALIGN:0x200 first and see if that helps. This is IMO the most likely culprit.
  • No image base. Probably not the cause since the image should be relocated, but link with /BASE:0x180000000 just in case.
  • Subsystem version not set (should be 1.0): link with /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER,1.0 to correct this.
  • Checksum not set: it could be that OP's BIOS is being pedantic about this (but if so, it seems strange to me that it's not complaining about the file being unsigned). You can link with /RELEASE to set a checksum, though this will most likely be wrong because EDK2 runs some post-build steps that alter the image. Use something like CFF Explorer to fix this as the very last step.
  • SizeOfStackReserve, SizeOfStackCommit, SizeOfHeapReserve and SizeOfHeapCommit have all been zeroed. I have no idea if this matters for OP's BIOS (odds are it doesn't) but I would just set these to 0x100000, 0x1000, 0x100000 and 0x1000 respectively with a hex editor to be on the safe side.
  • The debug directory contents have been wiped by EDK, but the PE debug directory still exists and points to empty data. Zero 8 bytes at offset 0x168 to get rid of this entry.
  • The exception directory has been removed. I highly doubt this matters as there is no SEH in EFI, but on the other hand, the MS boot loaders do keep this directory around and they probably thought about this. I dunno.

To get most of the above in order you can put the following in the .inf file (obviously applies only to that package - for 'affects everything' scope edit tools_def.txt instead):

[BuildOptions]
  MSFT:*_*_*_DLINK_FLAGS = /ALIGN:0x1000 /FILEALIGN:0x200 /BASE:0x180000000 /SECTION:.pdata,!D /MERGE:.rdata=.text /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER,1.0 /RELEASE
  MSFT:*_*_X64_GENFW_FLAGS = --keepexceptiontable --keepzeropending --keepoptionalheader

@pbatard
Copy link
Owner

pbatard commented Sep 30, 2018

@Mattiwatti, many thanks for the extended analysis. This should indeed be exceedingly helpful, and I'll try to produce and upload drivers that apply your suggestions, to see if we can validate your hypotheses.

Repository owner deleted a comment from jjboy32 Oct 3, 2018
Repository owner deleted a comment from jjboy32 Oct 3, 2018
@pbatard
Copy link
Owner

pbatard commented Oct 4, 2018

Finally got around to compile a NTFS driver that includes @Mattiwatti's build options suggestion. So let's find out if it's really one of these compilation settings that is really causing the issue or something else.

The driver is called ntfs_x64_edk2_TEST1.efi and can be found at the usual place.

@zanechua, if you can conduct a test with this driver, as you did with the other ones, I would realy appreciate it.

Also, for the record, the linker issued the following warning, which means that the /MERGE:.rdata=.text was most likely ignored. But since the other parameters should have been applied that doesn't worry me too much:

LINK : warning LNK4253: section '.rdata' not merged into '.text';  already merged into '.data'

@Mattiwatti
Copy link
Contributor

That warning is not an issue, it just means some other EDK2 build setting already applied /MERGE:.rdata=.data instead. Both options will merge the .rdata section into another one, saving space, but neither of them is either harmful or useful as far as OP's issue goes.

The only difference is that /MERGE:.rdata=.text will make .rdata contents executable, which is usually not a problem. /MERGE:.rdata=.data will make .rdata contents writable, which can cause issues when debugging because .rdata is supposed to be for read-only data (hence the name) and any bug where you accidentally write to const data won't be detected. If your debug build configuration does not merge .rdata into .data then this is not an issue.

@zanechua
Copy link
Author

zanechua commented Oct 5, 2018

Tested ntfs_x64_edk2_TEST1.efi and I still get the same error.

@pbatard
Copy link
Owner

pbatard commented Oct 5, 2018

Thanks for the test, it was definitely worth a try. My guess would still have to be that EDK2 has deprecated APIs that gnu-efi hasn't, therefore I will replace all the x86 drivers with their gnu-efi version in the next release of Rufus. For the sake of not increasing the size of the Rufus executable too dramatically, I think I'll keep the ARM/EDK2 versions though, as I would surmise that any ARM UEFI firmware should be modern enough to work with drivers compiled with a recent EDK2.

@pbatard pbatard added this to the 3.4 milestone Oct 6, 2018
@devedse
Copy link

devedse commented Oct 19, 2018

Hi all,

Just an update, I'm also running into this error using my Dell Lattitude E6520. I'm currently trying to see if 2.18 also resolved this issue.

@pbatard
Copy link
Owner

pbatard commented Oct 19, 2018

@devedse, can you please indicate if you're getting errorcode 1 or 14, or some other.
Even if you are getting the same error, it always helps to copy the exact message you are getting, just to confirm that you are indeed seeing the same error.

@devedse
Copy link

devedse commented Oct 19, 2018

Didn't manage to make a screenshot as I went back to 2.18 and that worked. I'll see if I can create a new usb stick with the 3.3 version.

Also one thing I don't quite understand is that I can select fat32 in Rufus 2.18 but not in 3.3.

@devedse
Copy link

devedse commented Oct 19, 2018

I tried 2 diffent UEFI files on the disk, 1:
image
image

And 2:
image

(Just a black screen)
image

@pbatard
Copy link
Owner

pbatard commented Oct 20, 2018

Well, for 2 that's a given. ntfs_x64.exi is a driver, not a bootloader. You can't boot that.

As to the first test, you are getting a completely different error from the ones mentioned above. Interesting...

Also one thing I don't quite understand is that I can select fat32 in Rufus 2.18 but not in 3.3.

Well, UEFI:NTFS is of course only valid for NTFS, so obviously it makes no sense to have it with FAT. I sure hope you didn't use FAT32 for your 2.18 test, because then it makes your success there irrelevant.

@pbatard
Copy link
Owner

pbatard commented Oct 20, 2018

Also, why does the boot option name in your first screen say WinUsb2?
I'm sorry but your tests look super weird.

Care to post a full log of how you created the USB with Rufus 3.3?

@devedse
Copy link

devedse commented Oct 20, 2018

I did use NTFS for both tests. The WinUsb2 is just a name that you can manually enter.

Full process would be:

  1. Download img
  2. Download rufus
  3. Start rufus
  4. Load image in rufus without changing any settings.
  5. Open bios of dell laptop and add uefi boot option.
  6. Browse to uefi file as mentioned above on the usb drive

Same for 2.18 but there it did work.

@pbatard
Copy link
Owner

pbatard commented Oct 20, 2018

Still, I would really like to see the full log of you created the USB with Rufus 3.3.
Remember, the more data I have, the more I am able to help you.

@devedse
Copy link

devedse commented Oct 20, 2018

Will do tomorrow. :) thnx for the help already!

@pbatard
Copy link
Owner

pbatard commented May 9, 2020

Thanks for the testing. Let me first address your first post.

What I don't completely understand now is that we have an EFI\boot folder on both the UEFI disk and the main partition. Is this the thing you want?

Yes, absolutely.

The whole point of UEFI:NTFS is to have a bootloader on a small FAT partition (the one you point to) that loads the NTFS driver and then chain load the actual bootloader on the NTFS partition, so, yes, you definitely want to see a /efi/boot/bootx64.efi on both partitions.

What you ultimately want to do is execute the EFI bootloader that resides on the NTFS partition, so that's why if you don't have a /efi/boot/bootx64.efi, then there's no point (though of course, if you're just checking for failures of the UEFI:NTFS bootloader then you could theoretically test without the NTFS boot loader, but in that case UEFI:NTFS will always fail even when it should work).

In other words the boot process goes like this:

  1. The system (generally) can't boot the NTFS bootloader directly because it can't read NTFS, so it looks for /efi/boot/bootx64 on the small FAT partition (which it can always read) and execute it. This is the UEFI:NTFS bootloader that is provided by Rufus.
  2. The UEFI:NTFS bootloader above attempts to load the missing NTFS driver and, if successful, it hands over the boot process to the /efi/boot/bootx64.efi that resides on the NTFS partition. This usually is the boot loader provided by the Windows ISO (i.e. the Windows installer) or, in our case, since we don't want to have to extract a Windows ISO every time, the /efi/boot/bootx64.efi that I provided in the efi.zip archive above.

What I also tried was simply overwriting the files on the UEFI partition

Please do NOT do that. You should NOT touch the small UEFI_NTFS partition that already contains the /Efi and /Rufus folders. If you do that, then you will effectively erase the UEFI:NTFS bootloader with something that is not UEFI:NTFS (the bootloaders from efi.zip simply display a message, they are not "test" versions of UEFI:NTFS that you should replace the existing UEFI:NTFS with).

As you can see, in both these scenario's I do need to manually add the option to boot using this specific UEFI configuration, because if I don't it simply won't boot.

And now I realize that your boot selection window is not a Microsoft Windows application, but a UEFI menu. So yes, that's what you should use to select the device you want to boot from (though I'm a bit puzzled that Dell forces you to point to a specific file instead of just automatically looking for a boot partition, which is how UEFI is set to proceed... count on Dell to make it a lot less user intuitive than needs to be).

@pbatard
Copy link
Owner

pbatard commented May 9, 2020

I have a question though: What happens if you simply select Stick2 without explicitly clicking Add boot option and picking a file (or leave the File name field empty)? Coz, once again, UEFI is designed to automatically locate a and boot a /efi/boot/bootx64.efi boot loader on any partition that it can read, so the one from the FAT partition should be picked automatically by a UEFI compliant system simply by telling it that you want to boot the USB device (the whole device, not an individual partition on that device), in UEFI mode.

@pbatard
Copy link
Owner

pbatard commented May 9, 2020

Okay, now to comment on your videos:

  • https://youtu.be/iRyV4h4DTRQ
    Yes, that is the correct procedure (PS: Wow, you're uploading 4K videos - nice!! 😄)

  • https://youtu.be/Ra7u8Y5O_LU
    This looks like the gnu-efi vs EDK2 compiled issue that I am after. The NTFS driver that's included in Rufus 3.10 is the EDK2 version, which some folks report is causing trouble, as opposed to the gnu-efi compiled version.

  • https://youtu.be/DRkyNNmGllo
    No, that won't work. Forget about doing this, coz that's not what I want you to do.

  • https://youtu.be/BDeR_SRl9-o
    I really can't see how that can happen unless there was an issue during the drive creation and you ended up with a non UEFI:NTFS drive. Can you please re-test this, and post the complete log from Rufus from how you created the drive. I really can't see how the Dell UEFI firmware would skip the FAT partition, when we're using the exact same as 3.8 and 3.10 to create it.

  • https://youtu.be/bLqQ1WPUx5s
    Again, a Rufus log will help.

  • https://youtu.be/vzKiqIVPDHE
    Yes, this is the expected result, and the one you'd get from a system that has trouble with the EDK2 version of the NTFS file system driver but not with the gnu-efi version.

If possible, I'd be very curious about what happens if you run the UEFI Shell and try to load the v1.5 NTFS driver from there, and whether you'd see the same errors. Let me prepare a procedure to guide you through that.

@pbatard
Copy link
Owner

pbatard commented May 9, 2020

Okay, let's try this test:

  1. Download shell.vhd.gz (no need to extract it once downloaded). This is a disk image with everything you need.
  2. Use it in Rufus to create a drive like you would use any other .iso. Don't worry, Rufus can handle a .gz files. Once done, you will end up with a small drive that contains 2 small partitions, one FAT and one NTFS.
  3. Boot the drive (by pointing to the /efi/boot/bootx64.efi file on the FAT partition if needed
  4. This should open the UEFI Shell. In the Shell environment type:
    • map → This lists all the drives (FS#: that you can access)
    • ver → This provides some information about your UEFI environment.
    • FS#: then dir, with # being replaced by a number from 0 to 9 until you see ntfs-g.efi and ntfs-e.efi being listed from the dir command.
    • load ntfs-e.efi → This loads the EDK2-compiled version of the NTFS file system driver
    • map -r → This lists all the drives and re-maps new one. Don't forget to add the -r option. You should see new FS#: drive(s) being listed from the previous map command, for the NTFS partition. Make sure to remember their numbers.
    • For each new drive that appeared above, type FS#: where # is one of the numbers you picked above then dir until you see a rufus.txt file being listed.
    • Finally type type rufus.txt (yes, type is also a commandline instruction you need to input here) → This should produce the output This is the NTFS volume.
  5. If there is any error with this procedure, then reboot the computer and repeat from step 3. but this time use load ntfs-g.efi to load the gnu-efi version of the driver instead of load ntfs-e.efi and see what happens.

Screenshots from any errors (no need to upload videos) as well as the ouptut of the ver and map commands would be appreciated.

I hope this doesn't sound too much like I'm just ordering you around (feel free to take your time if needed or ask for clarification), because I really appreciate your willingness to help and I am very grateful for the tests you've already performed today!

@devedse
Copy link

devedse commented May 13, 2020

Sorry for the delayed response but I haven't really been able to find time to look into the issue further. I'll let you know once I've executed the steps you requested.

@albertopasqualetto
Copy link

The problem happens also to me, but with 3.9 works.

@devedse
Copy link

devedse commented May 18, 2020

@pbatard , I've found some time to run through the scenario. I've tried to execute step 3:
Boot the drive (by pointing to the /efi/boot/bootx64.efi file on the FAT partition if needed

However if I do this I only manage to see a black screen.

At the end of your post is stated that I should also try to booth with NTFS-G.efi. I've tried that too, but the same black screen appeared again. For NTFS-E.efi the same happened.

What am I doing wrong 😄?

Repository owner deleted a comment from startergo May 18, 2020
Repository owner deleted a comment from startergo May 18, 2020
Repository owner deleted a comment from startergo May 18, 2020
Repository owner deleted a comment from startergo May 18, 2020
Repository owner deleted a comment from startergo May 18, 2020
@devedse
Copy link

devedse commented May 18, 2020

So if you want we can drop on discord and do some tests together if that's easier for you.

@pbatard
Copy link
Owner

pbatard commented May 18, 2020

@devedse, thanks for testing.

However if I do this I only manage to see a black screen.

Interesting. It means that you can't even execute a recent UEFI Shell v2.2 (because that's what the bootx64.efi file on the FAT partition really is), which would tend to indicate that this whole driver incompatibility could indeed have something to do with the suggested theory that the UEFI firmware from these Dell's (and other affected computers) is too old to provide some of the APIs that modern EDK2 relies on...

I need to think about this some more, as well as investigate what breaking changes might have occurred in the EDK2.

if you want we can drop on discord and do some tests together if that's easier for you.

That shouldn't be necessary. Fingers crossed, I'm going to get my hands on a machine that is affected by this issue sometime this week, which should greatly improve my ability to troubleshoot that error. I can always use test reports though, as you can never be too sure that something you're seeing on one machine will be exactly the same on another...

@pbatard
Copy link
Owner

pbatard commented May 22, 2020

DAMMIT!

Finally got that Dell machine (an Optiplex 7010), but it looks like it isn't affected by the issue. I tried to downgrade the BIOS just in case, but UEFI:NTFS still works fine with the EDK2 driver there.

However, testing with that machine is giving me some hints as to where the issue might stem from (but not how to fix it yet).

Especially from @devedse's initial report and from #1536, I can see that the Device Paths reported when the issue occurs are a bit unexpected.

For instance, @devedse reports seeing PciRoot(0x0)/Pci(0x19,0x0)/INT13(,0x81)/HD... for the USB drive @Chatplosion reports seeing PciRoot(0)/Pci(0x1D,0x0)/VenHw(AA7BA38A-DAFB-40C3-8D18-B55B39609EF7)/HD..., whereas on the 7010 I am seeing PciRoot(0)/Pci(0x1D,0x0)/USB(0x1,0x0)/USB(0x2,0x0)/HD..., which is the kind of path I'd expect for a USB device.

Moreover, according to this, VenHw(<GUID>) is used when the device path node is not defined by the UEFI specification such as for generic memory mapped controllers. And INT13 is a BIOS interface that is commonly used by devices to provide custom disk access. Both can be used, for instance, by RAID controller cards, since regular SATA wouldn't work.

It is however very puzzling to see either of the above being used for USB devices when UEFI should have a relatively solid implementation to handle USB devices...

Now, the NTFS driver itself should be entirely agnostic to the type of controller being used (the proof of that being that the gnu-efi compiled version of the same code works), but I'm starting to wonder if the EDK2 may not be doing some things behind the scenes where the type of controller being used to access USB devices does matter...

Anyway, while I am trying to source yet another Dell machine that might be able to replicate this issue, I have the following to ask:

  1. Can any of the people affected by this issue tell me if they added extra controller cards to their computer? Especially, did you add a USB 3.0 add-in card?
  2. Could you please run a test with the following image (just open it in Rufus and press START) and report the ouput: test1.vhd.gz

Thanks

@Aftabnack
Copy link

This thread helped me resolve my issue with doing this on ubuntu.
I have documented it in detail here: https://askubuntu.com/a/1243565/1087068

@pbatard
Copy link
Owner

pbatard commented May 25, 2020

A-HA!

I finally got my hand on a problematic machine (Dell Optiplex 390), and now everything is crystal clear, as per pbatard/EfiFs#21.

The main cause of the problem is that some old UEFI firmwares don't provide a protocol that allows to convert a Device Path (a Device Path is a binary blob that UEFI uses to uniquely indentify a device such as a USB drive or a USB partition) to a Device Path string, which is what I decided to use as a unique identifier for partitions in EfiFs because that's what grub_device_open() takes as a parameter (and I am reusing the GRUB file system drivers).

In other words, it made a lot of sense to use a DevicePath string at the time, especially as, regardless of whether the UEFI firmware did provide a protocol to convert a Device Path to a string, I could always fall back to a function, DevicePathToStr(), that the gnu-efi library provides, that also does the same thing.

But then I added support for EDK2 compilation, skirted around the absence of DevicePathToStr() there, and, since all my testing seemed to work fine, because they were conducted on machines that had the Device Path to string conversion protocol, completely forgot that, unlike gnu-efi, EDK2 did not have a fallback in case the protocol was absent.

So, what happens on the problematic machines is that, when we try to bind the NTFS driver to a partition, the string conversion for the Device Path of that partition fails, and the driver says it can't open that partition (with error [3] Unsupported because the protocol we need is not supported on that platform). And that's the reason why UEFI:NTFS fails with the EDK2 version of the NTFS driver, but works with the gnu-efi version of the same driver, because in case of gnu-efi, if the protocol isn't there, we can still get the string we need by calling the gnu-efi specific function DevicePathToStr()...

I have to say, I am a bit ashamed that I didn't spot this earlier, because, of course, it makes total sense now that it's clear what happened, and I feel like I should have been able to figure this out, without having to go through the trouble of purchasing a couple of these old Dell systems.

Once I have figured out a good way to work around the issue of not being able to rely on Device Path string conversions in EfiFs, I will produce a v1.6 release of the EfiFs drivers and hopefully, this time, that should be the end of this obnoxious issue.

@devedse
Copy link

devedse commented May 25, 2020

Amazing that you figured it out :), keep us posted!

@pbatard
Copy link
Owner

pbatard commented May 26, 2020

Okay, I have now updated the UEFI:NTFS image in Rufus and I will close this issue, as I expect it to be fixed for good.

For those who want to test, I have also uploaded an early TEST version of Rufus 3.11 with the updated UEFI:NTFS here.

@startergo
Copy link

Okay, I have now updated the UEFI:NTFS image in Rufus and I will close this issue, as I expect it to be fixed for good.

For those who want to test, I have also uploaded an early TEST version of Rufus 3.11 with the updated UEFI:NTFS here.

do you have the UEFI:NTFS as an img? I want to dd write the img

@startergo
Copy link

I just tried the NTFS.efi driver in OpenCore, which is an EDKII build. At boot in the EDK environment upon loading the driver I get:

OC: Driver NTFS.efi at 2 is being loaded...
OC: Driver NTFS.efi at 2 cannot be started - Incompatible Version!

@pbatard
Copy link
Owner

pbatard commented May 30, 2020

do you have the UEFI:NTFS as an img? I want to dd write the img

Since the software is 100% Open Source, the latest image can always be downloaded from the GitHub repo here

upon loading the driver I get:

You are most likely not using the proper driver from your arch.

I'm afraid this is NOT the right thread to ask for help about using the driver. This thread is about getting errors [1] or [14] or [3] when using the UEFI:NTFS version installed by Rufus. If you have unrelated questions, please create a thread either in the UEFI:NTFS project or the EfiFs project. Otherwise, I will have no choice but to lock this issue to prevent further unrelated issues.

@DoSpamu
Copy link

DoSpamu commented Jun 14, 2020

Thank you for this fix in the 3.11 ver.
I haved the same problem on Dell and now all works well.

dyeske pushed a commit to dyeske/rufus that referenced this issue Jul 3, 2020
dyeske pushed a commit to dyeske/rufus that referenced this issue Jul 3, 2020
* Now use version 1.6 of the EfiFs drivers that enables firmwares that
  don't provide EFI_DEVICE_PATH_TO_TEXT_PROTOCOL to mount NTFS partitions
  regardless.
* Also use the latest version of UEFI:NTFS that displays additional info
  about the system.
* Closes pbatard#1213.
* Also update version to Rufus-next.
@github-actions
Copy link

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue if you think you have a related problem or query.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 28, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants