Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read error: [0x0000001B] The drive cannot find the sector requested when writing VHD image #2468

Closed
AMIGASYSTEM opened this issue Apr 29, 2024 · 28 comments

Comments

@AMIGASYSTEM
Copy link

AMIGASYSTEM commented Apr 29, 2024

Hi Carlo (70 Amiga User) I am also the developer of AROS One x86, some years I had asked you if Rufus could write an ISO of AROS x86.

Later I solved it by installing AROS x86 on Pendrive, and creating a USB Image from the Pendrive.

Now I have encountered a problem with Rufus 4.x, basically the USB Images (Pendrive) created with Rufus 3.x are no longer compatible with Rufus 4.x

Which version do you recommend I use in the future?

If you want to do some testing, you can find the AROS USB image (created with Rufus 3.x) on my link:

https://sites.google.com/view/arosone

Then test the Live Pendriver created on a supported PC, on older PCs it should boot.

@pbatard
Copy link
Owner

pbatard commented Apr 29, 2024

Hi Carlo,

Can you elaborate on your "are no longer compatible with Rufus 4.x" statement? What error do you see, and where does it happen?

I am not seeing any issue writing the AROS One Image-USB-2.4.zip with Rufus 4.4, and since it's a VHD image, whatever happens after Rufus wrote the image should have to do with the content of the image itself and not with Rufus.

In short, as much as I'd like to help, I'm afraid you're not providing enough details with regards to the exact nature of the issue you see, so more details would be welcome...

@AMIGASYSTEM
Copy link
Author

AMIGASYSTEM commented Apr 29, 2024 via email

@pbatard
Copy link
Owner

pbatard commented Apr 30, 2024

Please go to #2468 and attach the screenshot there, as screenshots sent through e-mail replies are not attached, and nobody but you can see them.

Also, as requested in the checklist you were asked to follow when opening the issue, it is MUCH BETTER, if you want your issue to be properly investigated, to attach the log from Rufus (the checklist tells you explicitly how to access it), as screenshots are next to worthless when I cannot replicate an issue, as is the case here.

For instance, here's my log writing AROS One Image-USB-2.4.zip with Rufus 4.4 on Windows 11:

Rufus x64 v4.4.2103
Windows version: Windows 11 Pro for Workstations x64 (Build 22631.3527)
Syslinux versions: 4.07/2013-07-25, 6.04/pre1
Grub versions: 0.4.6a, 2.06
System locale ID: 0x0809 (en-GB)
Will use default UI locale 0x0809
SetLGP: Successfully set NoDriveTypeAutorun policy to 0x0000009E
Localization set to 'en-US'
Found 517 officially revoked UEFI bootloaders from embedded list
Found 2351 additional revoked UEFI bootloaders from this system's SKUSiPolicy.p7b
Found USB device 'Mushkin Ventura Ultra USB Device' (174C:55AA) [ID]
1 device found
Disk type: FIXED, Disk size: 120 GB, Sector size: 512 bytes
Cylinders: 14593, Tracks per cylinder: 255, Sectors per track: 63
Partition type: GPT, NB Partitions: 1
Disk GUID: {5E972E5D-6AA2-4108-88C5-4DDAF3F3313A}
Max parts: 128, Start Offset: 17408, Usable = 120034089472 bytes
Partition 1:
  Type: EFI System Partition
  Name: 'Blah'
  Detected File System: FAT32
  ID: {CDFBD781-63DF-4C5C-9496-6325E9A364FA}
  Size: 8.0 GB (8587951616 bytes)
  Start Sector: 2048, Attributes: 0xD000000000000005
Scanning image...
ISO analysis:
  'D:\Imgs\AROS One Image-USB-2.4.zip' doesn't look like an ISO image
Disk image analysis:
  Image is a compressed bootable disk image
Using image: AROS One Image-USB-2.4.zip (1.9 GB)
Timeout while retrieving conflicting process list

Format operation started
Requesting disk access...
No drive letter was assigned...
Will use 'F:' as volume mountpoint
Opened \\.\PhysicalDrive9 for exclusive write access
Requesting logical volume handle...
No logical drive found (unpartitioned?)
Writing compressed image:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Opened \\.\PhysicalDrive9 for shared write access
Timeout while waiting for logical drive

Found USB device 'Mushkin Ventura Ultra USB Device' (174C:55AA) [ID]
No logical drive found (unpartitioned?)
1 device found
No volume information for drive 0x89
Disk type: FIXED, Disk size: 120 GB, Sector size: 512 bytes
Cylinders: 14593, Tracks per cylinder: 255, Sectors per track: 63
Partition type: MBR, NB Partitions: 2
Disk ID: 0x000A86CB
Drive has an unknown Master Boot Record
Partition 1:
  Type: Extended (0x05)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011393024 bytes)
  Start Sector: 192, Boot: Yes
Partition 5:
  Type: Unknown (0x30)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011196416 bytes)
  Start Sector: 576, Boot: No

For the record, Timeout while waiting for logical drive is expected for file systems that are unrecognized, and it is not an error. And as you can see, I do not get any read-error.

Now, my guess is that the issue you may be facing probably has to do with the fact that you are using a VHD image, which is NOT a flat disk image, as it contains an additional 512 bytes of data, which you are trying to restore to a drive of the exact same size as the one you used to create the VHD. If you do that, then, these extra 512 bytes should produce an error because you are effectively using a VHD in lieu of a compressed flat disk image, and this is not something Rufus can support (the fact that it might have somehow worked before is no guarantee that it should ever have been supported), and you need to decide as to whether you want to use a proper compressed flat disk image, in which case it should just be that (i.e. not a VHD with the extract 512 bytes, but the part before that) in a zip file, or a proper compressed Microsoft VHD image, a.k.a. a VHDX. You simply cannot have it both way, because Rufus was only ever designed to support flat disk images inside of compressed archive, and any support of VHD inside of an archive would have been an accident that has been rectified, and that I have no plan to "revert".

In short: Do not zip uncompressed VHD images and expect software to be able to handle those. Either compress a pure .img or use a .vhdx.

@AMIGASYSTEM
Copy link
Author

AMIGASYSTEM commented Apr 30, 2024

Thanks for the advice Pete, unfortunately I cannot use the VHDX format, if I try to create a Pendrive Image I get the same error you see in the screenshot attached below "Read Error" attach log

Rufus x64 v4.4.2103
Windows version: Windows 10 Pro x64 (Build 19045.4046)
Syslinux versions: 4.07/2013-07-25, 6.04/pre1
Grub versions: 0.4.6a, 2.06
System locale ID: 0x0410 (it-IT)
Will use default UI locale 0x0410
SetLGP: Successfully set NoDriveTypeAutorun policy to 0x0000009E
Localization set to 'it-IT'
Found 517 officially revoked UEFI bootloaders from embedded list
Found 2351 additional revoked UEFI bootloaders from this system's SKUSiPolicy.p7b
Found USB 2.0 device 'General USB Flash Disk USB Device' (8633:800A)
No logical drive found (unpartitioned?)
1 device found
No volume information for drive 0x81
Disk type: FIXED, Disk size: 4GB, Sector size: 512 bytes
Cylinders: 487, Tracks per cylinder: 255, Sectors per track: 63
Partition type: MBR, NB Partitions: 2
Disk ID: 0x000A86CB
Drive has an unknown Master Boot Record
Partition 1:
  Type: Extended (0x05)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011393024 bytes)
  Start Sector: 192, Boot: Yes
Partition 5:
  Type: Unknown (0x30)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011196416 bytes)
  Start Sector: 576, Boot: No
Scanning image...
ISO analysis:
  'C:\Users\Win10\Desktop\AROS-One.vhdx' doesn't look like an ISO image
Disk image analysis:
  Image is a bootable VHD image
Using image: AROS-One.vhdx (3.8 GB)
Found USB 2.0 device 'General USB Flash Disk USB Device' (8633:800A)
No logical drive found (unpartitioned?)
1 device found
No volume information for drive 0x81
Disk type: FIXED, Disk size: 4GB, Sector size: 512 bytes
Cylinders: 487, Tracks per cylinder: 255, Sectors per track: 63
Partition type: MBR, NB Partitions: 2
Disk ID: 0x000A86CB
Drive has an unknown Master Boot Record
Partition 1:
  Type: Extended (0x05)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011393024 bytes)
  Start Sector: 192, Boot: Yes
Partition 5:
  Type: Unknown (0x30)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011196416 bytes)
  Start Sector: 576, Boot: No

Format operation started
Requesting disk access...
No drive letter was assigned...
Will use 'E:' as volume mountpoint
Opened \\.\PhysicalDrive1 for exclusive write access
Requesting logical volume handle...
No logical drive found (unpartitioned?)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Read error: [0x0000001B] (NB: This system was unable to provide an English error message)
Opened \\.\PhysicalDrive1 for shared write access

Found USB 2.0 device 'General USB Flash Disk USB Device' (8633:800A)
No logical drive found (unpartitioned?)
1 device found
No volume information for drive 0x81
Disk type: FIXED, Disk size: 4GB, Sector size: 512 bytes
Cylinders: 487, Tracks per cylinder: 255, Sectors per track: 63
Partition type: MBR, NB Partitions: 2
Disk ID: 0x000A86CB
Drive has an unknown Master Boot Record
Partition 1:
  Type: Extended (0x05)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011393024 bytes)
  Start Sector: 192, Boot: Yes
Partition 5:
  Type: Unknown (0x30)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011196416 bytes)
  Start Sector: 576, Boot: No

Strange though that with Rufus 3.2 I never had any problem with all the USB VHD images created so far, I have created many AROS One Images (Many versions) and all compacted in ZIP and never had any problem.

Regarding the strange screenshot that on your E-Mail does not show the screenshots, I on my E-mails never had any problem to display the screenshots attached to my E-mail !

Anyway I attach link with the screenshot showing the error

https://i.ibb.co/rm8L4c9/Rufus-Error.jpg

@pbatard
Copy link
Owner

pbatard commented Apr 30, 2024

VHDX is a different issue that will be fixed in the next release. Please show me the log from when you use the official AROS One Image-USB-2.4.zip you told me had a problem. VHDX could not be created with Rufus 3.x (this is a very new feature which is why I am still fixing bugs with it), and your original issue was, I quote:

USB Images (Pendrive) created with Rufus 3.x are no longer compatible with Rufus 4.x

(which by the way is inaccurate since Rufus does not create zipped VHD, so if you do zip a .vhd created by Rufus, you are not dealing with an image that was created by Rufus).

So can I please have the log of what Rufus reports when you write AROS One Image-USB-2.4.zip?

As to e-mail attachments, please bear in mind that I do not receive your e-mail replies directly. Instead, GitHub receives them, processes the content (and strips any attachment), generates a GitHub issue update, and once that issue update has been added, it generates an e-mail notification from the update, and not from your original e-mail.

At any rate, screenshots when reporting issue are next to worthless. Please consider them the same as a picture of a body taken by the police at a crime scene. It'll just confirm that somebody is dead, and maybe give some hints, but it's pretty much never enough to tell you exactly what happened. On the other hand, the log is like CCTV footage of the crime scene when the murder happened, and it is invaluable of telling you the sequence of event of how the crime happened, as well as who committed it.

Which is why, when an application does include logging, and the developer asks you for a log, you shouldn't ignore their request and think a mere screenshot will be good enough...

@AMIGASYSTEM
Copy link
Author

Ok thanks I will no longer ZIP compress the VHD images, I was doing it to speed up the Download

I attach Log Error AROS One Image-USB-2.4.zipper with Rufus 4

Rufus x64 v4.4.2103
Windows version: Windows 10 Pro x64 (Build 19045.4046)
Syslinux versions: 4.07/2013-07-25, 6.04/pre1
Grub versions: 0.4.6a, 2.06
System locale ID: 0x0410 (it-IT)
Will use default UI locale 0x0410
SetLGP: Successfully set NoDriveTypeAutorun policy to 0x0000009E
Localization set to 'it-IT'
Found 517 officially revoked UEFI bootloaders from embedded list
Found 2351 additional revoked UEFI bootloaders from this system's SKUSiPolicy.p7b
Found USB 2.0 device 'SanDisk Cruzer Blade USB Device' (0781:5567)
1 device found
No volume information for drive 0x81
Disk type: Removable, Disk size: 8 GB, Sector size: 512 bytes
Cylinders: 973, Tracks per cylinder: 255, Sectors per track: 63
Partition type: MBR, NB Partitions: 2
Disk ID: 0x000A86CB
Drive has an unknown Master Boot Record
Partition 1:
  Type: Extended (0x05)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011393024 bytes)
  Start Sector: 192, Boot: Yes
Partition 5:
  Type: Unknown (0x30)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011196416 bytes)
  Start Sector: 576, Boot: No
Scanning image...
ISO analysis:
  'C:\Users\Win10\Desktop\AROS One Image-USB-2.3.vhd' doesn't look like an ISO image
Disk image analysis:
  Image is a bootable VHD image
Using image: AROS One Image-USB-2.3.vhd (3.7 GB)
Found USB 2.0 device 'SanDisk Cruzer Blade USB Device' (0781:5567)
1 device found
No volume information for drive 0x81
Disk type: Removable, Disk size: 8 GB, Sector size: 512 bytes
Cylinders: 973, Tracks per cylinder: 255, Sectors per track: 63
Partition type: MBR, NB Partitions: 2
Disk ID: 0x000A86CB
Drive has an unknown Master Boot Record
Partition 1:
  Type: Extended (0x05)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011393024 bytes)
  Start Sector: 192, Boot: Yes
Partition 5:
  Type: Unknown (0x30)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011196416 bytes)
  Start Sector: 576, Boot: No

Format operation started
Requesting disk access...
Will use 'E:' as volume mountpoint
Opened \\.\PhysicalDrive1 for exclusive write access
Requesting logical volume handle...
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Read error: [0x0000001B] (NB: This system was unable to provide an English error message)
Opened \\.\PhysicalDrive1 for shared write access
Re-mounted volume as E: after error

Found USB 2.0 device 'SanDisk Cruzer Blade USB Device' (0781:5567)
1 device found
No volume information for drive 0x81
Disk type: Removable, Disk size: 8 GB, Sector size: 512 bytes
Cylinders: 973, Tracks per cylinder: 255, Sectors per track: 63
Partition type: MBR, NB Partitions: 2
Disk ID: 0x000A86CB
Drive has an unknown Master Boot Record
Partition 1:
  Type: Extended (0x05)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011393024 bytes)
  Start Sector: 192, Boot: Yes
Partition 5:
  Type: Unknown (0x30)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011196416 bytes)
  Start Sector: 576, Boot: No
Found USB 2.0 device 'SanDisk Cruzer Blade USB Device' (0781:5567)
1 device found
No volume information for drive 0x81
Disk type: Removable, Disk size: 8 GB, Sector size: 512 bytes
Cylinders: 973, Tracks per cylinder: 255, Sectors per track: 63
Partition type: MBR, NB Partitions: 2
Disk ID: 0x000A86CB
Drive has an unknown Master Boot Record
Partition 1:
  Type: Extended (0x05)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011393024 bytes)
  Start Sector: 192, Boot: Yes
Partition 5:
  Type: Unknown (0x30)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011196416 bytes)
  Start Sector: 576, Boot: No
Closed Windows format prompt

@pbatard pbatard changed the title Rufus 3.x Vs Rufus 4.x Read error: [0x0000001B] The drive cannot find the sector requested when writing VHD image May 1, 2024
@pbatard
Copy link
Owner

pbatard commented May 1, 2024

Thanks. It looks like I am finally able to replicate the error (not sure why I didn't get it earlier on).

Let me investigate this so I can figure out where it comes from.

@pbatard
Copy link
Owner

pbatard commented May 1, 2024

This will be fixed in the next version. This is actually a benign error and the resulting drive contains all the data.

Ironically, I'm pretty sure you only got this error because you worked with the uncompressed VHD and not the zipped version, and I sure wish I had had your log right from the start so that I could see that you were opening the .vhd and not the .zip.

At any rate, thanks for the report, but next time, please do follow the check list, as it's only there to help you get your problem sorted in the most timely manner, by avoiding a lot of unnecessary back and forth.

@pbatard pbatard closed this as completed in 39a5ae6 May 1, 2024
@AMIGASYSTEM
Copy link
Author

AMIGASYSTEM commented May 11, 2024 via email

@pbatard
Copy link
Owner

pbatard commented May 11, 2024

if you need it I'll take a screenshot.

Can you please send me your log instead? Again, I very much need to see your full log whenever you think you encounter an issue, as a screenshot is next to useless, as it tells me practically nothing outside of "Yup, you are seeing an error", which I don't doubt, but which I will have a very hard time to try to replicate and fix unless you provide a full log...

@AMIGASYSTEM
Copy link
Author

AMIGASYSTEM commented May 11, 2024 via email

@pbatard
Copy link
Owner

pbatard commented May 11, 2024

As I explained earlier, sending log as attachments will not work, because you are sending them to a GitHub bot which will discard them.

You need to log on to GitHub and attach them to the issue using the web interface if you want them to be shared.

At any rate, I think I have found the issue, and I will fix it, but I wouldn't mind to see the logs to make sure that I am addressing the same thing, rather than have you come back after the Rufus 4.5 release to tell me you're still seeing a problem.

@AMIGASYSTEM
Copy link
Author

AMIGASYSTEM commented May 11, 2024

I attach the Logs here, fifth Log added

Rufus 3.20p Burn Image VHD create con Rufus 4.5 (Ok This Live Pendrive works fine)

Logs.zip

@pbatard
Copy link
Owner

pbatard commented May 12, 2024

Thanks. I believe the fix I am going to push, and which will be part of the 4.5 release, should address the issue.

pbatard added a commit that referenced this issue May 12, 2024
* Addresses the error reported in #2468.
* Also use memmove instead of memcpy where overlapping data is involved.
@AMIGASYSTEM
Copy link
Author

AMIGASYSTEM commented May 30, 2024

I tested the new version of Rufus v4.5 and found this:

Image creation VHD and VHDX, successful
Burned VHD and VHDX image, successful

Booting from PC the "PenDrive Live" Burned with both VHD and VHDX Image, after displaying the GRUB Menu, the AROS partition is not found, "Read error -6 on block 0".

Tried to burn the same VHD Image created with Rufus v4.5, with Rufus 3.20p and AROS One booted regularly, no error.

I could not test the VHDX Image because it is not supported by Rufus 3.20p.

I enclose Log Burning VHD Image with Rufus 4.5

Rufus x64 v4.5.2180
Windows version: Windows 10 Pro x64 (Build 19045.4046)
Syslinux versions: 4.07/2013-07-25, 6.04/pre1
Grub versions: 0.4.6a, 2.12
System locale ID: 0x0410 (it-IT)
Will use default UI locale 0x0410
SetLGP: Successfully set NoDriveTypeAutorun policy to 0x0000009E
Localization set to 'it-IT'
Found 517 officially revoked UEFI bootloaders from embedded list
Found 2351 additional revoked UEFI bootloaders from this system's SKUSiPolicy.p7b
Found USB 2.0 device 'General USB Flash Disk USB Device' (8633:800A)
No logical drive found (unpartitioned?)
1 device found
No volume information for drive 0x81
Disk type: FIXED, Disk size: 4GB, Sector size: 512 bytes
Cylinders: 487, Tracks per cylinder: 255, Sectors per track: 63
Partition type: MBR, NB Partitions: 2
Disk ID: 0x000A86CB
Drive has an unknown Master Boot Record
Partition 1:
  Type: Extended (0x05)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011393024 bytes)
  Start Sector: 192, Boot: Yes
Partition 5:
  Type: Unknown (0x30)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011196416 bytes)
  Start Sector: 576, Boot: No
Scanning image...
ISO analysis:
  'C:\Users\Win10\Desktop\AOSO One 2.5.vhd' doesn't look like an ISO image
Disk image analysis:
  Image is a bootable VHD image
Using image: AOSO One 2.5.vhd (3.7 GB)
Found USB 2.0 device 'General USB Flash Disk USB Device' (8633:800A)
No logical drive found (unpartitioned?)
1 device found
No volume information for drive 0x81
Disk type: FIXED, Disk size: 4GB, Sector size: 512 bytes
Cylinders: 487, Tracks per cylinder: 255, Sectors per track: 63
Partition type: MBR, NB Partitions: 2
Disk ID: 0x000A86CB
Drive has an unknown Master Boot Record
Partition 1:
  Type: Extended (0x05)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011393024 bytes)
  Start Sector: 192, Boot: Yes
Partition 5:
  Type: Unknown (0x30)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011196416 bytes)
  Start Sector: 576, Boot: No

Format operation started
Requesting disk access...
No drive letter was assigned...
Will use 'E:' as volume mountpoint
Opened \\.\PhysicalDrive1 for exclusive write access
Requesting logical volume handle...
No logical drive found (unpartitioned?)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Opened \\.\PhysicalDrive1 for shared write access
Timeout while waiting for logical drive

Found USB 2.0 device 'General USB Flash Disk USB Device' (8633:800A)
No logical drive found (unpartitioned?)
1 device found
No volume information for drive 0x81
Disk type: FIXED, Disk size: 4GB, Sector size: 512 bytes
Cylinders: 487, Tracks per cylinder: 255, Sectors per track: 63
Partition type: MBR, NB Partitions: 2
Disk ID: 0x000A86CB
Drive has an unknown Master Boot Record
Partition 1:
  Type: Extended (0x05)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011393024 bytes)
  Start Sector: 192, Boot: Yes
Partition 5:
  Type: Unknown (0x30)
  Detected File System: (Unrecognized)
  Size: 3.7 GB (4011196416 bytes)
  Start Sector: 576, Boot: No

@pbatard
Copy link
Owner

pbatard commented May 31, 2024

root@nas:~# dd if=/dev/zero of=/dev/sdj bs=512 count=34
34+0 records in
34+0 records out
17408 bytes (17 kB, 17 KiB) copied, 0.00256986 s, 6.8 MB/s
root@nas:~# dd if=/dev/zero of=/dev/sdj bs=512 count=34 seek=$((`blockdev --getsz /dev/sdj` - 34))
34+0 records in
34+0 records out
17408 bytes (17 kB, 17 KiB) copied, 0.00745663 s, 2.3 MB/s
root@nas:~# gdisk /dev/sdj
GPT fdisk (gdisk) version 1.0.9

Partition table scan:
  MBR: not present
  BSD: not present
  APM: not present
  GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-31277198, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-31277198, default = 31277055) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): Y
OK; writing new GUID partition table (GPT) to /dev/sdj.
The operation has completed successfully.
root@nas:~# dd if=/dev/urandom of=/dev/sdj1 bs=4M status=progress
15992881152 bytes (16 GB, 15 GiB) copied, 257 s, 62.2 MB/s
dd: error writing '/dev/sdj1': No space left on device
3818+0 records in
3817+0 records out
16012804096 bytes (16 GB, 15 GiB) copied, 265.859 s, 60.2 MB/s
root@nas:~# sync
root@nas:~# dd if=/dev/sdj of=original_disk.img bs=4M status=progress
15275655168 bytes (15 GB, 14 GiB) copied, 88 s, 174 MB/s
dd: error writing 'original_disk.img': No space left on device
3662+0 records in
3661+0 records out
15356379136 bytes (15 GB, 14 GiB) copied, 88.5318 s, 173 MB/s

Then from Rufus:

[Fri May 31 10:21:35 2024]
Rufus x64 v4.5.2180
Windows version: Windows 11 Pro for Workstations x64 (Build 22631.3672)
Syslinux versions: 4.07/2013-07-25, 6.04/pre1
Grub versions: 0.4.6a, 2.12
System locale ID: 0x0809 (en-GB)
Will use default UI locale 0x0809
SetLGP: Successfully set NoDriveTypeAutorun policy to 0x0000009E
Localization set to 'en-US'
Found 517 officially revoked UEFI bootloaders from embedded list
Found 2351 additional revoked UEFI bootloaders from this system's SKUSiPolicy.p7b
Found USB device 'SanDisk Extreme USB Device' (0781:5580) [ID]
No logical drive found (unpartitioned?)
1 device found
No volume information for drive 0x87
Disk type: Removable, Disk size: 16 GB, Sector size: 512 bytes
Cylinders: 1946, Tracks per cylinder: 255, Sectors per track: 63
Partition type: GPT, NB Partitions: 1
Disk GUID: {27C43B22-6FA7-4E90-90D0-317711AE56C2}
Max parts: 128, Start Offset: 17408, Usable = 16013908480 bytes
Partition 1:
  Type: Linux Data Partition
  Name: 'Linux filesystem'
  Detected File System: (Unrecognized)
  ID: {6DB072A5-7CE6-4A45-90B4-791A44FC6529}
  Size: 14.9 GB (16012804096 bytes)
  Start Sector: 2048, Attributes: 0x0000000000000000

Save to VHD operation started
Operation complete.

Found USB device 'SanDisk Extreme USB Device' (0781:5580) [ID]
No logical drive found (unpartitioned?)
1 device found
No volume information for drive 0x87
Disk type: Removable, Disk size: 16 GB, Sector size: 512 bytes
Cylinders: 1946, Tracks per cylinder: 255, Sectors per track: 63
Partition type: GPT, NB Partitions: 1
Disk GUID: {27C43B22-6FA7-4E90-90D0-317711AE56C2}
Max parts: 128, Start Offset: 17408, Usable = 16013908480 bytes
Partition 1:
  Type: Linux Data Partition
  Name: 'Linux filesystem'
  Detected File System: (Unrecognized)
  ID: {6DB072A5-7CE6-4A45-90B4-791A44FC6529}
  Size: 14.9 GB (16012804096 bytes)
  Start Sector: 2048, Attributes: 0x0000000000000000

Format operation started
Requesting disk access...
No drive letter was assigned...
Will use 'F:' as volume mountpoint
Opened \\.\PhysicalDrive7 for exclusive write access
Zeroing drive:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Found USB device 'SanDisk Extreme USB Device' (0781:5580) [ID]
1 device found
No volume information for drive 0x87
Disk type: Removable, Disk size: 16 GB, Sector size: 512 bytes
Cylinders: 1946, Tracks per cylinder: 255, Sectors per track: 63
Partition type: SFD (Super Floppy Disk) or unpartitioned
Scanning image...
ISO analysis:
  'C:\tmp\saved_by_rufus.vhd' doesn't look like an ISO image
Disk image analysis:
  Image is a bootable VHD image
Using image: saved_by_rufus.vhd (14.9 GB)

Format operation started
Requesting disk access...
Will use 'F:' as volume mountpoint
Opened \\.\PhysicalDrive7 for exclusive write access
Requesting logical volume handle...
No logical drive found (unpartitioned?)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Opened \\.\PhysicalDrive7 for shared write access
Timeout while waiting for logical drive

Found USB device 'SanDisk Extreme USB Device' (0781:5580) [ID]
No logical drive found (unpartitioned?)
1 device found
No volume information for drive 0x87
Disk type: Removable, Disk size: 16 GB, Sector size: 512 bytes
Cylinders: 1946, Tracks per cylinder: 255, Sectors per track: 63
Partition type: GPT, NB Partitions: 1
Disk GUID: {27C43B22-6FA7-4E90-90D0-317711AE56C2}
Max parts: 128, Start Offset: 17408, Usable = 16013908480 bytes
Partition 1:
  Type: Linux Data Partition
  Name: 'Linux filesystem'
  Detected File System: (Unrecognized)
  ID: {6DB072A5-7CE6-4A45-90B4-791A44FC6529}
  Size: 14.9 GB (16012804096 bytes)
  Start Sector: 2048, Attributes: 0x0000000000000000

Then back on Linux with the drive created by Rufus plugged back in (and the VHD saved by Rufus also copied into /mnt/ssd):

root@nas:~# cmp original_disk.img /mnt/ssd/saved_by_rufus.vhd
cmp: EOF on original_disk.img after byte 15356379136, in line 59982753
root@nas:~# cmp original_disk.img /dev/sdj
cmp: EOF on original_disk.img after byte 15356379136, in line 59982753
root@nas:~#

What the above demonstrates is that, no, Rufus 4.5 does not alter the data it saves to VHD from the original source disk data, or alter the data it writes to a target disk from the original VHD data and therefore that any differences you see come either from your environment, or from Windows itself interfering with the data on your drive because it might have decided that it needs to be "fixed". And you will be mindful that even though we wrote the same data as was previously on the drive during our test, we did zero the drive in between

I don't have access to your environment, or your source images or your hardware and I'm afraid that, since we have conclusively demonstrated that, for something that Windows does not know how to access, not a single byte is being modified when using Rufus to create or write VHDs, then if you observe an issue, you will have to investigate it on your own and/or report it to Microsoft if it turns out that, when using the Microsoft native APIs to read to/write from VHD (Rufus 4.5), content is being altered compared to using direct sector read/writes (Rufus 3.20).

There's only so much I can assume with the manner in which you conducted your tests and again, formal testing demonstrates that Rufus 4.5 does not alter VHD data (which it couldn't do since it's using the Windows APIs and is no longer manipulating sectors, so the only alteration that could occur would be to tell the API to stop writing data early, which is the issue that was happening earlier, but that got fixed). So you will have to explore exactly how your data is being altered, if it really is being altered, and where, by performing tests similar to the ones I laid out above, as it should give a clue as to what in your environment might be causing the alteration. And please also make sure to test your devices for bad blocks, as we can't rule out a defective device.

@AMIGASYSTEM
Copy link
Author

AMIGASYSTEM commented May 31, 2024 via email

@pbatard
Copy link
Owner

pbatard commented May 31, 2024

I'm sorry but I just don't have the time for this kind of testing. I have demonstrated that Rufus does write VHD data as expected, and therefore, that whatever happens must be external to Rufus as I cannot vouch for what Windows does with the drive post write (or even during write when using the Windows VHD writing APIs, if Windows interprets your specific data in a manner that it doesn't like).

If you have VHD images to share, I may take a look, but I am certainly not going to invest any time trying to create my own in the hope that I can replicate your issue as, again, there are way too many parameters, that are specific to your environment, that are outside my control and that can play.

There is only so far I will go to test VHD writing, when I have already demonstrated that VHD writing should and does work exactly as advertised with standard random data.

@AMIGASYSTEM
Copy link
Author

AMIGASYSTEM commented May 31, 2024 via email

@pineapple63
Copy link

pineapple63 commented May 31, 2024

It seems there might be something wrong with that image (either that or there is either a bug in Rufus or something is wonky with the system i was using)

(Turns out the download was corrupting, likely due to a download manager)

Log from Rufus:

Rufus x64 v4.5.2180
Windows version: Windows 11 Pro x64 (Build 22631.3672)
Syslinux versions: 4.07/2013-07-25, 6.04/pre1
Grub versions: 0.4.6a, 2.12
System locale ID: 0x0809 (en-GB)
Will use default UI locale 0x0809
SetLGP: Successfully set NoDriveTypeAutorun policy to 0x0000009E
Localization set to 'en-US'
Found 517 officially revoked UEFI bootloaders from embedded list
Found 2351 additional revoked UEFI bootloaders from this system's SKUSiPolicy.p7b
Found USB 3.0 device 'SanDisk Ultra USB 3.0 USB Device' (0781:5591)
Found USB 3.0 device 'SanDisk Ultra USB Device' (0781:5581)
2 devices found
Disk type: Removable, Disk size: 16 GB, Sector size: 512 bytes
Cylinders: 1869, Tracks per cylinder: 255, Sectors per track: 63
Partition type: MBR, NB Partitions: 1
Disk ID: 0x00000000
Drive has a Zeroed Master Boot Record
Partition 1:
  Type: FAT32 LBA (0x0c)
  Detected File System: FAT32
  Size: 14.3 GB (15375237120 bytes)
  Start Sector: 2048, Boot: No
Scanning image...
ISO analysis:
  'C:\Users\aidan\ArosOne-x86-v2.5-USB-IMG.vhd' doesn't look like an ISO image
Disk image analysis:
Could not open image 'C:\Users\aidan\ArosOne-x86-v2.5-USB-IMG.vhd': [0xC03A001A] The specified disk or diskette cannot be accessed.
Disk type: Removable, Disk size: 16 GB, Sector size: 512 bytes
Cylinders: 1869, Tracks per cylinder: 255, Sectors per track: 63
Partition type: MBR, NB Partitions: 1
Disk ID: 0x00000000
Drive has a Zeroed Master Boot Record
Partition 1:
  Type: FAT32 LBA (0x0c)
  Detected File System: FAT32
  Size: 14.3 GB (15375237120 bytes)
  Start Sector: 2048, Boot: No
Failed to scan image

@pbatard
Copy link
Owner

pbatard commented May 31, 2024

Nah, I can open and write it alright, and from what I can see, somehow the last 512 bytes of data from the image are not written as expected (whereas the rest of the image, up to the last 512 bytes matches 1:1). Note that I am talking about the last 512 bytes before the extra 512 bytes that any uncompressed VHD image also has, which are not part of the image data itself, but contain the VHD metadata.

However, those erroneous 512 bytes do not appear to be due to a short write (where Rufus would somehow "forget" to read or write the last 512 bytes of data) as we definitely do write (and read) the expected amount of bytes. Instead it does appear that the last 512 bytes of the buffer we use to write the final data have been misread somehow.

From what I can also see, when accessing it through different means, the data from the virtual drive we read from is the expected one, so it's currently a puzzle that will probably require time before I can figure it out. I also can't explain why this issue would not manifest itself on the other tests I had ran, such as the one I highlighted above, as if this was a consistent issue, we would have had comparison errors before EOF (which is actually how I found that the bytes started to differ at end of drive - 512). So the size of the source VHD seems to have an effect as well...

Again, when I have more, I will report, but don't expect much for a while.

@AMIGASYSTEM
Copy link
Author

AMIGASYSTEM commented Jun 1, 2024 via email

@pbatard
Copy link
Owner

pbatard commented Jun 1, 2024

@AMIGASYSTEM, I believe that, if you partition your drive so that the last partition does not end at the end of the disk, but one (or a few) sector(s) prior, then you should still be able to use Rufus while I figure out this thing. I'm assuming that, at the moment, you partition the disk to use all the space available, but if you are creating public images for people to use, I figure you can probably reduce the size of the SFS partition by a small amount, in which case the issue with Rufus should not trigger.

@AMIGASYSTEM
Copy link
Author

AMIGASYSTEM commented Jun 1, 2024 via email

@pbatard
Copy link
Owner

pbatard commented Jun 2, 2024

Well, the other option is to just rename the .vhd to .img and use that as a flat DD image, since a .vhd is just a flat image with a 512 byte footer, and you can definitely use it as a flat image (because these extra 512 bytes cannot interfere regardless of whether you write or don't write them). When not reading the source from a mounted virtual VHD (which is how Rufus copies the data from .vhd and .vhdx) but from the source image directly, the last 512 bytes are copied as expected (even though it's using the exact same code to read from the source image as it does to read from a mounted VHD... But, from what I can see, there's indeed some weird behaviour happening when you start reading the end of a mounted VHD).

@AMIGASYSTEM
Copy link
Author

AMIGASYSTEM commented Jun 2, 2024 via email

@pbatard
Copy link
Owner

pbatard commented Jun 3, 2024

This last issue will now be fixed in the next release of Rufus.

As I suspected, this didn't really have anything to do with Rufus, but instead with Windows completely losing its marbles when you attempt to read past the end of a mounted VHD/VHDX because, when you do that, not only will Windows return extra data that doesn't exist, but it will also corrupt the data that does exist.

In every other Operating System (or when reading directly from a file) attempting to read past the end of a file or device is something that's 100% allowed and handled gracefully (i.e you can rely on file I/O operations to return EOF or no data and let you go on your merry way), but it seems that the team that implemented VHD handling at Microsoft didn't get the memo, because boy will it screw you when you are issuing asynchronous operations. But I guess that's what you get from expecting an Amiga-level of proper OS design from the Redmond boys...

As a result, I reworked the code to make sure that we don't ever attempt to read past the end of the image/VHD source, which seems to keep Windows happy and now allows it to return the expected data.

pbatard added a commit that referenced this issue Jun 3, 2024
* If you attempt to read past the end of a mounted VHD, not only will Windows happily
  return data that doesn't exist (instead of returning End of Disk), but it will also
  corrupt the existing data!
* So, to appease the capricious Windows gods, we now make sure that we never attempt
  to read (or write) past the boundaries of the source or target when writing images.
* This should address the last issue from #2468.
* Also set version to rufus-next and make some small improvement in winio.h.
@AMIGASYSTEM
Copy link
Author

AMIGASYSTEM commented Jun 3, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants