Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Windows 10 Insider: Cannot repeatedly write an image with a mountable partition #883
Additionally (if applicable):
The problem appears to be that if the disk layout in a compressed disk image is similar to the disk layout on the destination drive, the image contents are not written to disk.
On Win10 Pro x64 Insider Fast Ring, build 14986.
Insert USB key, run diskpart, and clean the disk.
Re-insert the drive, cancel the Windows prompt to format the partition.
Insert USB key, run diskpart, and clean the disk.
Log files are in the attached zip:
rufus.clean.703.log, clean install
Thanks for the report. I'll take a look at it (and try to replicate your issue) when I get a chance.
I can however already answer one of your questions:
That's because the progression is based on the compressed input rather than the decompressed output. This means that, if the compressed image ends with section that is a few GB of zeroed data then, because that data will occupy next to no space in the compressed source, the progress will barely move even though Rufus will still be writing all these bytes to the USB. Thus the progress may seem frozen even though a lot of data is actually being processed.
I tried to see if there was a way to have the progress based on the uncompressed output, rather than the source, but the issue is that few compression formats include the total size of the decompressed output (which, of course, we need before we can set up a progress based on that output). Especially, if I recall correctly, gzip is one of the formats that doesn't have the decompressed size.
So, to be able to produce a more user-friendly progress, that would be based on the decompressed gzip output, we would first have to decompress the whole stream, to find its total size, and then decompress it a second time to write to USB... which isn't exactly optimal. Thus, we simply base progress on the amount of compressed bytes read, which may result in apparent slowdowns, when a large amount of data that compresses very well is processed, but ultimately results in the fastest completion.
Okay, I've been trying to replicate your issue, but no luck so far.
No, that's a wrong assumption. If that was the case, the many people who have rewritten similar images over and over would have picked that issue a long time ago, and I would be able to easily replicate your findings, which I am not able to do (Yes, I started with a
This could be due to multiple things. So, the first thing I'd like to ask you is to double check the SHA-1 of your
Then, I will ask you to see what happens when you try to write an uncompressed
Finally, if you're still seeing an issue, I would encourage you to see if you have a security solution that could interfere with USB access. There exists a number of security software out there that keeps a lock on USB drivers (which they may do if the drive cannot be mounted in Windows, in order to check boot records, etc.), which may explain why the decompression library fails early.
You're right, I described the symptoms, not the cause, the main UI did not reporting an error, and no "error" text in the log threw me off.
I tried to write 7.90 over 7.90, and I got the same "write" line, which I now assume means an error, and the UI does not report and error.
After the write the disk is no longer recognized in Windows, so I assume something was written, but not enough to result in a valid file system.
Speculation, maybe timing, maybe related to multiple partitions one FAT and recognized and one Ext4 and not recognized, maybe automatic drive letter assignment changes.
The SHA1 of the image is
I did a diskpart clean, wrote the uncompressed image, removed and remounted, and wrote the uncompressed image again.
If I again do a diskpart clean, and then write the image, compressed or uncompressed, all is well.
Logs attached for 790 over 790, one for the compressed image, and one for the uncompressed image.
Thanks for the update.
Well, the thing is, I used to do just that... and it was causing problems. So I'm reluctant to revert that change, especially as I can't replicate the issue, if the result is that it will help one person... and I will revert to receiving e-mails from people who can no longer write images.
I'd rather understand more precisely why, for your specific configuration, you are seeing such an error, when (as far as I know) nobody else seems to (since the layout of the LE images is common enough that I suspect other people would report a similar issue if it was that common).
Also, please be mindful that you are using an insider version of Windows (i.e. BETA), and I am acutely aware, as has happened in the past, that Microsoft is no stranger to screwing their USB stack in these builds.
So, one thing I'd really like you to test is: Do you see the same issue with the latest non-insider release of Windows.
I may try to replicate the issue on an insider build on my end, but I can't promise anything.
Also, have you tried with a different USB than SanDisk Cruzer? It seems to me that a lot of people are reporting issues/annoyances with these drives (even when not using Rufus at all) that disappear when using a different make/model, so, even as I have seen no verifiable proof that SanDisk Cruzer drives might actually be problematic (and to be fair, I do the bulk of my testing with SanDisk Extreme drives, so I don't have much of an issue with SanDisk as a brand), I'd encourage you to test with a different drive if you can.
Another thing I'd like to know, since it seems cleaning the drive is somewhat involved, is: What happens if you try Alt-Z to zero the drive, before attempting to write the image a second time? Be mindful that, since it writes the whole drive, as opposed to part of it, the zeroing operation may take a little time to complete.
Yes. I already implied that I wasn't happy with the error message for compressed images, so that's something I'll fix in 2.12.
I've noticed the problem with my Kingston drives as well, and I'll try to reproduce on a mainline Win10 build.
I also noticed that if I write, multiple partition warning, write, fail, and write again, the second write succeeds, so it could really just be timing related.
Can you give me some pointers to where the code is that fails, and I can debug that and see how timing changes, while running e.g. procmon.
I doubt that, as the image writing code is fairly dumb (since there's nothing that complex about copying bytes from an image onto a physical drive).
If I were to bet on something, and considering that LE creates an ESP (EFI System Partition), which Windows tries to hide most of the time (which is yet another very stupid decision from Microsoft, as it creates all kind of annoyances), I suspect that it could have to do with Insider builds going even further and taking a lock or something on some of the ESP files, when they see an ESP. Still, it's just a wild guess.
For uncompressed images, it's this one. You'll notice that it's the same code as the one that's used for zeroing a drive, which is why I'd like you to try Alt-Z.
The part of the code that calls on
Oh, and actually, I remembered wrong when I said that I removed the call to cleaning the drive before writing an image. What actually happened is that I had to add an extra call to zap partitions, before I could call on the same IOCTL as the one Microsoft's
For more on how Windows is super brittle when it comes to actually being able to write to a drive, see this. You may try to remove the
I did some more testing.
Most LE users would be using the LE USB-SD creator app, I just prefer to use Rufus as a general tool.
During debugging the error in Rufus happens after writing several successful blocks of data writes in
The LE USB creator app is written in Qt, and maybe I'll try to create a debuggable project, but for now I just followed the reasonably simple code flow:
The logic looks reasonably similar to Rufus, but there is no volume announcement during the image write in LE, and LE works every time.
Sadly, you're not reporting on the testing I asked, which was to confirm whether it is limited to Windows 10 insider (it is) and whether using Alt-Z before attempting to rewrite the data can be used as a workaround (it can).
I have already been burned with some of Windows 10 insider super aggressive policies that Microsoft introduces to test waters (similarly, last time, it had to do with aggressively trying to keep a lock on USB devices, which actually resulted in the safe ejection mechanism being broken), and a lot of people complaining that Rufus didn't work, whereas other applications did, and asking me to fix something that didn't need fixing, because Microsoft eventually identified that they went a bit too far, and fixed the issue they introduced before release.
All this to say that, while I appreciate your pointers, you'll excuse me for not seeing that issue as that high a priority, when it is confined to Insider Builds, and Microsoft is known to break stuff, that may affect an application but not another, in the Insider fast track.
So, for the time being, and especially since I have other stuff to keep me busy, I think I'll take a wait and see approach regarding this Insider specific problem...
That is because they limit themselves to dealing with only two types of compressed images:
Or 1024x1024 = 1MB which is the amount of free space most images/disks tend to reserve before the first mountable partition... and which allows me to deduce, as I suspected, that the issue has to do with Windows Insider trying, a bit too aggressively, to mount the first logical drive it can mount (which is the ESP, starting at the 1MB mark) as soon as it can, with the effect of preventing any physical drive access that also happens to be fall within that logical drive.
This has long been a major bane of Windows (after all, what's the point of locking a physical drive, which Rufus does, if Windows doesn't realize that it should also not allow anyone, including itself, to mount or lock a logical drive within said physical drive?), and I'm already taking all kind of provisions to prevent that from happening. So, before I go and try to add even more empiric code (and maybe some super dedicated code to prevent Windows from mounting said drive — I already go the trouble of temporarily disabling AutoRun, using the cumbersome LGP method, which is the one Microsoft advertises, for crying out loud!), I'd rather be certain that this is not yet another Insider bug that Microsoft will fix.
This makes me wonder if the issue could be that Microsoft broke LGPs or AutoRun in Insider builds.
Anyway, once again, I'd rather have some certainty that this is a real issue that will make its way into Release, before going down another rabbit hole. I spent way too much time last time around trying to explain to people that, yes, Insider can and does introduce major bugs, and no, it's not because an application might be affected by it, whereas another similar application is fine, that the first application needs to be fixed post hate.
This is even more true as, for the time being, Alt-Z can be used as a workaround. This means, I'll keep this issue open as low priority for now.
changed the title from
Compressed disk image not written after writing similar image multiple times
Windows 10 Insider: Cannot repeatedly write an image with a mountable partition
Jan 16, 2017
Tested second write on Win10 Pro x64 release 10.0.14393: Works fine
Tested Alt-Z after first write on Win10 Pro x64 Insider Fast 10.0.15007: Alt-Z, drive wipe took a very very long time, but I could second write after wipe. There were a couple of retry errors in the zeroing log:
I also tested using Win32DiskImager, and like the LE USB creator app, it works fine on both release and Insider builds of Win10...
Thanks for the test.
One thing I forgot to mention is that you can cancel the Alt-Z after a short while, rather than let it complete through the end. It just needs to overwrite the EFI partition, and then, since Windows won't see it any more, it won't try to reaccess it, which seems to be the issue (i.e. it appears that the Insider version of Windows is quite aggressive in trying to remount a partition that it previously had access to, even if was properly unmounted and AutoRun was disabled).
I can't say I'm not curious as to why Rufus seems to be more affected by this than other apps. If anything, we are going further than what other applications do, in that we are formally unmounting the logical volume (
If the problem still is still present in the next few releases of Insider, I'll try to have a closer look.
Okay, since Microsoft isn't showing much of any sign of altering this new behaviour in
Oh, and one thing I noticed in