Consider relaxing installation medium validation check to accommodate Windows

[andrewdavidwong]

From the developer of Rufus:

By the way, since that extra directory seems to throw off your disc validation, which is probably why people did report that images created with dd tools (not just Rufus) on Windows "failed", whereas it was just the disc validation being thrown off, I would suggest that you relax your validation check for this specific Windows behaviour on the ESP, else the only solution you'll be able to advise for Windows people to create Qubes bootable media that pass the check will be to use something else than Windows, so that the ESP is not automatically re-mounted in an OS that is designed to create an extra directory there.

Full message

[pbatard]

Just for clarification, as per the full message, the core of the issue is that the default modern Windows behaviour is that, whenever you plug a disk with a file system that Windows can recognize, it creates a System Volume Information\ directory on that file system, that contains a single file called IndexerVolumeGuid.

So the problem is that, after a Windows user writes the Qubes image in dd mode, the FAT32 ESP gets mounted by Windows, which results in the creation of the directory and file above.

Of course, this means that the FAT32 index gets modified and fails the byte comparison...

Possible solutions include:

• Relaxing the check for the ESP, so that it doesn't complain about extra files/directories there
• Creating the System Volume Information\IndexerVolumeGuid when mastering the image. IndexerVolumeGuid basically contains a single GUID, in little endian Unicode (e.g. {DADDB26C-2DB4-420B-A5FE-2D9F955E5799}), which I believe is randomly generated by Windows and doesn't get modified once set. You may have to watch out for the file attributes, as the System Volume Information\ directory must be set as hidden (and both the directory and the Guid file also have the archive attribute)

Note that there also exists a way to disable the policy that creates System Volume Information\, which is something I could try to enforce in Rufus. But I decided not to do that as changing system policies behind users' backs is bad practice (even temporarily), and of course, even if I were to do that, as soon as the user plugs their drive on a computer where the policy is not disabled, the directory will be created and the check will fail.

[pbatard]

Just for clarification, as per the full message, the core of the issue is that the default modern Windows behaviour is that, whenever you plug a disk with a file system that Windows can recognize, it creates a System Volume Information\ directory on that file system, that contains a single file called IndexerVolumeGuid.

So the problem is that, after a Windows user writes the Qubes image in dd mode, the FAT32 ESP gets mounted by Windows, which results in the creation of the directory and file above.

Of course, this means that the FAT32 index gets modified and fails the byte comparison...

Possible solutions include:

• Relaxing the check for the ESP, so that it doesn't complain about extra files/directories there
• Creating the System Volume Information\IndexerVolumeGuid when mastering the image. IndexerVolumeGuid basically contains a single GUID, in little endian Unicode (e.g. {DADDB26C-2DB4-420B-A5FE-2D9F955E5799}), which I believe is randomly generated by Windows and doesn't get modified once set. You may have to watch out for the file attributes, as the System Volume Information\ directory must be set as hidden (and both the directory and the Guid file also have the archive attribute)

Note that there also exists a way to disable the policy that creates System Volume Information\, which is something I could try to enforce in Rufus. But I decided not to do that as changing system policies behind users' backs is bad practice (even temporarily), and of course, even if I were to do that, as soon as the user plugs their drive on a computer where the policy is not disabled, the directory will be created and the check will fail.

[marmarek]

I think relaxing the check is really bad idea. The verification is based on byte comparison (not filesystem structure), so it isn't possible to tell whether such difference is harmless or not at this level.

But adding that directory with a file by default sounds like a reasonable choice. Could you generate one and attach here? Just to make sure it have proper format (newlines etc). I'm not sure if I can force specific fat attribute at ISO generation time, but hope Windows will not change them when the directory+file already exists...

Will it be enough? Or maybe just mounting the partition make some modification (like last mount time or so)?

[marmarek]

I think relaxing the check is really bad idea. The verification is based on byte comparison (not filesystem structure), so it isn't possible to tell whether such difference is harmless or not at this level.

But adding that directory with a file by default sounds like a reasonable choice. Could you generate one and attach here? Just to make sure it have proper format (newlines etc). I'm not sure if I can force specific fat attribute at ISO generation time, but hope Windows will not change them when the directory+file already exists...

Will it be enough? Or maybe just mounting the partition make some modification (like last mount time or so)?

[pbatard]

Could you generate one and attach here?

Sure. Here you go:
System Volume Information.zip

The attributes should be preserved in the archive, and I empirically confirmed that none of directory or files seemed to be updated when plugging the drive on a different computer.

Will it be enough?

I would think so. I'll be happy to test if you have an image that you want to check.

[pbatard]

Could you generate one and attach here?

Sure. Here you go:
System Volume Information.zip

The attributes should be preserved in the archive, and I empirically confirmed that none of directory or files seemed to be updated when plugging the drive on a different computer.

Will it be enough?

I would think so. I'll be happy to test if you have an image that you want to check.

[marmarek]

What WPSettings.dat does?

[marmarek]

What WPSettings.dat does?

[pbatard]

Ah shoot. That's an extra file that Windows 10 seems to create. But it isn't created right after the drive is mounted, which is why I didn't notice it.

From what I can find, this seems to be related to Windows Phone support in Windows 10 and may be used by Microsoft to tie a removable card to a phone (or pseudo-phone since I'm not using any Windows Phone in any way shape or form, and I have no idea why the £$%^&* Microsoft thought it needed to create even more unwanted content on removable devices). Of course, its content and the reason for its creation are undocumented by Microsoft... 😠

If you look at the timestamps, you'll see that it was created long after the Guid file was, so I'm a bit two minds about it: Either ignore it altogether, with the expectation that users will have unplugged their drive before that file gets a chance to get written. But of course, that's taking a bit of a chance, and won't help if the user replug their drive after a while. Or just duplicate the WPSettings.dat that you got from the .zip, which is what I'd advocate.

Looking at the actual content of the file, we see that we have:
0C 00 00 00 1C BC C9 5B 3A 57 50 23

It's pretty clear that 0x0000000C (little endian) is the size of the payload, and with 3A 57 50 23 spelling :WP# which is probably an ASCII marker for "Windows Phone", the only actual data we may be concerned about is: 1C BC C9 5B

So I tried to see if this could be tied to a disk or FAT32 partition serial number, but the one reported for that partition is 7D21-B357 and the disk ID is 78285DAE so they don't seem to be linked.

For good measure I also searched the ISOHybrid for occurrences of 1C BC C9 5B as well as 5B C9 BC 1C, and while I found a couple, those were way past the ESP, so they're unlikely to be what WPSettings.dat is referencing.

Right now, I have recreated a Qubes USB drive, and am waiting for the WPSettings.dat to be created on that drive again, to see if there is a difference in payload. I'll keep you posted.

[pbatard]

Ah shoot. That's an extra file that Windows 10 seems to create. But it isn't created right after the drive is mounted, which is why I didn't notice it.

From what I can find, this seems to be related to Windows Phone support in Windows 10 and may be used by Microsoft to tie a removable card to a phone (or pseudo-phone since I'm not using any Windows Phone in any way shape or form, and I have no idea why the £$%^&* Microsoft thought it needed to create even more unwanted content on removable devices). Of course, its content and the reason for its creation are undocumented by Microsoft... 😠

If you look at the timestamps, you'll see that it was created long after the Guid file was, so I'm a bit two minds about it: Either ignore it altogether, with the expectation that users will have unplugged their drive before that file gets a chance to get written. But of course, that's taking a bit of a chance, and won't help if the user replug their drive after a while. Or just duplicate the WPSettings.dat that you got from the .zip, which is what I'd advocate.

Looking at the actual content of the file, we see that we have:
0C 00 00 00 1C BC C9 5B 3A 57 50 23

It's pretty clear that 0x0000000C (little endian) is the size of the payload, and with 3A 57 50 23 spelling :WP# which is probably an ASCII marker for "Windows Phone", the only actual data we may be concerned about is: 1C BC C9 5B

So I tried to see if this could be tied to a disk or FAT32 partition serial number, but the one reported for that partition is 7D21-B357 and the disk ID is 78285DAE so they don't seem to be linked.

For good measure I also searched the ISOHybrid for occurrences of 1C BC C9 5B as well as 5B C9 BC 1C, and while I found a couple, those were way past the ESP, so they're unlikely to be what WPSettings.dat is referencing.

Right now, I have recreated a Qubes USB drive, and am waiting for the WPSettings.dat to be created on that drive again, to see if there is a difference in payload. I'll keep you posted.

[pbatard]

Well, I waited more than one hour, and tried to replicate what I did before by plugging the USB on a different computer, but still no WPSettings.dat. I suggest you ignore the file then.

[pbatard]

Well, I waited more than one hour, and tried to replicate what I did before by plugging the USB on a different computer, but still no WPSettings.dat. I suggest you ignore the file then.

[marmarek]

@pbatard take a look at https://ftp.qubes-os.org/~marmarek/Qubes-DVD-x86_64-20160607.iso (signature)

[marmarek]

@pbatard take a look at https://ftp.qubes-os.org/~marmarek/Qubes-DVD-x86_64-20160607.iso (signature)

[pbatard]

Unfortunately, that's a NO_GO.

Looks to me like you created the System Volume Information on the wrong partition, as this is what I see from Linux:

root@pi ~ # dd if=/share/Qubes-DVD-x86_64-20160607.iso of=/dev/sda bs=8M
357+0 records in
357+0 records out
2994733056 bytes (3.0 GB, 2.8 GiB) copied, 148.286 s, 20.2 MB/s
root@pi ~ # fdisk -l /dev/sda
Disk /dev/sda: 14.9 GiB, 16013942784 bytes, 31277232 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x15d79229

Device     Boot Start     End Sectors  Size Id Type
/dev/sda1  *        0 5849087 5849088  2.8G  0 Empty
/dev/sda2        1168   64127   62960 30.8M ef EFI (FAT-12/16/32)
root@pi ~ # mount -t vfat /dev/sda2 /mnt/hd
root@pi ~ # ls -alFh /mnt/hd
total 22K
drwxr-xr-x 3 root root  16K Jan  1  1970 ./
drwxr-xr-x 3 root root 4.0K Jan 19 21:45 ../
drwxr-xr-x 3 root root 2.0K Jun  8 00:17 EFI/
root@pi ~ #

This is corroborated by the fact that a new System Volume Information (with a new GUID file) gets created when I write the image using Rufus' dd mode on Windows, and the ESP is mounted.

I do see the System Volume Information\ directory on the ISO9660 partition (/dev/sda1) if I mount it...

[pbatard]

Unfortunately, that's a NO_GO.

Looks to me like you created the System Volume Information on the wrong partition, as this is what I see from Linux:

root@pi ~ # dd if=/share/Qubes-DVD-x86_64-20160607.iso of=/dev/sda bs=8M
357+0 records in
357+0 records out
2994733056 bytes (3.0 GB, 2.8 GiB) copied, 148.286 s, 20.2 MB/s
root@pi ~ # fdisk -l /dev/sda
Disk /dev/sda: 14.9 GiB, 16013942784 bytes, 31277232 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x15d79229

Device     Boot Start     End Sectors  Size Id Type
/dev/sda1  *        0 5849087 5849088  2.8G  0 Empty
/dev/sda2        1168   64127   62960 30.8M ef EFI (FAT-12/16/32)
root@pi ~ # mount -t vfat /dev/sda2 /mnt/hd
root@pi ~ # ls -alFh /mnt/hd
total 22K
drwxr-xr-x 3 root root  16K Jan  1  1970 ./
drwxr-xr-x 3 root root 4.0K Jan 19 21:45 ../
drwxr-xr-x 3 root root 2.0K Jun  8 00:17 EFI/
root@pi ~ #

This is corroborated by the fact that a new System Volume Information (with a new GUID file) gets created when I write the image using Rufus' dd mode on Windows, and the ESP is mounted.

I do see the System Volume Information\ directory on the ISO9660 partition (/dev/sda1) if I mount it...

[marmarek]

Looks like mkefiboot doesn't like to put more than one directory there... Will look into some other way of doing it (probably mount, cp, umount).

[marmarek]

Looks like mkefiboot doesn't like to put more than one directory there... Will look into some other way of doing it (probably mount, cp, umount).

[marmarek]

Automated announcement from builder-github

The package pykickstart-2.13-3.fc23 has been pushed to the r3.2 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

[marmarek]

Automated announcement from builder-github

The package pykickstart-2.13-3.fc23 has been pushed to the r3.2 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

[marmarek]

Automated announcement from builder-github

The package pykickstart-2.13-3.fc23 has been pushed to the r3.2 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

[marmarek]

Automated announcement from builder-github

The package pykickstart-2.13-3.fc23 has been pushed to the r3.2 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

[pbatard]

Sorry for the delay in testing.

I'm afraid I have to report that Qubes-R3.2-rc2-x86_64.iso is still NO_GO in terms of dd writing from Windows and passing the checksum test.

For a start, it seems that Windows 10 (1607 refresh) now consistently creates a System Volume Information\WPSettings.dat. So I think you'll at least need to add such a file as well. Now, the problem we have is that Microsoft still hasn't mentioned anywhere what this file is for, and its content seems to be different every time it is created. For instance, this is the hex data I got in WPSettings.dat on 4 successive attempts:

• 0C 00 00 00 C9 C7 5E 91 C7 DB C8 1A
• 0C 00 00 00 11 4B 2A 0C EF 88 DB 0C
• 0C 00 00 00 0D D9 9B 50 64 C0 39 E5
• 0C 00 00 00 72 1B D8 07 1C AF 72 11

Now, my guess is that this file just contains a random 8 byte sequence, that acts as a fingerprint for the media, to be used when plugged into a Windows Phone, and therefore that simply creating a 12 byte file containing 0x0000000C + a random 8 byte sequence will do.

However, even on the one attempt when I didn't get a WPSettings.dat created, and where the test was expected to work, I still got a failure on the checksum, due to Windows having modified one of the FAT indexes as follows:

NB: This is at offset 0x089290 on the ISO.

So even with an extra WPSettings.dat, the checksum test might still not work. Maybe there's something in the FAT file system creation tools on Linux that don't quite match Windows wants to see, and that Windows corrects on its own, or maybe it's something else. I'm afraid I'd have have to go into FAT specs to find out what this change is all about, which I don't have time for, but maybe you can figure that one out yourselves...

[pbatard]

Sorry for the delay in testing.

I'm afraid I have to report that Qubes-R3.2-rc2-x86_64.iso is still NO_GO in terms of dd writing from Windows and passing the checksum test.

For a start, it seems that Windows 10 (1607 refresh) now consistently creates a System Volume Information\WPSettings.dat. So I think you'll at least need to add such a file as well. Now, the problem we have is that Microsoft still hasn't mentioned anywhere what this file is for, and its content seems to be different every time it is created. For instance, this is the hex data I got in WPSettings.dat on 4 successive attempts:

• 0C 00 00 00 C9 C7 5E 91 C7 DB C8 1A
• 0C 00 00 00 11 4B 2A 0C EF 88 DB 0C
• 0C 00 00 00 0D D9 9B 50 64 C0 39 E5
• 0C 00 00 00 72 1B D8 07 1C AF 72 11

Now, my guess is that this file just contains a random 8 byte sequence, that acts as a fingerprint for the media, to be used when plugged into a Windows Phone, and therefore that simply creating a 12 byte file containing 0x0000000C + a random 8 byte sequence will do.

However, even on the one attempt when I didn't get a WPSettings.dat created, and where the test was expected to work, I still got a failure on the checksum, due to Windows having modified one of the FAT indexes as follows:

NB: This is at offset 0x089290 on the ISO.

So even with an extra WPSettings.dat, the checksum test might still not work. Maybe there's something in the FAT file system creation tools on Linux that don't quite match Windows wants to see, and that Windows corrects on its own, or maybe it's something else. I'm afraid I'd have have to go into FAT specs to find out what this change is all about, which I don't have time for, but maybe you can figure that one out yourselves...

[marmarek]

Re WPSettings.dat what about creating empty file? Or all-zero instead of those "random" bytes? Does any of this prevent windows from modifying it?
As for the other modification - maybe it's about file/directory attributes? I see there part of those windows-related names.

[marmarek]

Re WPSettings.dat what about creating empty file? Or all-zero instead of those "random" bytes? Does any of this prevent windows from modifying it?
As for the other modification - maybe it's about file/directory attributes? I see there part of those windows-related names.

[pbatard]

I'd think that creating a WPSettings.dat file with 0C 00 00 00 00 00 00 00 00 00 00 00 as content would do.

However, looking at FAT Directory Entry specs, which is the other change we're dealing with, I'm pretty sure that those two bytes are the 'Last access date' (offset 0x12).

0x490F = 100100 1000 01111 = (1980+36=2016)/08/15 which is the date of when I created the drive.

And indeed I can see that, when I look at the properties for IndexerVolumeGuid (even as I didn't look at its content), the access date is set to the current date.

My guess is that Windows does access IndexerVolumeGuid as soon as the partition is mounted, and updates the access time accordingly.

So, short of specifically designing your checksum test to ignore those specific 2 bytes, I don't think there's an easy solution for Windows users...

[pbatard]

I'd think that creating a WPSettings.dat file with 0C 00 00 00 00 00 00 00 00 00 00 00 as content would do.

However, looking at FAT Directory Entry specs, which is the other change we're dealing with, I'm pretty sure that those two bytes are the 'Last access date' (offset 0x12).

0x490F = 100100 1000 01111 = (1980+36=2016)/08/15 which is the date of when I created the drive.

And indeed I can see that, when I look at the properties for IndexerVolumeGuid (even as I didn't look at its content), the access date is set to the current date.

My guess is that Windows does access IndexerVolumeGuid as soon as the partition is mounted, and updates the access time accordingly.

So, short of specifically designing your checksum test to ignore those specific 2 bytes, I don't think there's an easy solution for Windows users...

[marmarek]

Is it possible to mark that FAT as read only (in some metadata)?

[marmarek]

Is it possible to mark that FAT as read only (in some metadata)?

[marmarek]

Alternative, crazy idea - patch checksum checker to restore original value in those 2 bytes before. Something like touch -a -r /boot/efi/EFI /boot/efi/System*/IndexerVolumeGuid.

[marmarek]

Alternative, crazy idea - patch checksum checker to restore original value in those 2 bytes before. Something like touch -a -r /boot/efi/EFI /boot/efi/System*/IndexerVolumeGuid.

[pbatard]

I don't think there's a way to make FAT read-only (and, as a possible follow up in that direction, I'm pretty sure that setting a FAT file to read-only doesn't prevent the access time from being altered).

The touch idea might work...

[pbatard]

I don't think there's a way to make FAT read-only (and, as a possible follow up in that direction, I'm pretty sure that setting a FAT file to read-only doesn't prevent the access time from being altered).

The touch idea might work...