truncate and cp: sparse could be supported on Windows

[janko-jj]

Trying coreutils.exe cp --sparse=always filea fileb results in cp: --sparse is only supported on linux. However Windows supports sparse files:

DeviceIoControl( h, FSCTL_SET_SPARSE, 0, 0, 0, 0, 0, 0 ); is used to set the sparse flag. To skip over the parts that shouldn't be allocated the fseek is used just like on Linux or Mac, and the file is truncated with SetFilePointerEx and SetEndOfFile.

[oech3]

I have no accesfor WIndows now. But how does truncate.exe work now?

[janko-jj]

Cygwin built cp.exe works correctly (creating the sparse target file) with --sparse=always option passed, also when started from the cmd.exe on Windows.

Above, I mean the GNU coreutils' cp.exe installed with Cygwin's installer https://cygwin.com/install.html

I was also able to run the same cp.exe (with the necessary dlls) using wine on Ubuntu (that means you don't have to have an actual Microsoft's Windows to test that behavior):

wine cp --sparse=always filea fileb

and the fileb was also sparse and used less disk space, verified using du -h fileb

wine cp --version displays a lot, but the key lines are:

cp (GNU coreutils) 9.0
Packaged by Cygwin (9.0-1)

(I've used the above line in the directory where cp.exe and its dlls were located)

[janko-jj]

Using busybox-w32 busybox truncate on Windows will also create sparse files, see the resolved issue:

rmyorston/busybox-w32#501

Testing using wine on Ubuntu:

wine busybox.exe truncate -s 10M aa

works as expected (du -h reports 0 for aa).

Also the truncate.exe from cygwin behaves the same, also both in MS Windows and using wine

[oech3]

How about https://packages.msys2.org/search?q=uutils-coreutils (not mingw build)?

[janko-jj]

The truncate from that msys build of uutils does produce "zero on disk" file for e.g. -s 10M but there's more than one way to achieve that: the files with no bytes can be constructed that way even without the FSCTL_SET_SPARSE flag so I don't think that will help you much.

The truncate from https://github.com/uutils/coreutils/releases/download/0.8.0/coreutils-0.8.0-i686-pc-windows-msvc.zip produces 10MB on disk file for -s 10M which is wrong.

In none of these the cp --sparse=always works (instead gives cp: --sparse is only supported on linux) which is also wrong.

The goal it to set the said FSCTL_SET_SPARSE flag at least when --sparse=always is passed, for all Windows builds, and then to use the fseek during copying like it would be done in Linux.

[oech3]

@ognevny Are you familiar with DeviceIoControl?

[ognevny]

no

[oech3]

We should fix truncate before cp. @janko-jj Please add truncate: to title too. Then someone saw this might try.

[janko-jj]

In that specific case DeviceIoControl is just a call to set that 'sparse' flag for the newly created file, at least when --sparse=always is passed. For the uses in truncate you have also an example in busybox-w32. The same way it should be used when --sparse=always is passed to the cp.

[janko-jj]

See https://github.com/rmyorston/busybox-w32/blob/e06df64f035f613829a00f2611242b83549ba0a4/win32/mingw.c#L2465

[oech3]

GPL code is not compat with uutils. Sorry.

[oech3]

I'm not sure is this possible with std only, But it seems some APIs on https://docs.rs/rustix/latest/rustix/fs/index.html has Windows support.