supporting zstd

[cmurf]

Downstream Fedora bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1743029

Summary: I'm getting very close resulting size (but not identical) when using -Xcompression-level 19 vs -Xcompression-level 22. Also the memory consumption isn't changed, and time is alone slightly different. I can't account for the small differences (caching? nothing else was running on the test system), but also I'd expect level 22 to be a lot slower and use a lot more memory and produce something very close to xz, yet it's ~5-7% larger. I used 256K block size for zstd, and default 128K blocksize for xz.

bytes
1877843968 30min squashfsxzbcj.img
2033696768 13min squashfsz15.img
2020720640 31min squashfsz19.img
2020712448 41min squashfsz22.img

My suspicion based on the zstd man page and CLI interface, is that --ultra needs to be passed to unlock compression levels 20+, otherwise compression level 19 is used. I'm not certain this applies to the libzstd interface mksquashfs uses. Bruno also expressed confusion about the interface.

Downstream work supporting zstd since squashfs-tools-4.3-21
https://src.fedoraproject.org/rpms/squashfs-tools/c/258bc8ea26b48e872f6d886bd830f774634e6fc9?branch=master

Vaguely interesting, Facebook's XAR project, which leverages squashfs, are using level 16 by default.
https://github.com/facebookincubator/xar/blob/master/xar/commands/bdist_xar.py

Just to make it clear, I'm asking a question if what I'm seeing is expected and level 22 is really being used. I'm not asking for levels 20+ to be enabled, necessarily. The resource requirements are so much higher for both compression and decompression, in particular level 22, that I'm not certain, one way or another, they should be made available.

[eworm-de]

Looks like limiting compression level is done from cli tool: https://github.com/facebook/zstd/blob/d1cc9d279704057c08ab976618c8bd629bc18f7e/programs/zstdcli.c#L1162
Could not find anything relevant in the library. I can not explain why everything with compression level 19 and up looks very similar, though.

[eworm-de]

Can you check size if you specify -always-use-fragments?

squashfs-tools/USAGE

Lines 497 to 507 in 81e9fe7

	The -always-use-fragments option tells mksquashfs to always generate
	fragments for files irrespective of the file length. By default only small
	files less than the block size are packed into fragment blocks. The ends of
	files which do not fit fully into a block, are NOT by default packed into
	fragments. To illustrate this, a 100K file has an initial 64K block and a 36K
	remainder. This 36K remainder is not packed into a fragment by default. This
	is because to do so leads to a 10 - 20% drop in sequential I/O performance, as a
	disk head seek is needed to seek to the initial file data and another disk seek
	is need to seek to the fragment block. Specify this option if you want file
	remainders to be packed into fragment blocks. Doing so may increase the
	compression obtained BUT at the expense of I/O speed.

[plougher]

This issue has been unresolved for over a year. It is also unclear whether there is a bug here, or whether it is expected behaviour. What is clear, however, is this isn't an issue with Mksquashfs pe se, but with the Facebook supplied zstd wrapper code. As such the issue should really be directed at the Facebook zstd developers, as they're the only people who can give a definitive answer.

Closing this due to lack of information.