Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scripts/image: use maximum lzo and zstd compression level for images #2892

Merged
merged 1 commit into from Aug 28, 2018

Conversation

Projects
None yet
3 participants
@antonlacon
Copy link
Contributor

commented Aug 16, 2018

The compression methods used by mksquashfs take compression levels, which are trade offs between time spent compression and size of the compressed file. This makes use of them for lzo and zstd. For lzo, it's a ~40kb size savings, and for zstd it's about 500kb size savings.

I also tested gzip, where the size reduction was 4kb, and xz where size reduction was 8kb. If those reductions should be included, let me know. gzip is a simple -o squashfs_compression == gzip to the lzo test, while xz requires exporting a variable.

@MilhouseVH

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2018

Thanks!

Could you add the gzip support, as we use that for Generic. Could you add it as:

if [ "$SQUASHFS_COMPRESSION"  = "lzo" -o "${SQUASHFS_COMPRESSION:-gzip}" = "gzip" ]; then

as that will ensure the compression level is applied even when the compression method isn't configured.

(Note that we don't use == as it's not the POSIX (or coding) standard - single = should be used)

We don't currently use xz with any projects (and I don't think we have any plans to do so at this time) so there'd be no point adding extra complexity for something we don't use. That said, we probably will use zstd in future, so that's a nice to have.

scripts/image: use maximum lzo and zstd compression level for images
For LZO, this saves about 40kb on img.gz size.

For GZIP, this saves about 4kb on img.gz size.

For ZSTD, this saves about 500kb on img.gz size.

Signed-off-by: Ian Leonard <antonlacon@gmail.com>

@antonlacon antonlacon force-pushed the antonlacon:compression-levels branch from 7358cd9 to aecf282 Aug 16, 2018

@antonlacon

This comment has been minimized.

Copy link
Contributor Author

commented Aug 16, 2018

Adjusted. Cheers,

@CvH

This comment has been minimized.

Copy link
Member

commented Aug 17, 2018

@antonlacon Remember higher compression levels are mostly resulting in higher cpu/time usage at extraction. Generic is powerful enough for everything, RPix and even S905 have problems with the basic compression levels. Difficult to predict if the additional time matters or not. Pls do some time comparison too (how long the RPi... takes to read the squashfs).
The results for zstd at ARM are not that great speedwise, ofc we done the test with an much older zstd version.

@antonlacon

This comment has been minimized.

Copy link
Contributor Author

commented Aug 18, 2018

I used a script to copy test data to a ramdisk, and then create multiple squashfs images with different settings. Those images were then mounted and had the files within cat'ed to /dev/null. The test data was a filesystem built from an LE-9 git pull from earlier this week without debug. Symlinks and empty directories were deleted from /, leaving /etc, /opt and /usr. Its uncompressed size was about 303MB.

Software libraries to build the images:

zstd: 1.3.4
zlib 1.2.11
lzo: 2.10

Results from an RPi3 running LE-9 git from earlier this week built for aarch32 off an sd card. It was running kodi (idle) and whatever else LE runs in the background active during the test. Compression level 8 is the default for gzip and lzo when invoked by mksquashfs. 15 is the default for zstd.

Filename UTIME
uncompressed 0m14.94s
squashfs-gzip-8-131072.squashfs 0m8.63s
squashfs-gzip-9-131072.squashfs 0m8.64s
squashfs-lzo-8-131072.squashfs 0m7.39s
squashfs-lzo-9-131072.squashfs 0m7.40s
squashfs-zstd-15-131072.squashfs 0m9.40s
squashfs-zstd-22-131072.squashfs 0m9.36s

Uncompressed was done by hand (same command as script). The rest were by scripts. The scripts are available here: https://github.com/antonlacon/squashfs-experiments/tree/LE-testing

Edit the scripts for desired block size and compression level to test, or run on all 240 combinations. Images are built with test-mksquashfs.sh, and tested on the LE device with test-mksquashfs-device.sh. report.sh outputs the results to csv.

@CvH

This comment has been minimized.

Copy link
Member

commented Aug 18, 2018

Iirc we had much much worse timings for zstd at rpi (or was it rpi 1+2). Still 2 secs where i have no idea how to assess it. Maybe something Milhouse could elaborate.

@MilhouseVH

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2018

Maybe something Milhouse could elaborate.

I can't recall any concerns with zstd performance, maybe I had some initially with early versions but v1.34 introduced several performance improvements.

I'm OK with this change and have had it in my test builds for a week or so, and I think we should merge it.

@CvH

CvH approved these changes Aug 28, 2018

@CvH

This comment has been minimized.

Copy link
Member

commented Aug 28, 2018

lets merge and see whats happen - if something pops up we could easily revert/fix if needed

@CvH CvH merged commit a588776 into LibreELEC:master Aug 28, 2018

@antonlacon antonlacon deleted the antonlacon:compression-levels branch Aug 29, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.