Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scripts/image: use 256kb blocksize for squashfs images #2992

Merged
merged 1 commit into from Oct 3, 2018

Conversation

@antonlacon
Copy link
Contributor

commented Sep 21, 2018

Squashfs allows configuring a blocksize between 4KB and 1MB. The default is
128KB. Increasing blocksize, in general, increases compression efficiency
at a cost of increased access time. Using 256KB for a blocksize appears to be
a sweet spot balancing the two.

Gzip decreases image by ~700KB.
Lzo decreases image by ~850KB.
Zstd decreaes image by ~2.5MB.

Benchmarking device is an RPi3 running an AArch32 build of LE-9 git from 9/19/18. Test data is a LE9 git filesystem (the /etc and /usr directories - 303MB) from the end of August; the same used for compression level testing. Filename is squashfs-compression method-compression level-blocksize. UTIME is access time to decompress and dump the image to /dev/null.

Filename Size Ratio UTIME
squashfs-gzip-9-4096.squashfs 141212 49.27% 0m11.37s
squashfs-gzip-9-8192.squashfs 135588 47.33% 0m9.80s
squashfs-gzip-9-16384.squashfs 131064 45.76% 0m9.19s
squashfs-gzip-9-32768.squashfs 126864 44.30% 0m8.58s
squashfs-gzip-9-65536.squashfs 124016 43.31% 0m9.03s
squashfs-gzip-9-131072.squashfs 122516 42.79% 0m8.67s
squashfs-gzip-9-262144.squashfs 121788 42.54% 0m8.81s
squashfs-gzip-9-524288.squashfs 121476 42.43% 0m10.12s
squashfs-gzip-9-1048576.squashfs 121328 42.38% 0m11.34s
squashfs-lzo-9-4096.squashfs 159424 55.62% 0m7.85s
squashfs-lzo-9-8192.squashfs 152772 53.33% 0m7.93s
squashfs-lzo-9-16384.squashfs 146800 51.26% 0m7.71s
squashfs-lzo-9-32768.squashfs 140892 49.20% 0m7.57s
squashfs-lzo-9-65536.squashfs 136336 47.61% 0m7.36s
squashfs-lzo-9-131072.squashfs 133668 46.69% 0m7.42s
squashfs-lzo-9-262144.squashfs 132436 46.26% 0m7.51s
squashfs-lzo-9-524288.squashfs 131860 46.06% 0m7.49s
squashfs-lzo-9-1048576.squashfs 131552 45.95% 0m8.20s
squashfs-zstd-22-4096.squashfs 136072 47.48% 0m11.66s
squashfs-zstd-22-8192.squashfs 129508 45.21% 0m10.38s
squashfs-zstd-22-16384.squashfs 124008 43.30% 0m9.66s
squashfs-zstd-22-32768.squashfs 118996 41.56% 0m9.31s
squashfs-zstd-22-65536.squashfs 115388 40.30% 0m9.18s
squashfs-zstd-22-131072.squashfs 112624 39.34% 0m9.22s
squashfs-zstd-22-262144.squashfs 109632 38.29% 0m9.33s
squashfs-zstd-22-524288.squashfs 106472 37.19% 0m10.20s
squashfs-zstd-22-1048576.squashfs 103540 36.17% 0m10.71s
@antonlacon

This comment has been minimized.

Copy link
Contributor Author

commented Sep 21, 2018

@MilhouseVH

This comment has been minimized.

Copy link
Contributor

commented Sep 22, 2018

Yes, the 256KB blocksize looks pretty good - thanks!

zstd also looks promising, though not entirely sure when we'll switch to that (but I'm sure we will eventually).

@MilhouseVH MilhouseVH added the LE 9.0 label Sep 22, 2018
@MilhouseVH

This comment has been minimized.

Copy link
Contributor

commented Sep 22, 2018

One thought might be to make the blocksize configurable - 256KB may not be the sweetspot for all compression algos. Adding -b 262144 to SQUASHFS_COMPRESSION_OPTION would give more flexibility:

if [ "$SQUASHFS_COMPRESSION"  = "lzo" -o "${SQUASHFS_COMPRESSION:-gzip}" = "gzip" ]; then
  SQUASHFS_COMPRESSION_OPTION="-Xcompression-level 9 -b 262144"
elif [ "$SQUASHFS_COMPRESSION" = "zstd" ]; then
  SQUASHFS_COMPRESSION_OPTION="-Xcompression-level 22 -b 262144"
fi

I know it's the same blocksize for lzo/gzip/zstd but anyone using a different compression algo (eg. xz, lz4 etc.) can then specify their own blocksize and compression level.

In fact we probably only want to set SQUASHFS_COMPRESSION_OPTION if it's not already set, ie.:

if [ -z "$SQUASHFS_COMPRESSION_OPTION" ]; then
  if [ "$SQUASHFS_COMPRESSION"  = "lzo" -o "${SQUASHFS_COMPRESSION:-gzip}" = "gzip" ]; then
    SQUASHFS_COMPRESSION_OPTION="-Xcompression-level 9 -b 262144"
  elif [ "$SQUASHFS_COMPRESSION" = "zstd" ]; then
    SQUASHFS_COMPRESSION_OPTION="-Xcompression-level 22 -b 262144"
  fi
fi

as this would allow Xcompression-level and blocksize to be custom configured even for lzo/gzip/zstd.

And then the change to the mksquashfs line can be dropped.

Squashfs allows configuring a blocksize between 4KB and 1MB. The default is
128KB. Increasing blocksize, in general, increases compression efficiency
at a cost of increased access time. Using 256KB for a blocksize appears to be
a sweet spot balancing the two for gzip and zstd. Blocksize 512KB appears
right for lzo.

Gzip decreases by ~700KB.
Lzo decreases by ~1.25MB.
Zstd decreaes by ~2.5MB.

Signed-off-by: Ian Leonard <antonlacon@gmail.com>
@antonlacon antonlacon force-pushed the antonlacon:blocksize branch from ea71d11 to cd6287c Sep 22, 2018
@antonlacon

This comment has been minimized.

Copy link
Contributor Author

commented Sep 22, 2018

Adjusted and split lzo out to its own test using 512kb blocksize based on the above testing. New image size reduction there is ~1.25MB.

@MilhouseVH

This comment has been minimized.

Copy link
Contributor

commented Oct 3, 2018

No issues reported from almost 2 weeks testing RPi/RPi2/Generic, where I saw similar reductions in size as you found. Best way to test this now is to merge it - many thanks!

@MilhouseVH MilhouseVH merged commit 6499607 into LibreELEC:master Oct 3, 2018
@antonlacon antonlacon deleted the antonlacon:blocksize branch Oct 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.