Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
nixos: compress make-ext4-fs with zstd #75592
Motivation for this change
The AArch64 build has been failing for a few days with the output of it's ext4 image being >2.0GB. This compresses it with bz2, thus solving the problem.
I'll add that currently we compress all these large images with bz2, which is very slow, I'd like to suggest a change to zstd or lzo. This would be specially meaningful in architectures where usually there are more cores at the expense of clock (since compression/decompression is single threaded.) In particular ARM families would benefit, with no downside to other arches. I am preparing this in another branch, but would like to see this one merged first.
Difference between bzip2 and zstd
samueldr left a comment
As it is, this breaks the API of
I do not know what is our guarantee with internal~ish builders like those.
This could be fixed by adding a
I don't think that the following is an issue with which we should concern ourselves right now, but to be kept in mind for the eventual rewrite.
Writing a temporary file is costly. It could result in failing builds if the
For the second part (streaming into the final image), it would need metadata written in $out (so $out would not be directly an image file) that tells the image builder what is the size of the image. For the first part, (compression) I guess it depends on what compresses the image file. Are there ways that it can be done in-place? Though, if it's writing in a
A final strategy for the future could be to merge multiple snippets into one big honking derivation that does all the work. The main drawback would be that the derivation is less flexible, and takes more time to build. A well architected solution could possibly do this with an option.