Use zstd for sdk container images #950
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We currently use gzip together with pigz (parallel gzip) for importing container images, and this is a lengthy operation (takes multiple minutes). By moving to zstd we gain on all fronts: zstd produces smaller files, and is faster to decompress/compress then pigz while using less resources. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
This replaces pigz, so remove the related variables (PIGZ). Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
pothos
reviewed
Jun 28, 2023
|
Maybe one more CI job with an SDK build, otherwise it looks good |
Test report for 3648.0.0+nightly-20230627-2100 / amd64 arm64Platforms tested : qemu_uefi-amd64 qemu_update-amd64 qemu_uefi-arm64 qemu_update-arm64
Diagnostic output for qemu_uefi-amd64, run 1
Diagnostic output for qemu_update-amd64, run 1
Diagnostic output for qemu_uefi-arm64, run 5Diagnostic output for qemu_uefi-arm64, run 3Diagnostic output for qemu_uefi-arm64, run 2Diagnostic output for qemu_uefi-arm64, run 1
Diagnostic output for qemu_uefi-amd64, run 3Diagnostic output for qemu_uefi-amd64, run 2Diagnostic output for qemu_uefi-amd64, run 1
Diagnostic output for qemu_uefi-amd64, run 1
Diagnostic output for qemu_uefi-amd64, run 1
Diagnostic output for qemu_uefi-amd64, run 1
|
Zstd is now used for container images compression so make sure it is part of our runners. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
|
Pushed a fix to install zstd in our github actions runners. The sdk job is running here: http://192.168.42.7:8080/job/container/job/sdk/886/cldsv/. |
|
/update-sdk |
krnowak
approved these changes
Jul 3, 2023
dongsupark
added a commit
that referenced
this pull request
Aug 10, 2023
Since #950 was merged, tarball files `flatcar-{packages,sdk}-*.tar.zst` have been created with mode 0600 instead of 0644. As a result, the files with mode 0600 were uploaded to bincache, but afterwards `copy-to-origin.sh` that in turn runs rsync from bincache to the origin server could not read the tarballs. To fix that, it is necessary to chmod from 0600 to 0644 to make it readable by rsync during the release process. All of that happens because zstd sets the mode of the output file to 0600 in case of temporary files to avoid race condition. See also facebook/zstd#1644.
dongsupark
added a commit
that referenced
this pull request
Aug 10, 2023
Since #950 was merged, tarball files `flatcar-{packages,sdk}-*.tar.zst` have been created with mode 0600 instead of 0644. As a result, the files with mode 0600 were uploaded to bincache, but afterwards `copy-to-origin.sh` that in turn runs rsync from bincache to the origin server could not read the tarballs. To fix that, it is necessary to chmod from 0600 to 0644 to make it readable by rsync during the release process. All of that happens because zstd sets the mode of the output file to 0600 in case of temporary files to avoid race condition. See also facebook/zstd#1644.
dongsupark
added a commit
that referenced
this pull request
Aug 11, 2023
Since #950 was merged, tarball files `flatcar-{packages,sdk}-*.tar.zst` have been created with mode 0600 instead of 0644. As a result, the files with mode 0600 were uploaded to bincache, but afterwards `copy-to-origin.sh` that in turn runs rsync from bincache to the origin server could not read the tarballs. To fix that, it is necessary to chmod from 0600 to 0644 to make it readable by rsync during the release process. All of that happens because zstd sets the mode of the output file to 0600 in case of temporary files to avoid race condition. See also facebook/zstd#1644, facebook/zstd#3432.
dongsupark
added a commit
that referenced
this pull request
Aug 11, 2023
Since #950 was merged, tarball files `flatcar-{packages,sdk}-*.tar.zst` have been created with mode 0600 instead of 0644. As a result, the files with mode 0600 were uploaded to bincache, but afterwards `copy-to-origin.sh` that in turn runs rsync from bincache to the origin server could not read the tarballs. To fix that, it is necessary to chmod from 0600 to 0644 to make it readable by rsync during the release process. All of that happens because zstd sets the mode of the output file to 0600 in case of temporary files to avoid race condition. See also facebook/zstd#1644, facebook/zstd#3432.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Use zstd for sdk container images
This PR switches to using zstd for compressing/decompressing container images. Zstd tarballs are a bit smaller and compression/decompression is significantly faster gzip (even with pigz). Zstd can handle decompressing gzip payloads, so introduce a fallback to fetch gzip payloads so that we can still consume nightly sdks produced before this commit.
How to use
./run_sdk_container -tor in CI.Testing done
running: http://192.168.42.7:8080/job/container/job/packages_all_arches/2066/cldsv/
changelog/directory (user-facing change, bug fix, security fix, update)/bootand/usrsize, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc./update-sdk