Use zstandard compression for snapshot archives - #1195
Merged
Alenar merged 14 commits intoSep 1, 2023
Conversation
dlachaume
force-pushed
the
ensemble/876/use-zstandard-compression-for-snapshot-archives
branch
from
August 31, 2023 17:52
0ca682c to
19be76a
Compare
dlachaume
temporarily deployed
to
testing-preview
August 31, 2023 18:02 — with
GitHub Actions
Inactive
Alenar
marked this pull request as ready for review
August 31, 2023 18:17
jpraynaud
approved these changes
Sep 1, 2023
jpraynaud
left a comment
Member
There was a problem hiding this comment.
LGTM 👍
Just need to make the few adjustments we discussed!
Alenar
force-pushed
the
ensemble/876/use-zstandard-compression-for-snapshot-archives
branch
from
September 1, 2023 09:31
70691d5 to
08cfbbe
Compare
added 3 commits
September 1, 2023 11:31
…Snapshotter Rename GzipSnapshotter to CompressedArchiveSnapshotter
Alenar
force-pushed
the
ensemble/876/use-zstandard-compression-for-snapshot-archives
branch
2 times, most recently
from
September 1, 2023 09:42
cf22cb4 to
23645ae
Compare
Alenar
temporarily deployed
to
testing-preview
September 1, 2023 09:50 — with
GitHub Actions
Inactive
added 6 commits
September 1, 2023 11:52
Alenar
force-pushed
the
ensemble/876/use-zstandard-compression-for-snapshot-archives
branch
from
September 1, 2023 09:55
23645ae to
6faa297
Compare
This allow to specify a ratio per algorithm.
Alenar
force-pushed
the
ensemble/876/use-zstandard-compression-for-snapshot-archives
branch
from
September 1, 2023 10:00
6faa297 to
445d76d
Compare
Alenar
temporarily deployed
to
testing-preview
September 1, 2023 10:13 — with
GitHub Actions
Inactive
Alenar
deleted the
ensemble/876/use-zstandard-compression-for-snapshot-archives
branch
September 1, 2023 10:16
Alenar
requested review from
turmelclem
and removed request for
ghubertpalo and
turmelclem
August 6, 2026 08:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Content
This PR add the Zstandard compression algorithm as an option for archive creation.
In order to do so a field named
compression_algorithmis added to theSnapshotentity and message. For the message this field is optional and will default to the previous algorithm,Gunzip, if missing.Furthermore Zstandard algorithm can be configured in the aggregator configuration: we can set the compression level (default to
9) and the number of worker (default to4, using0will disable multi-threading).Pre-submit checklist
Comments
Note: the
snapshotter::verify_archivefunction add to be updated to allow concurrent execution. Before it always used the same folder to work, now the folder name is appended with the archive name.Issue(s)
Closes #876