Skip to content

Use zstandard compression for snapshot archives - #1195

Merged
Alenar merged 14 commits into
mainfrom
ensemble/876/use-zstandard-compression-for-snapshot-archives
Sep 1, 2023
Merged

Use zstandard compression for snapshot archives#1195
Alenar merged 14 commits into
mainfrom
ensemble/876/use-zstandard-compression-for-snapshot-archives

Conversation

@Alenar

@Alenar Alenar commented Aug 31, 2023

Copy link
Copy Markdown
Collaborator

Content

This PR add the Zstandard compression algorithm as an option for archive creation.

In order to do so a field named compression_algorithm is added to the Snapshot entity and message. For the message this field is optional and will default to the previous algorithm, Gunzip, if missing.

Furthermore Zstandard algorithm can be configured in the aggregator configuration: we can set the compression level (default to 9) and the number of worker (default to 4, using 0 will disable multi-threading).

Pre-submit checklist

  • Branch
    • Tests are provided (if possible)
    • Crates versions are updated (if relevant)
    • Commit sequence broadly makes sense
    • Key commits have useful messages
  • PR
    • No clippy warnings in the CI
    • Self-reviewed the diff
    • Useful pull request description
    • Reviewer requested
  • Documentation
    • Update README file (if relevant)
    • Update documentation website (if relevant)
    • Add dev blog post (if relevant)

Comments

Note: the snapshotter::verify_archive function add to be updated to allow concurrent execution. Before it always used the same folder to work, now the folder name is appended with the archive name.

Issue(s)

Closes #876

@github-actions

github-actions Bot commented Aug 31, 2023

Copy link
Copy Markdown

Test Results

    3 files  ±0    17 suites  ±0   5m 2s ⏱️ -13s
673 tests +2  673 ✔️ +2  0 💤 ±0  0 ±0 
729 runs  +2  729 ✔️ +2  0 💤 ±0  0 ±0 

Results for commit 445d76d. ± Comparison against base commit a6caa1c.

♻️ This comment has been updated with latest results.

@dlachaume
dlachaume force-pushed the ensemble/876/use-zstandard-compression-for-snapshot-archives branch from 0ca682c to 19be76a Compare August 31, 2023 17:52
@dlachaume
dlachaume temporarily deployed to testing-preview August 31, 2023 18:02 — with GitHub Actions Inactive
@Alenar
Alenar marked this pull request as ready for review August 31, 2023 18:17
@Alenar
Alenar temporarily deployed to testing-preview August 31, 2023 18:29 — with GitHub Actions Inactive

@jpraynaud jpraynaud left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

Just need to make the few adjustments we discussed!

@Alenar
Alenar force-pushed the ensemble/876/use-zstandard-compression-for-snapshot-archives branch from 70691d5 to 08cfbbe Compare September 1, 2023 09:31
@Alenar
Alenar force-pushed the ensemble/876/use-zstandard-compression-for-snapshot-archives branch 2 times, most recently from cf22cb4 to 23645ae Compare September 1, 2023 09:42
@Alenar
Alenar temporarily deployed to testing-preview September 1, 2023 09:50 — with GitHub Actions Inactive
@Alenar
Alenar force-pushed the ensemble/876/use-zstandard-compression-for-snapshot-archives branch from 23645ae to 6faa297 Compare September 1, 2023 09:55
@Alenar
Alenar force-pushed the ensemble/876/use-zstandard-compression-for-snapshot-archives branch from 6faa297 to 445d76d Compare September 1, 2023 10:00
@Alenar
Alenar temporarily deployed to testing-preview September 1, 2023 10:13 — with GitHub Actions Inactive
@Alenar
Alenar merged commit 6ced8e6 into main Sep 1, 2023
@Alenar
Alenar deleted the ensemble/876/use-zstandard-compression-for-snapshot-archives branch September 1, 2023 10:16
@Alenar Alenar mentioned this pull request Aug 6, 2026
12 tasks
@Alenar
Alenar requested review from turmelclem and removed request for ghubertpalo and turmelclem August 6, 2026 08:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use zstandard compression for snapshot archives

3 participants