Skip to content
This repository has been archived by the owner on Mar 4, 2024. It is now read-only.

Snapshot compress #207

Merged
merged 6 commits into from
May 28, 2021
Merged

Conversation

MathieuBordere
Copy link
Contributor

@MathieuBordere MathieuBordere commented May 27, 2021

This PR introduces snapshot compression with lz4. Next to compressing the snapshot, there's an additional data integrity check when decompressing the compressed snapshot that in practice should allow help to detect storage corruptions.

Nodes running compression are compatible with nodes running without compression, data over the wire hasn't changed, this should deal with issues where cluster members are only updated one at a time. The compressed snapshot files are regular lz4 files (without the .lz4 extension though) that can be decompressed with the lz4 cli tool.

Fixes #198

@stgraber If this is merged it will probably break some lxd builds as there will be a new dependency liblz4-dev.

@MathieuBordere MathieuBordere marked this pull request as draft May 27, 2021 09:45
@codecov-commenter
Copy link

codecov-commenter commented May 27, 2021

Codecov Report

Merging #207 (8d45e60) into master (34b3ae0) will decrease coverage by 0.15%.
The diff coverage is 78.75%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #207      +/-   ##
==========================================
- Coverage   87.26%   87.10%   -0.16%     
==========================================
  Files          47       48       +1     
  Lines        7646     7804     +158     
  Branches     2044     2107      +63     
==========================================
+ Hits         6672     6798     +126     
- Misses        880      900      +20     
- Partials       94      106      +12     
Impacted Files Coverage Δ
src/uv_fs.c 76.31% <ø> (ø)
src/compress.c 75.37% <75.37%> (ø)
src/uv_snapshot.c 77.72% <95.23%> (+0.83%) ⬆️
src/uv.c 87.40% <100.00%> (+0.43%) ⬆️
src/client.c 78.82% <0.00%> (ø)
src/membership.c 78.64% <0.00%> (ø)
src/replication.c 80.11% <0.00%> (ø)
src/byte.c 99.28% <0.00%> (+0.71%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 34b3ae0...8d45e60. Read the comment docs.

@MathieuBordere MathieuBordere marked this pull request as ready for review May 27, 2021 12:08
@stgraber stgraber merged commit edb8dc1 into canonical:master May 28, 2021
@MathieuBordere MathieuBordere deleted the snapshot-compress branch December 9, 2022 10:11
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Raft snapshots take up a lot of disk space
3 participants