compression: support zstd negative ("fast") levels - #10084
Merged
ThomasWaldmann merged 1 commit intoAug 12, 2026
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #10084 +/- ##
==========================================
+ Coverage 86.70% 86.77% +0.07%
==========================================
Files 98 98
Lines 17174 17277 +103
Branches 2609 2622 +13
==========================================
+ Hits 14890 14992 +102
- Misses 1586 1587 +1
Partials 698 698 ☔ View full report in Codecov by Harness. |
…9950 zstd levels below 1 trade compression ratio for speed. They are useful: on data with long repeats (disk/VM images, database files), zstd,-50 reached more than twice lz4's compression ratio at more than twice its speed. Which of them is worth using depends a lot on the data, so we accept the full range the level byte can hold rather than cutting it off somewhere: zstd,-128 .. zstd,22. The clevel byte is now interpreted per compressor: for zstd it is an int8_t, for everything else it stays unsigned as before. Levels 1..22 encode to the byte value they always had, so existing repositories keep their meaning and no repo format change is needed. The encoding lives in the new CompressorBase.encode_level/decode_level classmethods (identity by default, overridden by ZSTD), so only zstd is affected and every other compressor stores exactly the byte it did before. repo-compress compares the compression settings it wants against the ones found in the repo, so both sides have to use the same representation: get_csettings now returns the stored byte. Otherwise every object would be parsed and recompressed on every run, only to then keep the old bytes. format_compression_spec decodes the byte before checking it against 255 ("level not applicable"), as 255 is a valid zstd level (-1). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UowHQDcnmow3JnYBmaGDpN
ThomasWaldmann
force-pushed
the
zstd-negative-levels-9950
branch
from
August 12, 2026 08:06
eb6ccf5 to
bf6a449
Compare
This was referenced Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds support for zstd's negative ("fast") levels:
zstd,-128..zstd,22. Fixes #9950.Why
Negative levels trade compression ratio for speed. How much they are worth depends strongly on the data, which is why the full range the level byte can hold is accepted rather than cutting it off somewhere (benchmarks in #9950):
nonezstd,-50reached 2.2x lz4's ratio at 2.2x lz4's speed, and even-128beat lz4 on both axesborg cannot know the user's data, so it does not impose a cutoff libzstd does not have.
Compatible with existing repositories
The clevel byte is now interpreted per compressor: an
int8_tfor zstd, unsigned as before for everything else. Levels 1..22 encode to the byte value they always had, andCompressionSpecnever allowed anything outside 1..22 for zstd, so byte values 128..255 have never been written forctype=0x03by borg 1.x or 2.x. No repo format change.Verified by building unmodified borg, writing a repo with it (
zstd,3,zstd,22,lz4), then reading that repo with this branch: all archives extract byte-identical andcheck --verify-datapasses.Implementation
CompressorBase.encode_level/decode_levelclassmethods (identity by default, overridden byZSTD). Doing it per compressor rather than globally means only zstd changes meaning and every other compressor keeps storing exactly the byte it did before. A global signed interpretation was considered and rejected:ObfuscateSizeuses levels 110..123 and 250, which do not fit in anint8_t, and it would have changed the byte written for lz4/none.Two spots in
repo-compressneeded fixing, as they compare or display the byte without knowing the type:get_csettingsreturned the compressor object's level whiletransformcompares it against the stored byte. With them diverging, every object gets parsed and recompressed on every run, only to then keep the old bytes - wasteful and silent. It now returns the stored byte.format_compression_spechid the level when the byte was 255 ("level not applicable"), which would printzstd,-1aszstd. It now decodes before that check.Tests
Level encoding, data round trips (normal and legacy mode), a guard that no other compressor's byte changed, and a
repo-compresstest that a second run recognises the objects as already compressed - the regression guard for theget_csettingsfix (checking only "0 recompressed" would not catch it, since misread objects get recompressed to an identical result and are then counted as "kept as-is").Full test suite passes.
🤖 Generated with Claude Code
https://claude.ai/code/session_01UowHQDcnmow3JnYBmaGDpN