feat(storage): fold an existing store into compressed frames - #91
Merged
Conversation
BryanFRD
force-pushed
the
feat/compress-migration
branch
from
August 15, 2026 17:33
842cb3b to
955ee3b
Compare
BryanFRD
enabled auto-merge (squash)
August 15, 2026 17:33
SonarQube — aucune nouvelle issueComparaison entre le projet bac à sable de cette PR et la branche par défaut : SonarQube Community n'analyse pas les PR, ce delta est calculé côté CI. Détail |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second half of #89, stacked on #90 — review that one first, this branches from it.
#90 compresses what arrives next. A store that predates it keeps its own shape forever, which for the homelab is all 3.4 GB of it. This is the migration, plus the piece that keeps it from making things worse.
Objects that will not compress
A frame that gives up less than five percent of itself is written as it arrived and flagged in the index; an object whose framed form would be no smaller is left exactly where it is. Half a game store is PNG and OGG, and wrapping those costs a header, an index and CPU to make them marginally larger.
The flag is the top bit of the frame's index entry — a frame is at most sixteen megabytes, so the bit was free.
I broke the format doing this and the tests did not notice: the index offset is the sum of the frame lengths, and I summed the entries with the flag still set. Every existing test used compressible data, so no entry ever carried the flag. The two tests that now cover incompressible objects fail on that commit, which is the only reason it did not ship.
The migration
POST /{org}/{repo}/objects/compress,lfsx compress --repo <org/repo> [--dry-run], admin only.Shared objects stay shared. After the deduplication migration, most objects are one file with a link per repository — so replacing this repository's link with a compressed copy would quietly undo that. The copy under
.contentis what gets replaced, and this repository is relinked to it. A repository that has not had its turn keeps the older bytes alive through its own link, so the store holds both forms until every repository has run. Run it everywhere.The dry run is measured, not estimated. It compresses everything and throws the result away, so the number it prints is the saving. It costs the same CPU as the real thing, which for 3.4 GB is worth paying once before deciding.
Verified against its own digest before being rewritten — the last moment that check is simple, since afterwards the file is no longer the bytes it is named after. Anything that fails is left alone and counted.
Refused outright when
LFSX_COMPRESSIONis unset, with a409: rewriting a store into a format the server was not told to use is not a decision to take on an operator's behalf.Tests
Two more on the format: an incompressible object round-trips through raw frames and costs no more than not trying, and a range inside a raw frame lands correctly.
Seven on the migration: an old object is folded in and still reads; a second run finds the work done; an incompressible object keeps its bytes; an object that lies about its digest is refused with nothing left behind; a dry run measures without writing; a server without compression refuses; and —
#[cfg(unix)], so it runs in CI rather than on my machine — compressing one repository leaves a second one sharing those bytes still able to read them.