Skip to content

feat(storage): fold an existing store into compressed frames - #91

Merged
BryanFRD merged 1 commit into
mainfrom
feat/compress-migration
Aug 15, 2026
Merged

feat(storage): fold an existing store into compressed frames#91
BryanFRD merged 1 commit into
mainfrom
feat/compress-migration

Conversation

@BryanFRD

Copy link
Copy Markdown
Contributor

Second half of #89, stacked on #90 — review that one first, this branches from it.

#90 compresses what arrives next. A store that predates it keeps its own shape forever, which for the homelab is all 3.4 GB of it. This is the migration, plus the piece that keeps it from making things worse.

Objects that will not compress

A frame that gives up less than five percent of itself is written as it arrived and flagged in the index; an object whose framed form would be no smaller is left exactly where it is. Half a game store is PNG and OGG, and wrapping those costs a header, an index and CPU to make them marginally larger.

The flag is the top bit of the frame's index entry — a frame is at most sixteen megabytes, so the bit was free.

I broke the format doing this and the tests did not notice: the index offset is the sum of the frame lengths, and I summed the entries with the flag still set. Every existing test used compressible data, so no entry ever carried the flag. The two tests that now cover incompressible objects fail on that commit, which is the only reason it did not ship.

The migration

POST /{org}/{repo}/objects/compress, lfsx compress --repo <org/repo> [--dry-run], admin only.

Shared objects stay shared. After the deduplication migration, most objects are one file with a link per repository — so replacing this repository's link with a compressed copy would quietly undo that. The copy under .content is what gets replaced, and this repository is relinked to it. A repository that has not had its turn keeps the older bytes alive through its own link, so the store holds both forms until every repository has run. Run it everywhere.

The dry run is measured, not estimated. It compresses everything and throws the result away, so the number it prints is the saving. It costs the same CPU as the real thing, which for 3.4 GB is worth paying once before deciding.

Verified against its own digest before being rewritten — the last moment that check is simple, since afterwards the file is no longer the bytes it is named after. Anything that fails is left alone and counted.

Refused outright when LFSX_COMPRESSION is unset, with a 409: rewriting a store into a format the server was not told to use is not a decision to take on an operator's behalf.

Tests

Two more on the format: an incompressible object round-trips through raw frames and costs no more than not trying, and a range inside a raw frame lands correctly.

Seven on the migration: an old object is folded in and still reads; a second run finds the work done; an incompressible object keeps its bytes; an object that lies about its digest is refused with nothing left behind; a dry run measures without writing; a server without compression refuses; and — #[cfg(unix)], so it runs in CI rather than on my machine — compressing one repository leaves a second one sharing those bytes still able to read them.

Copilot AI lite review requested due to automatic review settings August 15, 2026 17:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@BryanFRD
BryanFRD force-pushed the feat/compress-migration branch from 842cb3b to 955ee3b Compare August 15, 2026 17:33
@BryanFRD
BryanFRD enabled auto-merge (squash) August 15, 2026 17:33
@github-actions

Copy link
Copy Markdown

SonarQube — aucune nouvelle issue

Comparaison entre le projet bac à sable de cette PR et la branche par défaut : SonarQube Community n'analyse pas les PR, ce delta est calculé côté CI. Détail

@BryanFRD
BryanFRD merged commit 195b925 into main Aug 15, 2026
19 checks passed
@BryanFRD
BryanFRD deleted the feat/compress-migration branch August 15, 2026 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants