Skip to content

feat(server): collect objects no longer referenced by any commit - #23

Merged
BryanFRD merged 2 commits into
mainfrom
feat/garbage-collection
Aug 14, 2026
Merged

feat(server): collect objects no longer referenced by any commit#23
BryanFRD merged 2 commits into
mainfrom
feat/garbage-collection

Conversation

@BryanFRD

Copy link
Copy Markdown
Contributor

Closes #4.

Objects were written and never removed. A repository that rewrites history, drops a branch or replaces a large asset left the old blobs on disk forever.

The server never sees the Git history, so it cannot decide what is unreferenced. POST /{org}/{repo}/objects/retain takes the set of oids the repository still references and sweeps the rest:

{ "oids": ["a1b2…", "c3d4…"], "dry_run": true }
→ { "swept": 42, "bytes": 3221225472, "within_grace": 3, "dry_run": true }

Nothing is deleted while dry_run is set, and the report is the same either way, so the dry run is a real preview rather than a different code path.

The two things that stop it eating live data

An object is uploaded before the commit referencing it is pushed. A naive sweep deletes objects mid-push, and the client then fails on a 404 for something it just uploaded. Anything touched within LFSX_GC_GRACE is therefore never taken — that is the within_grace count. The default is two weeks, matching git's own gc.pruneExpire.

Transfers in flight are skipped too: sweep only considers filenames that are valid oids, so the {oid}.{ticket}.part staging files are invisible to it. That also keeps this out of the way of #8.

Sweeping requires push rights, like any other write.

Notes

Empty fanout directories are removed as they are emptied, otherwise a long-lived repository accumulates thousands of empty two-character directories.

An empty oids set legitimately means nothing is referenced any more, and outside the grace window that sweeps the repository clean. That is the honest semantic — the README says so and tells you to run it dry first.

Tests

Six, covering both directions: an unreferenced object is swept, one still inside the grace period survives, a listed oid is kept however old it is, a dry run reports the same numbers and frees nothing, a staging file is never collected, and a read-only token gets 403.

cargo test: 30 passing. Clippy clean. The README documents the flow with a git lfs ls-files --all --long pipeline, which I checked against a real repository rather than writing from memory.

Copilot AI lite review requested due to automatic review settings August 14, 2026 11:45
@BryanFRD
BryanFRD enabled auto-merge (squash) August 14, 2026 11:45

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@ferrfleet ferrfleet Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid implementation. sweep_directory's three safety checks (non-oid filenames like .part staging files, retained oids, and the grace period) are each independently correct and well-tested — retained objects are kept regardless of age, staging files are never touched, and nothing under LFSX_GC_GRACE is swept. Directory cleanup (remove_dir on emptied fanout/prefix dirs) is race-safe against concurrent uploads since create_dir_all is idempotent and remove_dir on a non-empty directory just fails silently. The six new tests exercise exactly the properties the PR description claims.

Nit: in sweep_directory (server/src/storage.rs), entry.metadata().await? and fs::remove_file(entry.path()).await? propagate a hard error and abort the entire sweep on any IO hiccup — e.g. if two retain calls race on the same repo and one removes a file the other already listed. Objects deleted earlier in the loop stay deleted, but the caller just gets a 500 with no report of what actually happened. Consider treating NotFound there as already-swept-by-someone-else and continuing, mirroring the best-effort let _ = fs::remove_dir(...) pattern already used elsewhere in this function, so a transient/concurrent error doesn't throw away an otherwise-accurate report.

@BryanFRD
BryanFRD merged commit 3157ca9 into main Aug 14, 2026
14 checks passed
@BryanFRD
BryanFRD deleted the feat/garbage-collection branch August 14, 2026 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Garbage-collect objects no longer referenced by any commit

2 participants