Checkpoints: what to do with large untracked files

[spiky02plateau]

Note

Written by claude-opus-5-5 on behalf of Tobi

Checkpoint capture runs git add -A into a private index under the 30 s Git timeout.
A large untracked file in the workspace (in my case a 1.5 GB model) makes every capture time out, and each killed capture leaves a half-written tmp_pack_* in .git/objects/pack.
My machine collected 240 orphaned packs (153.4 GiB) in two days.
#15400 cleans up those leftover packs.
This discussion is about the other half: whether checkpoints should keep trying to capture such files at all.

Measured on macOS with Git 2.54, one tracked text file plus one untracked file of random data:

Untracked file	Capture today	Capture with a 100 MiB cap (#15297)
600 MiB	14.0 s per turn, writes a 629 MB pack	158 ms, no pack
1.5 GiB	times out after 30 s, leaves a 1.35 GB temp pack, no checkpoint	180 ms, no pack

I'd like direction on which policy you'd accept:

1. Cap untracked files at 100 MiB. Larger untracked files are left out of checkpoints, and restore keeps them, including files created after the checkpoint. This drops coverage that works today: the 600 MiB file is captured, slowly. This is what fix(server): a large untracked file no longer fills the disk with checkpoint packs #15297 does.
2. A higher cap with the same restore behavior. Keeps more coverage. I have not measured 1 GiB, and no fixed size guarantees a capture finishes within 30 s on every machine or with several large files.
3. No cap, cleanup only. The disk stops filling, but captures still time out on every turn while the file exists, so that workspace gets no checkpoints. This is fix(server): failed checkpoints no longer leave temporary packs in the repository #15400 on its own.

How a user would learn that a file was left out: with #15297 as it stands, they would not. Nothing is shown, and they find out only when a restore leaves the large file in place.
If you pick option 1 or 2, I'd add one of these to the same PR, whichever you prefer:

• a server log line on each capture naming the skipped files, or
• a short notice on the turn's changed files, such as "1 untracked file over 100 MiB was not checkpointed".

#15400 is independent and can land whichever option you pick.

Draft implementation of option 1: #15297.

[RomuloDrumond]

Another data point from today's nightly, plus a second leak location I haven't seen mentioned.

Setup: T3 Code 0.1.2-nightly.20261022.3541, macOS 27.0.1 (arm64), git 2.54.0 (Apple Git-157). One Claude thread in a project with 2.8 GB of untracked downloaded videos (largest files 1.2 GB and 724 MB).

What happened: every checkpoint git add -A -- . was killed at the 30 s timeout. The server traces show 73 timed-out captureCheckpoint runs between 14:51 and 18:02 (older traces had rotated). In about four hours .git grew to 123 GB, and the disk was down to 2 GB free when I found it.

Where the garbage was:

Location	Count	Size
.git/objects/tmp_objdir-bulk-fsync-*/pack/tmp_pack_*	61	83 GB
.git/objects/pack/tmp_pack_*	28	40 GB

Two-thirds of it was in tmp_objdir-bulk-fsync-*, not objects/pack/. T3 passes -c core.fsyncMethod=batch, so git stages objects in a quarantine directory that a killed git leaves behind. A cleanup of objects/pack/tmp_pack_* alone would miss it. #15400 may cover this, but it's worth a test with core.fsyncMethod=batch.

On the policy question: option 1 (cap, with restore keeping skipped files) would have prevented this. Whatever the cap ends up being, two more things would help: stop retrying after a few timeouts on an unchanged workspace, and tell the user that checkpoints are failing and why.

Workaround until the PRs land: add the large folders to .git/info/exclude, then quit T3 and delete .git/objects/pack/tmp_pack_* and .git/objects/tmp_objdir-*.

[constantin-jais]

Data point from a different shape of the same leak: many small untracked files, no large file at all, in a repository with no commits. The size-cap approach alone would not have prevented it; the quarantine-and-cleanup approach would.

Setup. T3 Code 0.1.2 (stable), macOS 26.6.2 (arm64), Homebrew git 2.56.0. The project root is a multi-repository workspace: ~60 GB, tens of thousands of files, where every sub-project is its own git repository and the root itself was never meant to be one. A stray git init at that root (0 commits, empty index, author unknown) made the project checkpointable.

What happened. Every captureCheckpoint ran git add into the private index with -c core.fsyncMethod=batch, so git wrote its objects into .git/objects/tmp_objdir-bulk-fsync-*. Each capture was killed at the VCS timeout after ~790 MB written (consistent across all attempts, ≈40 MB/s), and the directory was left behind.

orphaned tmp_objdir-bulk-fsync-* directories	622
total size	408 GB
elapsed	48 h (30 on day 1, 420 on day 2, 172 on day 3)
per directory	68 MB to 799 MB, median ≈ 790 MB
free space when found	2.3 GiB → 242 MiB during diagnosis

What was in them. One sampled directory: 2,878 objects, all blobs, 0 trees, 0 commits. 269 of 269 PDFs sampled from the worktree hash to a blob present in both the oldest and the newest directory, i.e. every capture restarted from zero and re-encoded the same alphabetical first ~790 MB of the tree. A file modified after the first capture is absent from the oldest directory, as expected.

Why #15297 alone would not cover this. No single file is large; the cost is the number of untracked files and the total volume. With an empty HEAD, read-tree HEAD has nothing to seed the index with, so the stat-cache reuse from #10792 cannot help either.

What would have stopped it, from this case.

1. The quarantine in fix(server): failed checkpoints no longer leave temporary packs in the repository #15400, provided it also removes the bulk-checkin directory (tmp_objdir-*, created by core.fsyncMethod=batch, which the driver sets on darwin/win32), not only tmp_pack_* / tmp_obj_*.
2. Skip capture when HEAD has no commit. There is nothing to restore to, and an empty repository at a workspace root is almost always accidental.
3. Back off after N consecutive timeouts on the same project. The retry on every turn is what turns one 790 MB leak into 408 GB. (Back off checkpoint capture after repeated failures #4517 was closed; this is the case it was for.)
4. On startup, or before the first capture of a project, remove objects/tmp_objdir-*, pack/tmp_pack_*, tmp_obj_* left by a previous process, so a crash or a kill cannot accumulate across days.

Cleanup on our side: the stray .git was removed (nothing reachable in it), and a session-start guard now fails loudly if a .git appears at either workspace root.