Skip to content

Background git polling + stale index.lock fills .git/lfs/tmp (~150 GB/day) in LFS repos #9776

Description

@arthur-albuquerque

Summary

In a repo with large Git LFS files, T3 Code's background git activity (checkpoint git add -A plus frequent git status --porcelain=2 --branch polls) filled .git/lfs/tmp with roughly 150 GB of stranded temp files in about 9 hours. The trigger was a stale .git/index.lock, but T3 kept hammering the repo through the failure all day instead of backing off, and never cleaned up the partial files it left behind.

Environment

  • T3 Code (Alpha), macOS 15.6 (Darwin 24.6.0)
  • git 2.50.1 (Apple), git-lfs 3.7.1
  • Repo with LFS-tracked files up to ~160 MB in the worktree

What happened

  1. A stale, empty .git/index.lock appeared at 08:44 (likely left by an interrupted git process; T3's checkpointing runs git add -A on this repo constantly, so it is the most frequent lock holder).
  2. With the lock in place, git update-index --refresh fails, so git can never cache clean-filter results in the index.
  3. Every T3 poll (git status --porcelain=2 --branch, run every few seconds) therefore re-ran git lfs clean over every stat-dirty LFS file. Each pass wrote fresh multi-hundred-MB temp files into .git/lfs/tmp and then failed at the index write, stranding them.
  4. By evening, .git/lfs/tmp held ~150 GB. After I wiped it, it regrew at roughly 4 GB/hour until I found and removed the stale lock.

Confirmed the parentage with ps: the headless git status and git-lfs filter-process processes were children of T3 Code (Alpha).app/Contents/Resources/app.asar/apps/server/dist/bin.mjs. The temp file sizes matched the repo's large LFS files exactly (159 MB, 144 MB x2, 89 MB x2).

Removing the orphaned lock fixed it immediately: the next git status cached its results and .git/lfs/tmp now stays at 0 bytes, including through checkpoint runs.

Suggested fixes

Any one of these would have contained the damage; together they'd prevent it:

  1. Recover from stale locks. Before polling or checkpointing, if .git/index.lock exists, is old (minutes, not seconds), and no live process holds it (lsof comes back empty), remove it or at least stop polling and surface a warning. Git itself prints "another git process seems to be running" hints for this case.
  2. Back off on repeated failure. The status poll failed the same way for ~9 hours at a few-second cadence. Any error backoff would have cut the damage by orders of magnitude.
  3. Clean up .git/lfs/tmp. Stranded partials there are cheap to detect (files older than a few minutes with no open file handle). Even git-lfs itself doesn't reap them aggressively, so a periodic sweep in repos T3 manages would help.
  4. Consider excluding large LFS files from checkpoint git add -A, or at least rate-limiting checkpoints in repos where the clean filter is expensive. A side effect worth noting: refs/t3/checkpoints/* pin LFS objects, so git lfs prune retains more than users expect.

Happy to provide more detail from the session logs if useful.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions