Parallel steps invoking git collide on index.lock, with none of the tolerance the stash path got

[k0pernikus]

Drafted with the help of Claude Opus.

hk schedules steps in parallel whose fix commands write the git index, and they collide on
.git/index.lock. The contention is hk's own — no external process, no stale lock, nothing planted.

Two things follow, and the second matters even if you disagree with the first:

1. Unlike the stash path, nothing tolerates the contention. #1060
   added a bounded wait because a transient index.lock was judged worth surviving there. A step
   command gets none.
2. When it fires, nothing in the output suggests the remedy, which already exists — one Group per
   git-invoking step serialises them and the collision disappears.

Reproduction

Steps with disjoint globs, each invoking git — here directly, so the call is visible in the config
itself:

["chmod0"] {
  glob = "index_trigger0.txt"
  fix = "bash random_fast_sleep.sh && git update-index --chmod=+x script0.sh"
  stage = List()
}
["chmod1"] {
  glob = "index_trigger1.txt"
  fix = "bash random_fast_sleep.sh && git update-index --chmod=+x script1.sh"
  stage = List()
}
// … 48 of them

Each step sleeps a random 10–149 ms before its git call, so nothing synchronises them; the overlap comes
from scheduling. Staging the triggers and running hk run pre-commit:

  chmod0 – fatal: Unable to create '/tmp/repo/.git/index.lock': File exists.
  chmod0 – Another git process seems to be running in this repository, e.g.
  chmod0 – an editor opened by 'git commit'. Please make sure all processes
  chmod0 – are terminated then try again.
✗ chmod0

hk exits 128, git's own code from the child process — so this is not the stash path, and
run_git_stash is not involved.

Repository, seven scenarios, one CI job each:
k0pernikus/hk-partial-commit-index-lock-repro

Job	Result
48 steps running git update-index	1/5 runs collided on the runner, 5/5 locally
48 steps whose script runs git add	4/5 runs collided
the same 48 steps, one Group each	4/4 clean

Byte-for-byte identical steps in the last two rows — same script, same random delays — differing only in
the grouping. The hit rate tracks core count, which is what a scheduling race should look like.

The same repository pins the behaviours that are already correct, so this is not confused with them: a
lock released inside the 775 ms window recovers, a lock nothing ever releases still aborts (deliberate,
per #1060), and a config with only one git-invoking step showed no collisions in 8 runs — the race
needs two or more.

Why I think the contention is hk's to avoid

• hk chose the concurrency. Step locking keys on the files a step declares. The git index is not a
  declared file, so two steps that both write it look independent — yet for a git hook runner it is the
  resource steps are most likely to share.
• hk already treats the index as exclusive for its own writes. stage = "…" goes through add()
  (src/git.rs:1487) behind hk's Repo mutex and never races. The guarantee hk maintains internally
  simply stops at the process boundary.
• fix(stash): wait for transient index locks #1060 already set the precedent. It added a bounded wait before the shell git stash push
  precisely because a brief index.lock is survivable. The identical contention in a step command gets
  nothing: add() has no retry, and a step's own git has none either, so an overlap of a few
  milliseconds is fatal.

That last point is the crux, and it needs no inference about what any command does. I am not suggesting
you parse command strings — git log writes nothing, a wrapper hides the call, a script conceals it
entirely. Tolerating the contention requires knowing none of that.

What would help

Any of these would help, most useful first:

• Serialise steps that may write the index, the way hk already serialises its own writes. An opt-in step
  field would avoid guessing which ones those are.
• Or tolerate it: a bounded retry when a step command fails on index.lock, on fix(stash): wait for transient index locks #1060's reasoning.
• At minimum, name the remedy in the failure. Today hk forwards git's message verbatim, and git's advice
  does not fit: "Another git process seems to be running" is true, but it is another hk step in the same
  run; "a git process may have crashed … remove the file manually" suggests deleting a healthy lock. So
  the investigation starts with editor integrations and filesystem quirks instead of with the config.

✗ chmod0
  hint: this step ran concurrently with other steps. If more than one step writes the git
  hint: index (`git add`, `git update-index`, or a script that stages its own output),
  hint: serialise them with `depends` or place each in its own group.
  hint: https://hk.jdx.dev/configuration#group

Detecting that needs no inference either — only recognising git's index-lock message after the fact.

From the docs on <GROUP>
(source):

A group is a collection of steps that are executed in parallel, waiting for previous steps/groups to
finish and blocking other steps/groups from starting until it finishes.

The green run above puts one writer in each group. It works, at the cost of a group per writer and full
serialisation rather than just of the git call.

Happy to send a PR for whichever of these you want, in whatever shape you prefer.

Environment

• hk 1.54.0 (behaviour identical on 1.51.0)
• git 2.54.0 (GitHub runner), 2.47.3 (local)
• Linux, stash = "git", jobs default

Related

• #1056 — my earlier report on the stash path
• #1060 — the bounded wait added there
• #247, #234 —
  RFCs on stash/error recovery