fix(farm): serialize git worktree registry access so a task stops escalating under load (#515)

[SUaDtL]

Closes #515.

AC-1 — the cause

Neither of the issue's leading candidates. Not mock-server contention, not a gate timeout under load: it is a race between concurrent git invocations over shared repository state.

.git/worktrees/ is a shared registry and git mutates it non-atomically. git worktree add scans the existing entries while creating its own, so a second concurrent add can read a sibling's directory after it exists but before its commondir is written.

prepareWorktree ran worktree remove → rm → branch -D → worktree add with no lock, from up to FARM_CONCURRENCY workers at once.

Reproduced at 1 of 3 full coverage runs. #518's diagnostic named it on the first reproduction — exactly what it was built for:

[bounded] bulk4: escalate after 0 attempt(s) — worktree add failed: Preparing worktree (new branch 'farm/bulk4')
fatal: failed to read .git/worktrees/bulk0/commondir: No error

Then measured directly against real git, outside the suite entirely — six concurrent prepare sequences, 40 rounds each way:

CONCURRENT   6 workers x 40 rounds -> 2 rounds had a failed worktree add
    e.g. Preparing worktree (new branch 'farm/bulk3') | fatal: failed to read .git/worktrees/bulk2/commondir: No error
SERIALIZED   6 workers x 40 rounds -> 0

AC-3 — fixed in farm.ts

Worktree creation, removal and prune now run under a dedicated worktreeChain lock.

• Separate from mergeChain deliberately. A merge holds its lock across a git merge plus its gate, which is long; registry calls are short. Sharing one lock would park every worktree setup behind an unrelated merge for no correctness gain. Nothing takes both in the opposite order, so they cannot deadlock.
• prune is the most dangerous of the three to leave unserialized — it deletes sibling metadata directories, which is precisely what a concurrent add is scanning.
• The retry backoff runs outside the lock, so one worker's retry schedule cannot stall the others.

AC-4 — satisfied, not dodged

FARM_CONCURRENCY stays at 6 and the dispatcher's parallelism is unchanged. The worker API call, the gate and the tests — everything that actually takes time — still run concurrently. Only the brief registry mutations queue.

Verification

Post-fix: 4/4 coverage runs green (the mode that reproduced it), full suite 521 passed.

The tests assert the property — no two registry commands ever in flight at once — rather than the absence of the flake. Asserting "the suite went green" would have passed ~95% of the time on the unfixed dispatcher and proved nothing.

4/4 mutants killed: lock-is-a-passthrough, chain-head-never-advances, prepare-drops-the-lock, remove-drops-the-lock.

What mutation testing caught

Three of four mutants survived the first round, including one that removed the lock from prepareWorktree entirely — the function the bug is actually in. Every test I had written exercised removeWorktreeVerified and the lock helper directly, and none of them touched prepareWorktree, which used a module-scope git and so could not be observed at all.

It now takes an injectable runner, defaulted, for the same reason removeWorktreeVerified already did. Two other survivors were genuinely equivalent mutants and are documented as such in the runner rather than papered over with a test; one test that could not distinguish its own mutant was deleted rather than kept.

Line endings — the half with teeth

This commit was blocked by H-09b demanding a crypto-compliance pass, for a concurrency fix that touches no crypto.

An editor pass flipped farm.ts and farm.unit.test.ts to CRLF. Git then read both entire files as added — and the H-09b backstop scans added lines for sensitive patterns. Every pre-existing hash call and every credential-shaped test fixture in those files counted as newly introduced.

README.md was immune, and its diff stayed at 2 lines while farm.ts's was 6165 — because /*.md was already pinned in .gitattributes and the .ts trees were not.

So .gitattributes now pins LF for the TS/JS/JSON plugin and site trees. #533 pinned the workflow tree for the cosmetic half of this; this is the half that fires gates on code nobody wrote. Verified before adding: no committed file under those globs is CRLF today, so it normalizes nothing retroactively.

Version

ca 2.10.0 → 2.10.1. farm.js is a declared shipped artifact, so this is a payload change and the gate landed in #532 requires the manifest to advance — its first real exercise, and it behaved correctly.