Repin: fix the pane root, take the transcript scan off the event loop

[lvwerra]

The reported symptom

The terminal works for 2-3s, freezes for ~5s, then repeats. Input typed during the freeze arrives all at once afterwards, but the cursor keeps blinking and you can switch windows — because the freeze is server-side, not in the browser. While the one event loop is blocked the server can neither drain PTY output nor forward keystrokes; the blinking cursor is a CSS animation and the window switch is your OS.

What it actually was

claudeTranscriptsSince() was a readdirSync per project dir plus a statSync per transcript — fully synchronous — running every REPIN_MS (20s) for the life of every live session. CLAUDE_CONFIG_DIR is on the FUSE bucket (/data/state/claude). Replaying that exact scan against the real directory, 8 times:

one scan: 12 readdirSync + 32 statSync -> 2 recent
8 scans (ms): 2, 3, 3, 3, 3, 77, 1111, 1260

Bimodal: ~3ms when the mount's attribute cache is warm, 1.1-1.3s cold (~37ms per statSync round trip). Each cold scan was a full main-thread block, once per live pane, every 20 seconds.

Confirmed from the outside too. Probing /api/health — which does no I/O, so its latency is block time — gave blocks of 0.9-1.8s at periods of 20.2, 21.3, 21.2, 21.4s. Not 20.0s: the drift equals the block, which is the signature of a setTimeout(tick, N) chain that reschedules after its work. Under load the watchdog logged the same thing as main loop STALLED 5-6s … maxLag=12671ms.

Ruled out along the way: buildTraces (11.6s of wall time but only 26ms of blocking — properly async, and the activity=buildTraces breadcrumb is misleading because tracked() spans awaits); cgroup CPU throttling (nr_throttled flat, load 4.1/8); GC or CPU-bound JS (zero CPU ticks consumed across each block); sync fs on /data generally (p50 0.2ms); stdout pipe backpressure (async on POSIX).

How this got here

Both halves of this PR trace back to specific earlier changes, and the order matters — this is not a regression introduced by something recent.

when	what	effect
5b5da0e 06-29	defines the tmuxName helper (tmux session name per pane)	—
12a7e8c 07-30	Replace tmux with a server-held libghostty grid — deletes that definition	tmux idiom is gone from runner.js
9fb59f5 #17 07-31	conversation pins get a life-of-the-pane watcher on a 20s beat	the freeze: turns a once-per-launch scan into one every 20s per pane
7112a46 #23 08-04	adds paneRootPid(), calling tmuxName and execFileSync	dead on arrival: neither identifier exists in that file

#17 is where the cost comes from. It changed the pin from captured-once-at-launch to re-checked forever, which was the right fix for a real bug (/clear orphaning the pin, so --resume restored the pre-/clear thread). Its own description anticipated exactly this hazard — "this scan now runs for the life of every session against a FUSE mount rather than once per launch" — and memoized transcript heads to bound it. That memoization holds; what it did not cover is the statSync sweep that finds the candidates in the first place, which is the 1.2s measured above.

#23 added the breadcrumb path to fix what the scan cannot do: attribute a new conversation when several live claude panes share a folder. But it introduced paneRootPid() as a tmux list-panes call five days after 12a7e8c deleted tmuxName, and execFileSync is not imported in runner.js either. git log -S tmuxName --all shows the definition added in 5b5da0e, removed in 12a7e8c, and referenced again only by 7112a46; paneRootPid itself first appears in 7112a46. So it has never returned anything but null — it throws a ReferenceError straight into its own bare catch on every tick.

A null pane root makes pidTrusted false for every breadcrumb, which is what the live logs show, invariably:

[claude] agent-manager-4-ba3fbf: breadcrumb rejected (pid not in pane)
[claude] agent-manager-1-f79577: breadcrumb rejected (pid not in pane)

So #23's mechanism has never run, and the bug it was written for — /clear in a shared folder never being followed — is still live today. This PR is the first point at which that mechanism can work at all, which is also why the cheap fix for the freeze (lean on the breadcrumb, scan less) had to be preceded by repairing the thing it leans on.

Three changes

1. The pane root

paneRootPid now reads hosts.get(id).pty.pid: the same semantic tmux's pane_pid had — the process this server spawned for the session, which exec claude replaces in place — as a property read, with no subprocess per tick.

2. The scan is awaited

The scan's cost was never the I/O, it was doing the I/O on the event loop. claudeTranscriptsSince, transcriptHead, transcriptExists and claudeCandidate now await, and the watcher tick is async. Measured against the real bucket directory, same code path either way:

	wall time	event-loop block
sync	1111-1260ms	all of it
async	466ms cold, 5-8ms warm	1ms

Sequential rather than Promise.all on purpose: 32 parallel FUSE stats would saturate the 4-thread libuv pool and push every other fs operation in the process behind them, and nothing is waiting on the result.

3. The scan steps down to a backstop

For any pane whose hook has proven itself — a breadcrumb that named that pane's own conversation, either a re-pin or the resume no-op already pinned — the scan runs once a minute instead of every beat. A crumb rejected for any other reason proves nothing about the pane and does not count. A pane with no working hook keeps the old every-tick cadence, so nothing regresses where the scan is still the only mechanism.

Because the scan no longer blocks, this cadence is not trading staleness against freezing any more; it only limits how often we walk the bucket for an answer the breadcrumb already gave. That is why it is a minute and not the ten first proposed — a minute is also the longest a pin can stay stale if a crumb is ever lost (the hook fired but its crumb never reached us), and the cost of being wrong there is one late Overview digest.

Rearming moved out of tick into one place. An awaited tick that throws surfaces as a rejected promise rather than an exception in a setTimeout callback, so run catches, logs and rearms — dropping the watcher would silently stop following /clear for the rest of the pane's life. Exactly one rearm per tick, so a failure can neither kill the watcher nor arm two timers. The scan also re-reads the pin after awaiting rather than trusting the value read at the top of the tick.

Scope

Not touched: the codex and opencode watchers from #17 run on the same 20s beat, but read local disk (/home/node/local/codex-home, and opencode's SQLite was moved off FUSE in 1dfb753), so they are cheap. firstLine stays synchronous for the same reason — it is only on the codex rollout path. Claude's scan was the one on the bucket.

Testing

server/test/repin.test.mjs — 30 checks, up from 21. Full npm test green (all 5 suites).

• 3 new checks on the pane root, asserted against a real running session rather than a mock, because a mock cannot catch this class of bug. Verified they have teeth: restoring the tmux version fails 2 of them (got false want true). Note it is the live-session check that catches it — paneRootPid('unknown') returns null under both the broken and fixed versions, which is part of why Sessions: let the pane say which conversation it is on #23's own 21 checks passed while the function never worked.
• 6 new checks on the cadence via a pure claudeScanDue(): unchanged every-tick behaviour without a proven hook, no scan on the next beat or at 59s with one, due again at a minute, and a proven pane still scanning once before its first backstop.
• The 5 existing claudeCandidate checks now await it; their assertions are unchanged, so they still cover the launch window, folder scoping and other sessions' pins across the async conversion.
• The async block measurement above was taken by importing the new claudeCandidate and pointing it at the real /data/state/claude while sampling event-loop lag, rather than inferred from the shape of the change.

Not covered by tests: the run wrapper's throw-and-rearm behaviour, which needs a driven watcher loop rather than a pure function. And not verified end-to-end: that a /clear in a shared folder now re-pins via a trusted breadcrumb on the live Space — that needs this deployed, since the running container still has the broken paneRootPid, which is why the log evidence above shows only rejections.

🤖 Generated with Claude Code