Orthogonal wait-lease: stop misreading background-waiting agents as idle/stalled (design + minimal first PR)

[Fewmanism]

Problem

When an agent parks waiting for background work (a watched detached job, a long provider call), its screen goes quiet and herdr reports idle/complete (#1217 — we reproduced this across Claude Code, Codex, Hermes and Grok Build sessions). Operators and stall-detection tooling then misread a healthy wait as a stall; in our fleet this nearly caused a forced teardown of a billing GPU pod that was mid-generation.

Extending the state enum (an awaiting state) or making self-report/screen-detection precedence configurable both looked attractive and both fail review: awaiting is not an exclusive agent state (it overlaps idle/working), precedence knobs exist only because two signals are forced to fight over one slot, and push-style self-reports drop under load (#1049, #1323).

Proposed design: an orthogonal wait-lease

A lease is a time-boxed, verifiable claim that sits next to the display state and never changes it:

herdr pane wait-lease acquire --id <job-ref> --ttl-ms N
herdr pane wait-lease release --id <job-ref>

• Display state and detection stay untouched (100% backward compatible). The lease appears only as an annotation: idle (awaiting: issue150-gen, 12m left), plus a one-shot lease expired 3m ago notice — which is exactly the signal a human needs to distinguish healthy waits from stalls.
• TTL is mandatory — no open-ended waiting claims.
• Binding & races: a lease is bound to (pane, session/generation) so pane reuse can't inherit a stale lease; each lease carries a random token and all writes are conditional on it (an in-flight renew must not resurrect a released lease). Expiry is judged on a monotonic clock written only by the herdr CLI/daemon — clients never hand-write expiry.
• Transport: atomic file write (tmp→rename) under the herdr runtime dir, read on the existing poll tick — no socket push, so the pi integration drops "working" status reports under load (500ms socket timeout, no retry) #1049/OMP integration extension missing socket-report retry (v4) that Pi has (v5); no screen detection manifest for OMP #1323 drop class does not apply.

Renewal must verify the awaited target (not the agent, not a hook)

Agent-event-driven renewal is structurally wrong: while an agent blocks on background work, no events fire — that is the whole reason it is waiting — so leases would expire mid-healthy-wait by default, and the expired-lease display would then relabel the same false stall with higher confidence. And a naive keep-alive sidecar proves only the sidecar's liveness. Renewal should therefore be allowed only when the awaited target itself re-verifies:

herdr pane wait-lease hold --id <job-ref> --ttl-ms N --pid <job-pid>
# herdr owns the renew loop; renews only while pid(+starttime) verifies; auto-release on target death
# observer kinds: pid (local, pid+starttime to survive pid reuse) / provider-job (remote job-id state)
# verification failure => stop renewing, mark lease 'degraded' in display

Consumer contract (normative, not advisory)

A lease must gate suppression of stall alarms, never authorize kills. For destructive/expensive automation (e.g. tearing down billed resources), require all of: (a) lease absent/expired, (b) independent job-side heartbeat stale, (c) a policy time floor. Display state alone, lease state alone, or their AND must not trigger teardown.

Minimal first PR (what we'd propose to land first)

1. wait-lease.json schema + acquire/release only, atomic file, session/generation binding, token-conditional writes, daemon-owned monotonic expiry
2. Annotation in pane get/pane header: active lease remaining, one-shot expired notice
3. Docs for the consumer contract

Deferred to follow-ups: hold/renewal with target verification, compact tab-bar UI, per-agent hooks.

Relation to existing issues

• claude code is setting up the monitor and waiting, but the agent state in the header shows 'complete' instead of 'in-progress' #1217 (this is the structural fix we'd propose), pi integration drops "working" status reports under load (500ms socket timeout, no retry) #1049/OMP integration extension missing socket-report retry (v4) that Pi has (v5); no screen detection manifest for OMP #1323 (why transport is pull/file, not socket push)

We run a multi-agent fleet on herdr daily and are happy to prototype the client side (our job wrapper already tracks job pid + heartbeat, so hold-style renewal maps 1:1), test builds, and iterate on the schema.

---

(Originally filed as #1417; moved here per the issue-tracker policy — issues are for reproducible bugs only.)

[amin-lakhani]

Strong +1 on the problem framing — with field data from 0.7.3, and from shipping the closest approximation of your annotation that today's API allows.

We run a Claude Code fleet on herdr daily and hit this class constantly. Two pieces of evidence that may help evaluate the design:

1. On 0.7.3 the display state is decided by a single screen signal, and nothing agent-side can influence it. herdr agent explain --json during a background wait shows the deciding rule is osc_title_working (priority 1100): a spinner prefix in the pane's OSC title. Claude Code drops the spinner when its turn ends and backgrounded work paints nothing after that, so the pane flips to done ~24s after turn end while the job is still running (timing + traces in my comment on #1217). pane report-agent is accepted-but-ignored since the hooks removal — so the "two signals fighting over one slot" you describe has become one signal that is structurally blind to waits, which supports the orthogonal-annotation shape over precedence knobs.

2. We shipped the nearest available approximation — display-only state labels — and its gaps map exactly onto what the lease adds. Our hook relabels done/idle → ⏳ waiting (bg) since HH:MM via pane report-metadata --state-label with a TTL, cleared when the wait resolves. What we learned:

• Pane reuse inheriting stale display metadata is real. Recycled pane ids kept the prior occupant's labels; we had to add unconditional session-start clears. Your (pane, session/generation) binding addresses precisely the failure we hit.
• Mandatory TTL: agreed — 0.7.3's --ttl-ms expires within ~2s of deadline, a solid base. And we live the renewal problem you call out: during a healthy wait no agent events fire, so nothing refreshes our labels — we settled for a long TTL and accepted mid-wait expiry. Target-verified hold is the right fix; agent-event-driven renewal would indeed re-create the false stall with higher confidence.
• A display tweak can't serve machines. The relabeled word helps a human at a glance, but the status icon still shows done, the completion notification still fires early, and stall tooling has nothing verifiable to read. That's exactly the gap between annotation-as-display and lease-as-verifiable-claim.

Happy to port our hook to wait-lease acquire/release — it would delete most of our workaround.

[llvm-x86]

Agree this is a real problem, and the OSC title-spinner case illustrates why display-layer signals shouldn't be conflated with actual completion. The orthogonal wait-lease + TTL looks like the right shape: it stays evidence-based and doesn't force idle/working into a new enum. My one open question is who acquires the lease for a stock CLI like Claude Code that has no concept of herdr. If the tool itself has to announce "I'm waiting," most CLIs won't today, so the lease sits unused and we're back to inferring wait state from output — the guesswork #1217 is trying to escape. Could the herdr side acquire and renew the lease from signals it already holds (child process alive, no new manifest activity), with a true tool-emitted lease as the stronger signal when available? That way adoption doesn't hinge on every CLI cooperating, and the design avoids leaning on the same title/spinner heuristics it's meant to replace.

[Fewmanism]

Good push — adoption is the real risk here, and you're right that most CLIs will never emit a lease themselves. Having sat with it, I think your two-tier structure is right, with one amendment about what each tier is allowed to authorize.

1. The line I'd draw: not declared-vs-inferred, but what consumers may do with each. "Child process alive + no manifest activity" is a verifiable signal — but it verifies liveness, not progress: it's also the exact signature of a genuinely hung agent. So the amendment: leases carry kind: declared | inferred. An inferred lease adds grace and escalation on top of the UNKNOWN floor (which already bans automated termination of paid resources); it never satisfies "safe to keep billing" and never upgrades authority toward declared. Rejecting inference wholesale would be wrong; consuming it as a safety guarantee would recreate the original misreading with the sign flipped — false-healthy instead of false-stalled.

2. Who acquires the declared lease: the glue layer — and herdr should ship that glue. Tool-launched waits always have a launcher — a wrapper script, a hook — that already knows the job-ref and a sane TTL, and it's two lines away from wait-lease acquire. For Claude Code concretely: its user-level hook system can acquire when a task is backgrounded and release when it ends — zero upstream CLI changes, with the exact hook events pinned down in the PR 1 example. But this is not universal: raw uninstrumented panes and agent-internal waits (long provider calls) never get a declared lease. Two consequences: (a) herdr should ship the hook pair itself (e.g. herdr init installs it) so the declared path is default-on, and PR 1 should include one working example; (b) the uncovered classes are exactly where your inferred tier earns its place.

3. Where the safety actually lives: the consumer contract. Whatever we ship, "quiet + no lease" must be specified as UNKNOWN — distinct from idle — with a contractual floor: no automated termination of paid resources, longer grace, page a human. Without that, annotations are cosmetic.

4. One more adoption point: the TTL makes trust decay by default. The glue layer only guesses one number at acquire time; herdr's clock does the rest, and a wrong guess fails toward UNKNOWN — never toward kill, never toward continued trust. Two things must hold for that, and both belong in PR 1: a herdr-side hard max TTL (each grant of trust is bounded by the cap, never by the declared value — cumulative lifetime is a separate, deferred bound), and a clock spec that survives suspend and restarts — CLOCK_MONOTONIC freezes during laptop suspend while the cloud pod keeps billing, so this wants CLOCK_BOOTTIME / mach_continuous_time semantics plus defined restart behavior (e.g. all leases degrade to UNKNOWN).

I've folded these two PR 1 points, together with the other schema-level consequences (ledger fields, a renew verb, an external authority ceiling), into a separate scope-amendment comment — keeping this thread on the adoption question.

[llvm-x86]

Kind amendment closes this for us — inferred grace/escalation only, never billing-safe, and quiet+no lease = UNKNOWN on the consumer side. No further questions from us here; we'll leave the PR 1 ledger plumbing to the maintainer thread.

[Fewmanism]

PR 1 scope amendments

I've kept stress-testing the proposal since posting it. Same design goal — these amendments close holes that would otherwise freeze into the schema.

Add to PR 1:

1. Server-side hard max TTL. The cap bounds the authority any single acquire or renew can grant — never the declared value; without it, --ttl-ms permits effectively unbounded trust per grant. (Cumulative unattended lifetime is the deferred chain cap's job, below.) The cap MAY later become per-job-type; per-type caps may only tighten the global one, never exceed it.

2. Clock spec. The TTL clock must advance across suspend (CLOCK_BOOTTIME / mach_continuous_time semantics): CLOCK_MONOTONIC freezes while a laptop sleeps, but a cloud pod keeps billing — every lease would silently outlive its TTL by the sleep duration, in the worst possible direction. Across a herdr restart, persisted expiries are only meaningful if bound to and verified against the same boot/clock domain; when that cannot be verified — and always across a reboot — leases degrade to UNKNOWN.

3. renew, shipped together with the cap. Deferring renew while capping TTLs would force any job longer than the cap into declared-but-expired — the original false-stall, reintroduced. Semantically renew is small: re-authorize with the opaque token issued at acquire (so session+generation binding flows through the renewal path and a reused pane cannot renew a stale lease), plus at minimum a fresh observation of the wait target — never bare "my process is still running" — and reset expiry under the same clamps. The renewer is the same glue layer that acquired, driven by its own poll or progress events.

4. Ledger fields — recorded, not acted upon. Per lease: job_type (enforced enum with an escape hatch, not free-form), declared_ttl_ms, ended_at + end_reason: released | expired | orphaned, and a censoring marker (exact | right_censored | unknown) — an expired row proves only "longer than TTL". Two doc-level rules baked into the schema description: declared TTLs are not trustworthy training data (operators inflate them to avoid pages); only measured durations of released leases are (and release ≠ completion — the reason still matters). And no component may read the ledger on a decision path in PR 1.

5. External authority ceiling (forward-compat hook). An operator MAY supply an absolute wall-clock authorized_until on acquire/renew. Effective lease authority never extends past min(requested_expiry, grant_time + hard_max_ttl, authorized_until); past that instant, machine renewal is refused. Absent on acquire, behavior is unchanged. Once supplied, the ceiling is retained across renewals and can never be cleared or raised by machine renewal — only lowered, the lower value taking effect immediately; early revocation is a lowered ceiling, not a separate mechanism. A wall-clock rollback never extends authority: an unverifiable clock discontinuity degrades the lease to UNKNOWN. Expiry of the ceiling revokes authorization only — it does not by itself authorize resource termination. This sentence is the time-irreversible part of this comment: once consumers are built on "lease expiry is the only authority", a second clamp axis becomes a breaking behavioral change.

6. Reason codes in the consumer contract. Authority ends for different reasons with different dispositions but one shared invariant — the machine never terminates paid resources and never self-extends:

   • LEASE_STALE — claim went stale: re-verify → grace → page if unresolved
   • EVIDENCE_MISSING — target progress unproven: refuse renewal
   • CHAIN_CAP_REACHED — unattended-authorization limit hit: anomaly, page immediately
   • OWNER_WINDOW_ENDED — normal close of delegated time: drain and park, no page

   PR 1 discipline: LEASE_STALE and EVIDENCE_MISSING are live from day one; OWNER_WINDOW_ENDED fires only when authorized_until was supplied; CHAIN_CAP_REACHED is reserved for the deferred chain-cap fuse — consumers must handle it, but no PR 1 code path emits it.

   When several fire at once, keep all reasons and apply the strongest disposition; an anomaly is never suppressed by a normal close. Forward-compat rule: a consumer encountering an unknown reason code MUST apply the strongest safe disposition (no kill, no extend, page), and an unknown lease kind MUST be treated as inferred (weakest authority) — new enum values must degrade safely, not break consumers.

Deferred to a follow-up: the per-job-ref-chain absolute unattended-authorization limit (kept distinct from the delegation ceiling — one is a fuse, the other a grant); progress-token escalation (K identical renewals → page, which catches zombies faster than any lifetime bound); a measured-TTL suggester fed by the ledger; multi-lease-per-pane semantics (I'd argue for a set keyed by job-ref, the pane counting as waiting while any valid lease exists).

Deliberately out of herdr: window issuance and its authentication, spend envelopes, admission control, end-of-window ramps, checkpoint dispositions. Those belong to the operator's policy layer; herdr's job ends at the clamp.

One open plumbing question before the schema freezes: how does the pane token reach hooks (env injection, e.g. HERDR_PANE_TOKEN)? Token-conditional writes depend on it.