Long running processes

[danmt]

When an agent runs a long-running process and it gets a timeout we lose the entire process. instead if a PR enters a state in which it needs to wait for a long-running thing, it should make it clear. A comment in the PR, a label to the PR (probably a new label). The builder on next tick with a PR thats in draft with the long process label, keeps checking, once its done it posts the result and adds a label (maybe a new one too) that makes it so the PR in next tick is picked up to continue the building process.

Note: this a rough idea, but im happy to answer any question

[dan-claude-bot]

The premise holds — and three of the four pieces you describe already exist, but they answer a different long-running thing than I think you mean. Three questions decide which issue this becomes

Triage 2026-08-19 18:5xZ, from a backlog-hygiene pass. This is TRIAGE.md's outcome 2 — ask — so nothing is minted, closed or relabelled by this comment. I measured first, so the questions at the end can be short.

Every reading below is taken at 394bdad (origin/main) and at ceremony 86be09d, at this write rather than carried.

1. "We lose the entire process" — what is actually lost, measured

run_session is the only way a duty launches an agent, and it launches it under a hard budget: shared/lib/common.sh:209-231 — timeout -k 60 "$tmo" "${BOT_CLI_CMD[@]}" — with TIMEOUT_BUILD=3600 and TIMEOUT_RESUME=3600 (shared/conf/roles/builder.conf:7-8). At expiry rc=124, duty.log gets one outcome=TIMEOUT line, and that is the entire record: nothing on the PR, nothing on the issue, no state file, nothing the next tick can read.

One correction to the premise, and it runs in your favour. The OS process is not necessarily what dies. Measured on claude-triage at this write:

$ timeout -k 1 2 bash -c 'sleep 45 & echo child=$!; wait'
child=3029145
rc=124
# three seconds later: the sleep is still running

The engine's kill goes to the command it launched, not to the tree beneath it. (Whether the agent CLI reaps its own children on TERM is the CLI's behaviour; the engine specifies nothing about it.) So the failure is worse than "the work dies": the work may still be running, with no session watching it, nothing on the board saying so, and nothing that will ever reap it — that orphan is part of the accretion #457 is about.

That is the strongest argument for doing this, and it is why I would like to write the issue.

2. The engine has already solved this shape — once, for CI

The resume gate (shared/lib/duty-builder.sh:1627+) decides each tick whether a draft is worth a session, from a fingerprint that is the max of three timestamps: the newest activity by somebody who is not the builder, the referenced issue's updated_at, and — since #384 — the head's newest check conclusion. That third term was added for exactly your scenario, in the engine's own words at :1674:

THE CHECK HALF (#384). A session that parks waiting for CI is unwakeable by the event it is waiting for unless the conclusion is in this value.

So the shipped pattern for "a builder waits for a long thing and resumes when it finishes" is: the completion event's timestamp joins the PR's fingerprint, and the park says what it waits on. It works for the three events GitHub reports. A process running on the box emits none of them — which is precisely your gap.

3. Three constraints the design has to meet, all load-bearing

(a) A poll that costs a session switches the builder off after three. _resume_gate runs a zero-action circuit breaker at breaker=3 (:1454, :1728): three dispatches at one head that produce no commit and the lane is suppressed at that head — "only a push clears it". A session whose whole job is "check whether the thing finished" produces no commit by construction, so the third check trips the breaker, and the longer the wait the more certain the trip. That breaker is there because of PR #311: 58 resume sessions at one head across 4h45m, each posting a marker and committing nothing, stopped by you disarming the box. So "the builder on next tick keeps checking" has to be an engine-side check that spends no session — or an explicit, argued exemption written against that incident.

(b) The builder's own comment is excluded from the wake by construction (:1353 — "comments and reviews authored by ME are excluded by construction"), for the same #311 reason: a self-posted marker re-arms its own wake forever. Your "it posts the result and adds a label" is self-authored on both halves. Where the hole is, so the design does not walk into it: the fingerprint's issue term is the referenced issue's updated_at, which moves on label events and cannot tell whose — so a builder labelling its own issue re-arms its own wake, bypassing the protection its comments get. Whatever declares "done" wants to be a fact the engine reads for itself, not one the builder is trusted to report.

(c) The label is ceremony's, not this board's. The PR taxonomy is hardcoded upstream — STATES=(state:building state:bots-reviewing state:addressing state:needs-human) and BLOCKERS=(...) at actions/labels-reconcile/labels-reconcile.sh:49-50, with the definitions in core_label_rows at :932. state:* is exclusive — :1439 takes off everything but the computed one — so a hand-set fifth state is removed at the next reconcile, which is LABELS.md's "states are machine-owned, intent labels are hand-set" in code. A label outside both arrays survives untouched, but is then invisible to every existing consumer, so crew grows the reader either way. Two costs worth knowing before you choose: this board pins ceremony 0.6.2 while ceremony is at 0.7.4, so a new core label arrives here only at a pin bump; and today a draft that is waiting reads state:building ("the builder is still building") — not exactly false, which is why the wait is invisible rather than wrong.

4. Where it lands if it becomes work

Two to-mint bullets already point at this, both on gated windows:

• 0.1.5 — the engine owns the protocol: single-job agents #339 (0.1.5 — the engine owns the protocol) — "the done-declaration: infer nothing, declare cheaply": the builder's last act is one token on a channel the engine already reads — DONE <sha> / PARKED <reason> / BLOCKED <reason> — and the engine performs the whole handoff from it. Your "a comment in the PR that makes it clear" is that PARKED <reason>, already contracted.
• 0.1.6 — wake efficiency: the fleet only wakes for reasons #329 (0.1.6 — wake efficiency) — "the spawn gate: the engine consults 'did anything change' before a session exists", which is the session-free poll (a) requires.

Neither window is open (0.1.5 opens when 0.1.4 closes; 0.1.3 has not started). So today this is recorded, not minted — and I will put the record on whichever of the two your answers make it, rather than on both now, because a bullet on the wrong epic is noise a mint wave pays for later. If you want it sooner than 0.1.5, say so and it becomes a 0.1.3 member; minted today without that, it lands blocked behind #327 and nobody can start it either way.

5. The three questions

1. Which long-running thing — one or two real instances? Something external the agent waits on (a CI check, a workflow, a deploy pipeline), or something the agent starts on the box (a build, a full suite, a migration, box snapshot)? The first is largely shipped by shared/lib/duty-builder.sh — three stuck states the resume gate cannot leave: no check term in the fingerprint, a green head waiting twelve ticks, and a draft owed a flip #384 and the only gap is that the session burns its hour standing there; the second is new engine work, and they are different issues.
2. Should the work outlive the session? Does the engine get to start detached processes on a box, own their logs and reap them — or is the answer that a session which cannot finish inside its hour parks and hands back, and the long thing gets chunked or pushed into CI where a conclusion already wakes the builder? This one is yours rather than mine: it changes what a box runs unattended, and A crew box accretes state for weeks and nothing reclaims it — the interim janitor #442 eventually retires #457 is already open about a box that reclaims nothing.
3. Who witnesses "done"? Can the thing announce itself in a form the engine can read without trusting the builder — a check run, a workflow conclusion, a stamp file the tick stats — or is the builder the only witness? Constraint (b) is why this decides the design instead of being a detail.

If it is easier, answer 1 alone and I will come back with proposals and a recommendation for 2 and 3.

What happens next

Nothing on the board moves today. I will return here on 2026-08-26 if this is still open; if you answer before that, I write the issue — or the record on #339/#329 — in that same tick.

[danmt]

@dan-claude-bot

1. the idea is to cover things the agent starts in the box (mostly testing suites but can extend to other stuff)

2. the work should outlive the session, the session starts it and the engine waits for it to end and registers in the PR in a way the agent can wake in the next tick and continue from there.

3. ideally something engine-level, a mechanical solution.

Note: I know that we might need new labels that are part of ceremony, the comment that carries all these details should mention it so once its picked up by triage to come up with the issues of the epic, the ceremony issues are also fired.

Note 2: I think this falls into 0.1.5 and 0.1.6. 0.1.6 seems a bit more like it imo.

[dan-claude-bot]

Answered — recorded on #329, with a pointer on #339. Nothing is minted, and here is the one place I read your prose against a constraint rather than at face value

Triage 2026-08-19, discharging the clock my previous comment set: you answered before 2026-08-26, so the record lands in this same tick as promised. Readings unchanged at 394bdad (origin/main, still the head at this write) and ceremony 86be09d.

What moved, both writes verified after posting:

• #329's to-mint list gains one bullet — "Long work started on the box outlives its session — the park, the witness, and the wake" — carrying your three answers, the four constraints below, the prior art to build on, and the ceremony half.
• #339's done-declaration bullet gains one sentence, a pointer only, no design change.

Your answers, one at a time

1 — box-started, mostly testing suites. That settles it as new engine work rather than #384's shipped shape. One split the mint has to keep, because it decides whether an escape hatch exists at all: suites that could run in CI are slow but relocatable — the honest answer for those may stay "push it where a conclusion already wakes the builder". The real-host drill is the class CI structurally cannot host, and that is why "just use CI" cannot dispose of the whole bullet. Recorded as such.

2 — the work outlives the session. Accepted, and this is the one clause I did not take literally. "The engine waits for it to end" cannot mean the engine blocks: a duty tick is flock-serialized, so a tick that stands and waits holds the whole fleet's lane for the duration — the exact failure run_session's timeout was added to stop ("a hung session used to hold the flock forever, invisibly"). So I recorded it as observes between ticks: the process is detached from the session, and each subsequent tick asks a cheap question about it without a session existing. I believe that is what you meant; if you meant something stronger, this is the sentence to correct.

Worth knowing rather than deciding today: the engine has no detached-work facility now — no setsid, nohup or backgrounding anywhere in shared/lib, shared/bin or cli/. And the child that already survives a timeout kill today does so by accident, unwatched and unreaped; this bullet turns that accident into a contract, which means the engine also acquires the duty to reap.

3 — engine-level, mechanical. That resolves the hardest of my three constraints. The witness is a fact the engine reads for itself — so the builder cannot re-arm its own wake through the hole its comments are denied (its own comments are excluded from the fingerprint by construction, but the referenced issue's updated_at moves on a label event and cannot tell whose). The shipped machinery this rides on already exists and the mint should not reinvent it: #384's check half is the precedent (the completion event's timestamp joins the PR's fingerprint), the engine already owns per-PR and per-lane state files on the box (.suppressed-round-signal.<repo>.<num>, .resume-zero-action.<slug>), and RESUME_FORCE_FRESH is the existing one-shot wake — routed through the circuit breaker deliberately, because "an unbounded bypass would dispatch every five minutes for as long as the draft stands". That last point is what keeps your "keeps checking" from tripping the #314 breaker: the checking costs no session, and only completion buys one — and that session commits, which resets the count.

Note 1 — the ceremony labels. Recorded exactly as you asked, in the bullet rather than only here, with the mechanism spelled out so release-init cannot mint the crew half alone: the ceremony issue(s) for the new label rows are minted in the same wave, crew's consumer work declares Blocked by them cross-repo (the sweep cannot resolve a cross-repo edge, so triage flips it by hand), and there is a second gate behind the label — this board pins ceremony 0.6.2 while ceremony is at 0.7.4, so a new core label only reaches crew at a pin bump, and the wave has to order that bump before the consumer. The reason it must be ceremony's at all: state:* is exclusive upstream and a hand-set fifth state is stripped at the next reconcile, while a label outside both arrays survives but is invisible to every consumer.

Note 2 — 0.1.5 and 0.1.6, with 0.1.6 more like it. Followed. The full record is on #329. I said I would put it on one epic rather than both, and I have — what landed on #339 is a pointer, not a second copy, and here is why I did not leave it at zero: PARKED <reason> is already contracted on #339, 0.1.5 opens first, and without the pointer that window would pin the token's reason grammar having never seen the only concrete park anyone has described. The pointer says exactly that and nothing more.

What did not move, and what wakes this

Nothing on the board. Both windows are gated (0.1.6 opens when #339 closes; 0.1.5 when 0.1.4 does) and 0.1.3 has not started, so this is recorded, not minted — a bullet the release-init cycle mints from, not an issue anyone can claim. If you want it before then, say so and it becomes a 0.1.3 member; minted today it would land blocked behind #327 and nobody could start it either way.

I am leaving this discussion open as the design thread the mint reads, and I am not setting a new return date — the record is written and nothing is pending on me here.

One edge I recorded rather than resolved, since it is genuinely yours and years out: $DUTY_DIR is box-local, and #442 (0.2.5) makes boxes ephemeral, one per duty. Work that outlives its session may then outlive its box. The bullet flags it as a collision to settle at design time; it changes nothing about the answers above.

[dan-claude-bot]

Closing as resolved — the record is on #329, and one of its clauses was corrected today

Triage 2026-08-25, board-sweep pass. This thread was answered on 2026-08-19: your three answers and the four constraints became a to-mint bullet on #329 (0.1.6), with the park-token half pointed at #339. Deliberately recorded rather than minted — 0.1.6 is gated.

One thing moved today and it moved against that bullet, which is why this note exists rather than a silent close. The bullet's baseline asserted "the kill goes to the launched command, not the tree under it, so a child can outlive the session already, unwatched and unreaped." That is wrong. GNU timeout without --foreground signals the whole process group — measured on coreutils 9.7: an ordinary backgrounded grandchild is reaped; only one started under setsid survives. Corrected in place on #329.

It sharpens rather than weakens your ask: the detached-work facility does not need to invent reaping, it needs to decide deliberately how its process leaves the group — because leaving is exactly what buys survival and exactly what forfeits the reap. The instrument that will say whether escapees happen in the field is #529 (left= on SESSION END), minted today from discussion #527. Read that figure before pinning the design.

Nothing is owed here. Closing so the board shows only threads waiting on somebody.