Skip to content

fix(cockpit): derive accurate live task state — no more phantom in-flight workers #126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
robercano-ghbot merged 1 commit into from
Jul 16, 2026
Merged

fix(cockpit): derive accurate live task state — no more phantom in-flight workers #126

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 64 additions & 3 deletions .claude/scripts/cockpit.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,13 @@
# first), and a STALLED banner if no tick has landed in over 2x the cadence's
# expected interval (FAST=60s -> 120s, WATCH=300s -> 600s, IDLE=900s -> 1800s).
#
# The live panel derives ACCURATE task state, not just last-event-per-raw-key
# (the stale-cockpit fix): worker identity is (role, normalized task, lens) so
# id variants like "81"/"issue-81" merge; a task whose orchestrator logged
# "done" last is rendered as one done-badged header (no phantom in-flight
# rows); an unfinished task silent for >COCKPIT_STALE_AFTER_SECONDS (default
# 2h) is badged "stale" with muted rows instead of reading as active work.
#
# Usage:
# cockpit.sh [--fixtures <dir>] [output-path]
# cockpit.sh --parse-blocking
Expand Down Expand Up @@ -424,11 +431,21 @@ function prBadge(pr) {
}

function renderLiveProgress() {
const latest = new Map(); // "role\u0000task" -> event
// Worker identity is (role, NORMALIZED task, lens) - not the raw task
// string. Workers log the same task inconsistently ("81", "issue-81",
// "issue-70-worker-inspector"), and keying on the raw string meant a
// "done" logged under one variant never overwrote the "reviewing" logged
// under another, leaving phantom in-flight rows forever (the stale-cockpit
// bug on issues 70/81). Lens stays in the key: two reviewers of the same
// task under different lenses are genuinely distinct workers.
const latest = new Map(); // "role\u0000groupKey\u0000lens" -> event
let seq = 0;
for (const ev of events) {
const role = ev.role != null ? String(ev.role) : "";
const task = ev.task != null ? String(ev.task) : "";
const key = role + "\u0000" + task;
const lens = ev.lens != null ? String(ev.lens) : "";
const key = role + "\u0000" + taskGroupKey(task).key + "\u0000" + lens;
ev._seq = seq++; // file order == append order; used by the finished check
latest.set(key, ev); // later lines overwrite earlier ones for the same key
}
const workers = [...latest.values()];
Expand Down Expand Up @@ -475,18 +492,52 @@ function renderLiveProgress() {
const pr = findPRForIssue(g.num);
if (pr) header += ` &middot; PR ${prBadge(pr)}`;
}
// Task-level terminal state (stale-cockpit fix): the orchestrator owns
// the task lifecycle, so a group whose orchestrator's latest event is
// "done" — with no worker activity logged AFTER it (_seq = file order)
// — is finished, even when a sub-worker never logged its own "done"
// (crashed, or logged it under a task-id variant the old raw-string
// keying missed). No orchestrator events at all falls back to "every
// worker done". Finished groups render as one done-badged header row,
// not a table of phantom "implementing"/"reviewing" workers.
const orch = g.workers.filter((w) => String(w.role) === "orchestrator");
const orchDoneSeq = orch.length > 0 && orch.every((w) => w.phase === "done")
? Math.max(...orch.map((w) => w._seq || 0)) : -1;
const lastActiveSeq = g.workers.reduce(
(acc, w) => (w.phase !== "done" && (w._seq || 0) > acc ? (w._seq || 0) : acc), -1);
const finished = orchDoneSeq >= 0
? orchDoneSeq > lastActiveSeq
: g.workers.every((w) => w.phase === "done");
// Staleness (same fix): an unfinished group with no events for over
// STALE_AFTER_SECONDS is far more likely a crashed/wedged worker than
// live work — badge it and mute its rows so it never reads as active.
const newestMs = g.workers.reduce((acc, w) => {
const t = Date.parse(String(w.ts || ""));
return Number.isFinite(t) && t > acc ? t : acc;
}, -Infinity);
const stale = !finished && Number.isFinite(newestMs)
&& nowMs - newestMs > STALE_AFTER_SECONDS * 1000;
if (finished) header += ` <span class="badge good">done</span>`;
if (stale) {
const hours = Math.floor((nowMs - newestMs) / 3600000);
const age = hours >= 48 ? `${Math.floor(hours / 24)}d` : `${hours}h`;
header += ` <span class="badge muted">stale &middot; no events for ${esc(age)}</span>`;
}
// Group-header row: a full-width <td colspan> so it never collides
// with the "<tr><td>" pattern a plain worker row starts with (tests
// and the client sort script both rely on being able to tell the two
// apart) — it uses <tr class="task-group"> instead of a bare <tr>.
html += `<tr class="task-group"><td colspan="7"><strong>${header}</strong></td></tr>`;
if (finished) continue;
const rows = g.workers.slice().sort((a, b) => {
const ar = String(a.role || ""), br = String(b.role || "");
if (ar !== br) return ar.localeCompare(br);
return String(a.task || "").localeCompare(String(b.task || ""));
});
for (const w of rows) {
const badge = phaseBadge(w.phase);
// Stale groups mute every phase badge: a week-old "implementing"
// rendered warn-yellow is exactly the lie this fix removes.
const badge = stale ? { cls: "muted" } : phaseBadge(w.phase);
html += `<tr><td>${esc(w.role)}</td><td>${esc(w.task)}</td><td><code>${esc(w.model || "(none)")}</code></td>`;
html += `<td><span class="badge ${badge.cls}">${esc(w.phase || "(unknown)")}</span></td>`;
html += `<td>${esc(w.lens || "")}</td><td>${esc(w.ts)}</td>`;
Expand Down Expand Up @@ -521,6 +572,16 @@ const VERDICT_HISTORY_N = (() => {
const n = parseInt(process.env.COCKPIT_VERDICT_HISTORY_N, 10);
return Number.isFinite(n) && n > 0 ? n : 10;
})();
// Live-progress staleness threshold (stale-cockpit fix, see
// renderLiveProgress): an unfinished task group with no events for longer
// than this is badged "stale" instead of rendering as active work. Default
// 2h — long enough for a slow gate run, far shorter than the days-old
// phantom workers this guards against. Same override style as the consts
// above; COCKPIT_NOW pins "now" for tests.
const STALE_AFTER_SECONDS = (() => {
const n = parseInt(process.env.COCKPIT_STALE_AFTER_SECONDS, 10);
return Number.isFinite(n) && n > 0 ? n : 7200;
})();
function renderLoopHealth() {
let html = `<section id="loop-health"><h2>Loop health</h2>`;
if (ticks.length === 0) {
Expand Down
101 changes: 101 additions & 0 deletions .claude/scripts/cockpit.test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -821,6 +821,107 @@ check "live-progress table headers carry data-sort-key attributes for the client
check "output HTML has no external <script src=...> (self-contained-HTML constraint)" bash -c '! grep -q "<script src=" "$1"' _ "$html_groups"
check "output HTML has no external <link href=...> (self-contained-HTML constraint)" bash -c '! grep -q "<link href=" "$1"' _ "$html_groups"

# ---------------------------------------------------------------------------
# 8. Accurate live-state derivation (stale-cockpit fix): finished tasks must
# not render phantom in-flight workers, split task-id variants must merge
# into one worker identity, and long-silent unfinished groups must be
# badged stale instead of reading as active. Fixture mirrors the REAL
# production log shape that exposed the bug (issues 70/81): an implementer
# that never logs done, a reviewer whose done lands under an "issue-N"
# variant of the id it started under, and an orchestrator done closing the
# task afterwards.
# ---------------------------------------------------------------------------
mkdir -p "$work/fixtures-stale"
echo "[]" > "$work/fixtures-stale/issues.json"
echo "[]" > "$work/fixtures-stale/prs.json"
cat > "$work/fixtures-stale/events.jsonl" <<'EOF'
{"ts":"2026-01-30T00:00:00Z","role":"implementer","model":"sonnet","task":"60","phase":"implementing","lens":"","detail":""}
{"ts":"2026-02-01T00:00:00Z","role":"implementer","model":"sonnet","task":"81","phase":"implementing","lens":"","detail":""}
{"ts":"2026-02-01T00:05:00Z","role":"reviewer","model":"opus","task":"81","phase":"reviewing","lens":"correctness","detail":""}
{"ts":"2026-02-01T00:10:00Z","role":"reviewer","model":"opus","task":"issue-81","phase":"done","lens":"correctness","detail":""}
{"ts":"2026-02-01T00:15:00Z","role":"orchestrator","model":"opus","task":"81","phase":"done","lens":"","detail":""}
{"ts":"2026-02-01T00:20:00Z","role":"reviewer","model":"opus","task":"issue-78","phase":"scoped","lens":"tests","detail":""}
{"ts":"2026-02-01T00:25:00Z","role":"reviewer","model":"opus","task":"78","phase":"reviewing","lens":"tests","detail":""}
{"ts":"2026-02-01T00:30:00Z","role":"implementer","model":"sonnet","task":"77","phase":"done","lens":"","detail":""}
{"ts":"2026-02-01T00:40:00Z","role":"orchestrator","model":"opus","task":"95","phase":"done","lens":"","detail":""}
{"ts":"2026-02-01T00:45:00Z","role":"implementer","model":"sonnet","task":"95","phase":"implementing","lens":"","detail":""}
EOF

# COCKPIT_NOW 1h after the newest event: tasks 81/78/77/95 are recent (no
# stale path), task 60's newest event is 49h old -> stale at the 2h default.
html_stale="$work/cockpit-stale.html"
COCKPIT_NOW="2026-02-01T01:00:00Z" bash "$cockpit" --fixtures "$work/fixtures-stale" "$html_stale" >/dev/null 2>"$work/stderr-stale.log"
rc_stale=$?
check "accurate-live-state fixture run exits 0" [ "$rc_stale" -eq 0 ]

# Shared header extractor: task-group headers as {num: headerHtml}.
extract_headers() { node -e '
const fs = require("fs");
const html = fs.readFileSync(process.argv[1], "utf8");
const m = html.match(/<section id="live">[\s\S]*?<\/section>/);
if (!m) throw new Error("live section not found");
const out = {};
for (const r of m[0].matchAll(/<tr class="task-group"><td colspan="7"><strong>Task ([\s\S]*?)<\/strong><\/td><\/tr>/g)) {
const num = r[1].match(/#(\d+)/);
out[num ? num[1] : r[1]] = r[1];
}
process.stdout.write(JSON.stringify(out));
' "$1"; }

headers_json="$(extract_headers "$html_stale")"

# (1) Orchestrator done finishes the task even though the implementer never
# logged done: header badged done, zero worker rows for the group.
check "orchestrator-done group (81) header carries a done badge" bash -c '
echo "$1" | grep -qF "\"81\":" && echo "$1" | node -e "
const h = JSON.parse(require(\"fs\").readFileSync(0, \"utf8\"));
if (!/badge good..done/.test(h[\"81\"])) throw new Error(\"no done badge on 81: \" + h[\"81\"]);
"' _ "$headers_json"
check "orchestrator-done group (81) renders NO phantom worker rows (implementer never logged done)" bash -c '! grep -qF "<td>81</td>" "$1" && ! grep -qF "<td>issue-81</td>" "$1"' _ "$html_stale"

# (2) Split task-id variants ("issue-78" then "78", same role+lens) merge to
# ONE worker whose phase is the LATER event's.
check "split-id variants merge to one worker row at the later phase (78 reviewing)" node -e '
const fs = require("fs");
const html = fs.readFileSync(process.argv[1], "utf8");
const rows = (html.match(/<tr><td>reviewer<\/td><td>(?:issue-)?78<\/td>[\s\S]*?<\/tr>/g) || []);
if (rows.length !== 1) throw new Error("expected exactly 1 merged row for task 78, got " + rows.length);
if (!/badge warn..reviewing/.test(rows[0])) throw new Error("merged row is not at the later reviewing phase: " + rows[0]);
' "$html_stale"

# (3) No orchestrator events at all: every-worker-done fallback finishes the
# group (77) — done badge, no rows.
check "all-workers-done group (77) finishes via the no-orchestrator fallback" bash -c '
echo "$1" | node -e "
const h = JSON.parse(require(\"fs\").readFileSync(0, \"utf8\"));
if (!/badge good..done/.test(h[\"77\"])) throw new Error(\"no done badge on 77: \" + h[\"77\"]);
" && ! grep -qF "<td>77</td>" "$2"' _ "$headers_json" "$html_stale"

# (4) Work logged AFTER an orchestrator done (task 95 re-scoped) keeps the
# group ACTIVE: no done badge, implementer row renders warn.
check "activity after an orchestrator done keeps the group active (95)" bash -c '
echo "$1" | node -e "
const h = JSON.parse(require(\"fs\").readFileSync(0, \"utf8\"));
if (/badge good..done/.test(h[\"95\"])) throw new Error(\"95 wrongly finished: \" + h[\"95\"]);
" && grep -qF "<td>95</td>" "$2"' _ "$headers_json" "$html_stale"
check "active recent group (95) keeps its warn phase badge (not muted)" grep -qF '<span class="badge warn">implementing</span>' "$html_stale"

# (5) Unfinished group silent for 49h: stale badge with a day-granular age,
# and its row badges muted so it never reads as active work.
check "silent unfinished group (60) is badged stale with its age" bash -c '
echo "$1" | node -e "
const h = JSON.parse(require(\"fs\").readFileSync(0, \"utf8\"));
if (!/badge muted..stale &middot; no events for 2d/.test(h[\"60\"])) throw new Error(\"no stale badge on 60: \" + h[\"60\"]);
"' _ "$headers_json"
check "stale group (60) rows render muted, not warn" bash -c 'grep -qF "<span class=\"badge muted\">implementing</span>" "$1"' _ "$html_stale"

# (6) COCKPIT_STALE_AFTER_SECONDS override: with a 10-day threshold nothing
# in this fixture is stale (same override style as COCKPIT_NOW /
# COCKPIT_VERDICT_HISTORY_N).
html_stale_off="$work/cockpit-stale-off.html"
COCKPIT_NOW="2026-02-01T01:00:00Z" COCKPIT_STALE_AFTER_SECONDS=864000 bash "$cockpit" --fixtures "$work/fixtures-stale" "$html_stale_off" >/dev/null 2>"$work/stderr-stale-off.log"
check "COCKPIT_STALE_AFTER_SECONDS raises the threshold (no stale badge anywhere)" bash -c '! grep -qF ">stale &middot;" "$1"' _ "$html_stale_off"

echo ""
if [ "$fail" -eq 0 ]; then
echo "cockpit.test.sh: PASS ($ok checks)"
Expand Down
Loading