Summary
skipped is the harness's catch-all for anything that is not a completed run. A crash, a total failure of every task source, and a genuinely idle poll all produce the same signal — outcome=skipped in Prometheus, and in the crash case no runs row at all. Nothing on the dashboard or in Grafana distinguishes "healthy, nothing to do" from "broken".
This is not hypothetical. It has masked two separate incidents, one of them ten days long.
Three defects are covered here. None of them is fixed by the clean-checkout change in #61 — a fresh clone is still a network operation that can fail, and two of the three have nothing to do with repo prep at all.
Incident 1 — the crash that hid for 10 days
06:00:03Z run start: repo=rssrn/labro
06:00:07Z issue #59 created by proactive_improvement.fetch()
06:00:09Z ERROR labro.repo - Command failed (exit 1): git -C /data/repos/labro pull
Your configuration specifies to merge with the ref 'refs/heads/feat/publish-gate'
from the remote, but no such ref was fetched.
Traceback → cli.py:404 prepare_repo → repo.py:130 _run → CalledProcessError
(process exits; nothing after this line runs)
No comment on the issue, no runs row, no label, and a Grafana metric reading skipped. Discovered ten days later only because someone asked why #59 had no agent comment.
Defect 1 — non-atomic issue creation
task_sources/proactive_improvement.py:214 creates the GitHub issue inside fetch(), during picking, before prepare_repo runs:
item_number, item_url = _create_issue(project.repo, title, body)
Any failure between issue creation and the agent invocation orphans a real issue on a public repo. #59 had zero comments and no ai-failed label, so it was indistinguishable from a queued item. Worse, it counted against the open-suggestion cap: every labro-proactive run for the following ten days logged
proactive-improvement: skipping rssrn/labro — 1 open suggestion(s) >= cap 1
so one orphaned issue dead-locked the whole task source. That dead-lock, not the crash, was the actual ten-day outage. gh-dependabot-alert creates issues the same way and has the same exposure.
Options. (a) Reorder so repo prep happens before issue creation — cleanest, but prepare_repo runs in the run loop and issue creation in the task source, so this moves a boundary. (b) Keep the ordering and add compensating cleanup: on any failure before the agent runs, close the issue or apply ai-failed so it stops blocking the cap. Option (b) is smaller and also covers agent-invocation failures, not just repo-prep ones.
Defect 2 — crashes are unrecorded and mis-reported
logger.write_run() is the only INSERT INTO runs, and it is never reached when prepare_repo raises — so a crashed run leaves no database row at all. The dashboard cannot show what it has no row for.
Separately, cli.py:404 is unguarded, so the exception escapes to main(). The finally block at cli.py:626-637 still runs and pushes metrics:
metrics_mod.push_run(
project=project_name,
outcome=_run_outcome, # still "skipped" — the default from line 288
...
)_run_outcome is initialised to "skipped" (line 288) and only reassigned at lines 315, 354, 374 and 600 — all after the prepare_repo call. A hard crash is therefore reported to Grafana as skipped, identical to a healthy idle run.
Incident 2 — every task source failed, reported as a normal idle run
Same masking, different code path, and this one recurs. On 2026-08-28 at 03:15 all five sources failed and the run was recorded as an ordinary skip:
03:15:03 run start: repo=rssrn/labro
03:15:04 WARNING gh-dependabot-alert: failed to fetch alerts for rssrn/labro — unexpected end of JSON input
03:15:04 WARNING picker: skipped: source error — gh-author
03:15:05 WARNING picker: skipped: source error — gh-label
03:15:05 WARNING picker: skipped: source error — gh-label
03:15:06 WARNING picker: skipped: source error — gh-author
03:15:06 picker: no task found across 5 source(s)
03:15:06 run skipped: no eligible task found for project 'labro'
The four source error lines are gh api ... returned non-zero exit status 1 tracebacks. An earlier occurrence on 2026-07-20 was a GitHub HTTP 503 that hit all three projects at once. Six crashes of this class appear in labro.log.
Defect 3 — total source failure is indistinguishable from an idle poll
picker catches per-source exceptions, logs a WARNING, and continues. That degradation behaviour is correct — one broken source should not sink a run. But when every source fails, the picker still reports no task found across 5 source(s) and the run reports skipped, which is the same output as a healthy poll with nothing to do.
Suggested behaviour: if all sources errored (or if any did, with a distinct outcome), record a non-skipped outcome and write a runs row. A run that examined zero sources successfully has not established that there is no work.
Why this went unnoticed for so long
Repo prep runs on 49 of 1,690 run starts (2.9%) — most runs skip at the picker and never touch the working copy. The broken working copy was therefore a landmine rather than a continuous failure: it crashed once, on 2026-08-17, and would have crashed again only on the next run that actually picked a task. Low-frequency code paths plus a catch-all skipped outcome is what makes these incidents invisible for weeks.
Suggested fix
- Wrap the
prepare_repo call (and ideally the whole run body) so an unexpected exception produces _run_outcome = "failure" and a write_run row with the traceback in the summary.
- Introduce outcome values that separate harness faults from agent failures and from genuine idleness — e.g.
crashed / source_error vs skipped.
- Ensure a
runs row is written for every run that starts, whatever happens after.
- Once crashes and source failures land in the DB, alerts become possible: "no successful run for project X in N days", and "all sources errored".
Acceptance
- A forced
prepare_repo failure produces: a runs row with a non-success outcome, a metric that is not skipped, and either no orphaned issue or an orphaned issue clearly marked as failed.
- A run in which every task source raises does not report
skipped, and leaves a runs row.
- The proactive cap cannot be dead-locked by a run that never reached the agent.
Context
Diagnosed while investigating why #59 never received an agent comment. The root cause of the git pull failure itself was stale branch.main.merge = refs/heads/feat/publish-gate config left in the reused working copy by an agent — the working copy has since been repaired by re-cloning, and #61 removes the reuse that allowed the residue to accumulate.
Related: #61 (clean checkout per run), #62 (remove WIP preservation).
Summary
skippedis the harness's catch-all for anything that is not a completed run. A crash, a total failure of every task source, and a genuinely idle poll all produce the same signal —outcome=skippedin Prometheus, and in the crash case norunsrow at all. Nothing on the dashboard or in Grafana distinguishes "healthy, nothing to do" from "broken".This is not hypothetical. It has masked two separate incidents, one of them ten days long.
Three defects are covered here. None of them is fixed by the clean-checkout change in #61 — a fresh clone is still a network operation that can fail, and two of the three have nothing to do with repo prep at all.
Incident 1 — the crash that hid for 10 days
No comment on the issue, no
runsrow, no label, and a Grafana metric readingskipped. Discovered ten days later only because someone asked why #59 had no agent comment.Defect 1 — non-atomic issue creation
task_sources/proactive_improvement.py:214creates the GitHub issue insidefetch(), during picking, beforeprepare_reporuns:Any failure between issue creation and the agent invocation orphans a real issue on a public repo. #59 had zero comments and no
ai-failedlabel, so it was indistinguishable from a queued item. Worse, it counted against the open-suggestion cap: everylabro-proactiverun for the following ten days loggedso one orphaned issue dead-locked the whole task source. That dead-lock, not the crash, was the actual ten-day outage.
gh-dependabot-alertcreates issues the same way and has the same exposure.Options. (a) Reorder so repo prep happens before issue creation — cleanest, but
prepare_reporuns in the run loop and issue creation in the task source, so this moves a boundary. (b) Keep the ordering and add compensating cleanup: on any failure before the agent runs, close the issue or applyai-failedso it stops blocking the cap. Option (b) is smaller and also covers agent-invocation failures, not just repo-prep ones.Defect 2 — crashes are unrecorded and mis-reported
logger.write_run()is the onlyINSERT INTO runs, and it is never reached whenprepare_reporaises — so a crashed run leaves no database row at all. The dashboard cannot show what it has no row for.Separately,
cli.py:404is unguarded, so the exception escapes tomain(). Thefinallyblock atcli.py:626-637still runs and pushes metrics:_run_outcomeis initialised to"skipped"(line 288) and only reassigned at lines 315, 354, 374 and 600 — all after theprepare_repocall. A hard crash is therefore reported to Grafana asskipped, identical to a healthy idle run.Incident 2 — every task source failed, reported as a normal idle run
Same masking, different code path, and this one recurs. On 2026-08-28 at 03:15 all five sources failed and the run was recorded as an ordinary skip:
The four
source errorlines aregh api ... returned non-zero exit status 1tracebacks. An earlier occurrence on 2026-07-20 was a GitHubHTTP 503that hit all three projects at once. Six crashes of this class appear inlabro.log.Defect 3 — total source failure is indistinguishable from an idle poll
pickercatches per-source exceptions, logs a WARNING, and continues. That degradation behaviour is correct — one broken source should not sink a run. But when every source fails, the picker still reportsno task found across 5 source(s)and the run reportsskipped, which is the same output as a healthy poll with nothing to do.Suggested behaviour: if all sources errored (or if any did, with a distinct outcome), record a non-
skippedoutcome and write arunsrow. A run that examined zero sources successfully has not established that there is no work.Why this went unnoticed for so long
Repo prep runs on 49 of 1,690 run starts (2.9%) — most runs skip at the picker and never touch the working copy. The broken working copy was therefore a landmine rather than a continuous failure: it crashed once, on 2026-08-17, and would have crashed again only on the next run that actually picked a task. Low-frequency code paths plus a catch-all
skippedoutcome is what makes these incidents invisible for weeks.Suggested fix
prepare_repocall (and ideally the whole run body) so an unexpected exception produces_run_outcome = "failure"and awrite_runrow with the traceback in the summary.crashed/source_errorvsskipped.runsrow is written for every run that starts, whatever happens after.Acceptance
prepare_repofailure produces: arunsrow with a non-success outcome, a metric that is notskipped, and either no orphaned issue or an orphaned issue clearly marked as failed.skipped, and leaves arunsrow.Context
Diagnosed while investigating why #59 never received an agent comment. The root cause of the
git pullfailure itself was stalebranch.main.merge = refs/heads/feat/publish-gateconfig left in the reused working copy by an agent — the working copy has since been repaired by re-cloning, and #61 removes the reuse that allowed the residue to accumulate.Related: #61 (clean checkout per run), #62 (remove WIP preservation).