Context
fleet_dispatch.py already tracks repeated failures on an issue and eventually marks it needs_human, and my-director/escalate.py forwards needs_human blockers to a human. What's missing: why it kept failing. A model retrying a genuinely hard issue tends to fail differently each attempt (different wrong approach, different error). Failing with the identical error signature N times in a row is a much stronger, purely mechanical signal that points at the harness's own code — a tool bug, a bad prompt template, a broken pre-work step — not the model guessing badly on a hard problem.
Right now both cases collapse into the same needs_human bucket, so a human has to manually figure out which kind of failure they're looking at every time.
Proposal
- Add a signature check over the last N
run ledger entries for an issue: if their recorded error (or failure class) is set-equal across all N attempts, that's a harness-bug signal, not a model-variance signal.
- When it fires, file (or have
my-director escalate) a distinctly-labeled issue quoting the shared error, at the point needs_human currently triggers — same escalation path, different label/priority so a human immediately knows "the tool is broken" vs "this task is hard."
- Rank/surface harness-bug escalations above ordinary
needs_human items — if the tool itself is broken, every other in-flight attempt using it is suspect until it's fixed.
This is a small addition on top of existing retry-tracking (fleet_dispatch.py) and escalation (my-director), not a new subsystem.
Context
fleet_dispatch.pyalready tracks repeated failures on an issue and eventually marks itneeds_human, andmy-director/escalate.pyforwardsneeds_humanblockers to a human. What's missing: why it kept failing. A model retrying a genuinely hard issue tends to fail differently each attempt (different wrong approach, different error). Failing with the identical error signature N times in a row is a much stronger, purely mechanical signal that points at the harness's own code — a tool bug, a bad prompt template, a broken pre-work step — not the model guessing badly on a hard problem.Right now both cases collapse into the same
needs_humanbucket, so a human has to manually figure out which kind of failure they're looking at every time.Proposal
runledger entries for an issue: if their recorded error (or failure class) is set-equal across all N attempts, that's a harness-bug signal, not a model-variance signal.my-directorescalate) a distinctly-labeled issue quoting the shared error, at the pointneeds_humancurrently triggers — same escalation path, different label/priority so a human immediately knows "the tool is broken" vs "this task is hard."needs_humanitems — if the tool itself is broken, every other in-flight attempt using it is suspect until it's fixed.This is a small addition on top of existing retry-tracking (
fleet_dispatch.py) and escalation (my-director), not a new subsystem.