💡 Agent Pipeline Stall Detector & Automatic Recovery Escalation

[github-actions[bot]]

Summary

Build a scheduled monitor that detects stalled work items across the initiative → dev-lead → PR → review pipeline, distinguishes between "waiting on dependency" and "truly stuck," and triggers recovery actions (retry dispatch, escalation label, or owner notification). Currently, issues silently stall with no visibility until a human notices.

Market Signal

ICLR 2026 research identified five core production problems for multi-agent systems: latency, token costs, error cascades, brittle topologies, and poor observability. Error cascades and poor observability are exactly the failure modes this project experiences. As agentic pipelines grow longer (idea → plan → sub-issues → implementation → review → merge), the probability of a stall at any stage increases multiplicatively.

User Signal

• Bug dev-lead retry doesn't cover failed initial issue implementations (only rate-limited PRs) — issues silently stall #781: dev-lead retry doesn't cover failed initial issue implementations — issues silently stall
• Bug Self-host deadlock: force_review bypasses the advisory gate but not the CI gate, so CI-gate fixes can't merge through ring-0 #619: Self-host deadlock — force_review bypasses advisory gate but not CI gate, creating a stuck merge state
• The fleet monitor ([Fleet Monitor] petry-projects/.github-private — .github/workflows/pr-review-trigger.yml #766, [Fleet Monitor] petry-projects/.github-private — .github/workflows/daily-pr-review-health.yml #765) tracks workflow health but not pipeline-level work item progress
• Multiple fleet-tracker issues show recurring workflow failures that can cascade into stalled work items

Technical Opportunity

The fleet monitor (scripts/fleet_monitor.sh), initiative driver (scripts/initiative-driver.sh), and dev-lead retry (scripts/dev-lead-retry.sh) already implement pieces of this. A pipeline stall detector could query issue/PR state via gh API, compute time-in-state, compare against expected SLAs per stage, and dispatch recovery actions using the existing retry infrastructure.

Assessment

Dimension	Score	Rationale
Feasibility	high	gh-API + jq script; no LLM dependency; leverages existing retry infrastructure
Impact	high	Prevents weeks-long silent stalls; improves pipeline throughput visibility
Urgency	high	Bug #781 is an open, active problem; pipeline volume is increasing with initiatives

Adversarial Review

Strongest objection: A stall detector adds another scheduled workflow that could itself stall or generate false alarms, especially for legitimately paused work (initiative:hold, dev-lead:hands-off).

Rebuttal: The detector must respect existing hold/hands-off signals — it only flags items that lack an explicit hold and exceed their stage SLA. False alarms are low-cost (an informational label or comment) vs. the high cost of weeks-long silent stalls. The detector itself is a simple gh-API + jq script with no LLM dependency, making it robust against the failure modes it monitors.

Suggested Next Step

Define per-stage SLA thresholds (e.g., dev-lead pickup within 24h, PR review within 48h), build a gh API query script that checks time-in-state, and wire it into a daily scheduled workflow.