feat(heartbeat): escalate ccrotate capacity exhaustion as a coalesced issue (PEN-382)

[kkroo]

What

Follow-up to #299. When a ccrotate-capacity scheduled retry exhausts its 24-attempt budget (CCROTATE_CAPACITY_MAX_RETRY_ATTEMPTS), the run is cancelled with a warn lifecycle event — but that's easy to miss in run history. This files an operator-visible recovery issue (priority high, todo) via recoveryService so a pool that never recovers actually gets attention.

Design

• Coalesced per (company, ccrotate target). A real pool outage exhausts many agents' retries at once, so the escalation is keyed on the target (originKind=ccrotate_capacity_exhausted, originId=<target>) — one open issue per pool, not one per cancelled run. Re-uses the established (originKind, originId) open-issue dedup pattern (mirrors the liveness escalations).
• Unassigned (not assigned to an agent): pool exhaustion needs a ccrotate token refresh / capacity top-up, not LLM-agent action — so it's a flag for operators, not a work item.
• Best-effort: escalation is wrapped in try/catch — it never breaks the promotion sweep (the run is already cancelled and the warn event recorded).

Why now

error_code='rate_limit_exhausted' fired ~314× in the last 7 days (pre-#299, the immediate-cancel rate). Post-#299 most of those become silent auto-retries that recover; only genuine multi-hour pool starvation now reaches the 24-attempt cancel — making true exhaustion a rare, low-noise escalation trigger.

Files

• recovery/origins.ts — new ccrotateCapacityExhausted origin kind
• recovery/service.ts — escalateCcrotateCapacityExhausted() (dedup + create)
• heartbeat.ts — call it on the exhaustion path
• __tests__/heartbeat-ccrotate-capacity-retry.test.ts — assert one escalation on exhaustion + coalescing across two agents on the same pool

Tests

pnpm exec vitest run src/__tests__/heartbeat-ccrotate-capacity-retry.test.ts → 6 passed (4 existing + exhaustion-files-escalation + coalescing). Server typecheck clean.

🤖 Generated with Claude Code

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + gstack/codex.

Clean, focused follow-up to #299 (+141/-0, 4 files). The exhaustion path, scope, and best-effort wrapping all check out — one substantive finding about the coalescing guarantee.

Critical Issues (0)

None.

Important Issues (1)

• [gstack/review · code] recovery/service.ts (escalateCcrotateCapacityExhausted, the dedup SELECT … LIMIT 1 ~:4416) — the coalescing this PR sells is not enforced at the DB level, unlike every sibling escalation. The dedup is a bare check-then-create (SELECT open issue by (companyId, originKind, originId) → if none, issuesSvc.create). All five existing recovery origins back that same pattern with a dedicated partial unique index (issues_active_liveness_recovery_incident_uq, …_stale_run_evaluation_uq, …_productivity_review_uq, …_stranded_issue_recovery_uq, issues_open_routine_execution_uq), and recovery/service.ts:~2255 documents why: "The active-recovery uniqueness index prevents concurrent duplicates." This PR adds the ccrotate_capacity_exhausted origin kind but no matching migration/index (the 4 changed files are test/heartbeat/origins/service only — confirmed against the schema at this SHA).
  • Within one promoteDueScheduledRetries call the loop is sequential, so the test's two-call coalescing passes. But the sweep is not globally single-threaded — two overlapping sweeps (multi-replica / re-entrant timer) each win the conditional UPDATE for a different exhausted run targeting the same pool, then both SELECT (neither issue exists yet) and both INSERT → duplicate "ccrotate pool exhausted — " issues. This is most likely under exactly the scenario the PR is built for: a real outage exhausting many agents' retries at once.
  • Bounded and low-frequency (a couple of extra issues during a genuinely rare exhaustion, not a storm) — hence Important, not Critical.
  • Fix: add a migration mirroring issues_active_stranded_issue_recovery_uq for origin_kind = 'ccrotate_capacity_exhausted' (WHERE origin_id IS NOT NULL AND hidden_at IS NULL AND status NOT IN ('done','cancelled')). It composes cleanly with the existing best-effort try/catch in heartbeat.ts:~6418 — a concurrent unique-violation becomes the caught/logged/swallowed case, so no second issue and no sweep breakage.

Suggestions (2)

• [tests] __tests__/heartbeat-ccrotate-capacity-retry.test.ts (the coalesces … test, ~:309) — coverage only exercises the sequential path (two separate promoteDueScheduledRetries calls), which is precisely the path the missing index doesn't affect. It gives false confidence that coalescing holds under the concurrent mass-exhaustion the feature targets. Consider asserting the dedup at the DB level (or noting the concurrency gap) once the unique index lands.
• [code · comments] recovery/service.ts (issue body builder inside escalateCcrotateCapacityExhausted, ~:4440) — the escalation issue has no auto-resolution: it's created todo/unassigned and the description tells the operator to "mark it done once the pool recovers." If nobody closes it, a later genuine outage coalesces onto the stale-but-open issue, attaching a new incident to an old one's history. Defensible for v1 ("represents the whole outage"), but worth a follow-up to auto-close on pool recovery, or a runbook line so operators reliably close it.

Strengths

• Exhaustion path is correctly targeted: escalation fires only when exhausted is non-null, i.e. the promoter that actually flipped this run to cancelled via the guarded conditional UPDATE (status = 'scheduled_retry' AND scheduledRetryAt <= now) — so it's keyed to one real cancel, not every sweep pass.
• Best-effort try/catch around the escalation (heartbeat.ts:~6418) never breaks the promotion sweep; the run is already cancelled and the warn lifecycle event recorded before escalation runs. Right ordering, right failure mode.
• Dedup predicate (isNull(hiddenAt) + notInArray(status, ['done','cancelled']) + originKind + originId) matches the established house pattern exactly; all SQL is parameterized drizzle (no injection surface, no LLM trust boundary).
• Coalescing keyed on (company, ccrotate target) is the right grain for a pool outage; unassigned + todo keeps it out of the assigned-issue dispatch/stranded sweeps, so it won't be mistaken for agent work.

Recommended Action

1. Add the partial unique index migration for ccrotate_capacity_exhausted before merge — it's the one piece bringing this in line with the other five escalations and making the coalescing guarantee real.
2. Suggestions are opportunistic.

[kkroo]

Thanks @ally — addressed the Important finding in be76eb9.

Important #1 (DB-level coalescing): added migration 0106 + schema issues_active_ccrotate_capacity_exhaustion_uq — a partial unique index on (company_id, origin_kind, origin_id) WHERE origin_kind='ccrotate_capacity_exhausted' AND origin_id IS NOT NULL AND hidden_at IS NULL AND status NOT IN ('done','cancelled'), mirroring issues_active_liveness_recovery_incident_uq. Journaled (like 0069 and every real index migration) so it applies via migratePg too. escalateCcrotateCapacityExhausted now catches the 23505 and coalesces onto the winner; the existing best-effort try/catch in heartbeat.ts swallows it → no second issue, no sweep breakage — exactly the compose-cleanly path you described.

Suggestion 1 (DB-level test): added a test asserting the index rejects a concurrent open duplicate (23505 + constraint name) and allows a done duplicate (partial index excludes terminal states), so coverage now exercises the concurrent path, not just the sequential SELECT.

Suggestion 2 (auto-close stale escalation): agreed it's defensible for v1; deferring to a follow-up to auto-close on pool recovery (or a runbook line). Tracking separately.

7/7 ccrotate suite green + server typecheck clean locally.