Skip to content

fix: close recovered ccrotate capacity escalations - #542

Merged
kkroo merged 2 commits into
masterfrom
pen-407-ccrotate-escalation-autoclose
Jun 30, 2026
Merged

fix: close recovered ccrotate capacity escalations#542
kkroo merged 2 commits into
masterfrom
pen-407-ccrotate-escalation-autoclose

Conversation

@allyblockcast

@allyblockcast allyblockcast Bot commented Jun 30, 2026

Copy link
Copy Markdown

Thinking Path

  • Paperclip is the control plane for managing AI-agent companies and their execution lifecycle.
  • The heartbeat scheduler owns retry promotion, quota-aware wake deferral, and recovery surfaces for failed autonomous runs.
  • PEN-382 / PR feat(heartbeat): escalate ccrotate capacity exhaustion as a coalesced issue (PEN-382) #307 added coalesced ccrotate_capacity_exhausted issues when a pool never recovers within the retry budget.
  • The remaining gap was lifecycle cleanup: once capacity recovered, the open escalation stayed todo and kept coalescing later outages onto stale history.
  • This pull request closes the open recovery issue when a scheduled ccrotate capacity retry successfully promotes after the gate allows again.
  • The benefit is that recovered outages self-heal, the partial unique index stops blocking fresh future incidents, and operators see a durable recovery note.

Linked Issues or Issue Description

What Changed

  • Added closeRecoveredCcrotateCapacityEscalations() to the recovery service for open ccrotate_capacity_exhausted issues scoped by (company, target).
  • Called the close helper when a ccrotate_capacity scheduled retry re-checks the gate and capacity has returned.
  • Added a focused regression assertion that recovered capacity marks the stale escalation done, adds a recovery comment, and permits a later fresh outage issue for the same pool.

Verification

  • pnpm run preflight:workspace-links && pnpm exec vitest run server/src/__tests__/heartbeat-ccrotate-capacity-retry.test.ts (7 passed)
  • pnpm --filter @paperclipai/server typecheck
  • git diff --check

Risks

  • Low risk. The close path is restricted to ccrotate_capacity_exhausted issues with matching company and target, and it is best-effort so a close failure cannot block retry promotion.
  • If an older scheduled retry lacks ccrotateTarget, the code falls back to the adapter-to-target mapping and skips closing if no target can be determined.

Model Used

  • OpenAI GPT-5.5 (openai/gpt-5.5) with tool use for code inspection, editing, git, tests, and GitHub/Paperclip operations.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • If this change affects the UI, I have included before/after screenshots
  • I have updated relevant documentation to reflect my changes
  • I have considered and documented any risks above
  • All Paperclip CI gates are green
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups
  • I will address all Greptile and reviewer comments before requesting merge

Search added 2 commits June 30, 2026 02:01
Signed-off-by: Search <search@example.com>
Signed-off-by: Search <search@example.com>
@allyblockcast

allyblockcast Bot commented Jun 30, 2026

Copy link
Copy Markdown
Author

Follow-up pushed for PEN-407 after review wake.

  • Commit: c8638d9cb55445bce7c024562b41ced683e5f2c8
  • Change: guarded the recovered ccrotate escalation close with the full (company, originKind, target, open) predicate in the UPDATE, so a row changed after the initial scan is not closed accidentally.
  • Local verification: pnpm exec vitest run server/src/__tests__/heartbeat-ccrotate-capacity-retry.test.ts (7 passed), pnpm --filter @paperclipai/server typecheck, git diff --check.
  • Note: gh pr view is unauthenticated in the workspace and GitHub API tools showed no visible review threads/comments, so I addressed the concrete race-risk found during review of the touched code.

@kkroo
kkroo merged commit d1409b9 into master Jun 30, 2026
15 checks passed
@kkroo
kkroo deleted the pen-407-ccrotate-escalation-autoclose branch June 30, 2026 03:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant