Skip to content

feat: correlated-failure / thundering-herd controls (jitter + re-delivery caps + shared-cause pause) #1085

Description

@vybe

Summary

Add correlated-failure controls so the lease-expiry re-delivery primitive is safe at fleet scale: jittered re-poll / reconnect, per-agent and fleet-wide re-delivery rate caps, and a shared-cause pause that halts re-delivery for all agents on a common fault (Claude API outage, expired platform key, a bad skill pushed fleet-wide).

Context

From the 2026-06-05 review's completeness pass. A backend restart (which already happens routinely) means ~200 agents simultaneously re-poll, reconnect, and re-deliver against one database; a shared cause turns the per-agent-benign re-delivery into a self-amplifying retry storm. The candidate designs reasoned per-agent; the 200-agent failures are correlated.

Acceptance Criteria

  • Re-poll / reconnect carries jitter — no synchronized stampede after a backend restart
  • Per-agent and fleet-wide re-delivery rate caps
  • Shared-cause detector pauses re-delivery fleet-wide (Claude API down, platform key expired)
  • Soak gate validated against an induced backend restart with the fleet mid-flight, not just steady state

Technical Notes

  • Relevant once pull lands: the pull endpoints, the lease-reaper in src/backend/services/cleanup_service.py, and the src/backend/services/capacity_manager.py maintenance loop.

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

Labels

complexity-mediumComplexity: medium (board points 5-8)priority-p2Importantstatus-in-devMerged to dev, awaiting release cut to mainstatus-readyGreenlit and ready for development (vetted; counterpart to status-incubating)theme-reliabilityTheme: Reliabilitytype-featureNew functionality

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions