Agent Circuit Breaker on Abnormal Loss, De-Peg & Oscillation

[devsimze]

Problem Statement

The agent loop (src/agent/loop.ts) rebalances on a fixed hourly cadence with no notion of "conditions are abnormal — stop moving money." If a protocol's reported APY spikes because of a data glitch, if a stablecoin the platform treats as $1 de-pegs, if positions are taking a coordinated drawdown, or if the same batch has rebalanced repeatedly in a short window (flip-flopping between two protocols), the agent keeps trading into the anomaly, paying fees and compounding the damage. There is no kill switch short of stopping the whole process. This issue adds an agent circuit breaker: a set of pre-trade guards that trip on abnormal loss, de-peg, oscillation, or stale data, halt rebalancing (globally, per-protocol, or per-user), emit alerts and events, and require an explicit, audited reset — while leaving user-initiated withdrawals untouched.

Current State

• src/agent/loop.ts — rebalanceCheckJob runs 0 * * * *; groups positions into byProtocolAndStrategy batches; calls executeRebalanceIfNeeded per batch; on success publishes agent.rebalanced / portfolio.updated. On exception it sets lastError and determineHealthStatus() returns degraded — but the next tick still runs.
• src/agent/router.ts — compareProtocols already computes netImprovement after estimateRebalanceCosts and only fires when netImprovement > minimumImprovement. No history-aware guard (oscillation, drawdown).
• src/agent/scanner.ts — scanAllProtocols / getCurrentOnChainApy feed the comparison; a bad scan is only logger.warn'd.
• prisma/schema.prisma — ProtocolRate, ProtocolRiskScore, YieldSnapshot, Position.currentValue, AgentLog. OutboxOp has isUserHalted semantics already (src/outbox/service.ts — a "frozen user" concept exists for compliance halts, per AML Transaction Monitoring & Sanctions Screening Pipeline #321).
• src/services/alerting.ts — alertingService.emit({ severity, component, ... }) operator alerting.
• deploy/monitoring/prometheus/alert-rules.yaml, docs/OBSERVABILITY.md, docs/RUNBOOK.md.

Proposed Solution

1. Breaker model & scopes

enum BreakerScope { GLOBAL PROTOCOL USER }
enum BreakerState { CLOSED OPEN HALF_OPEN }

model AgentCircuitBreaker {
  id           String       @id @default(uuid())
  scope        BreakerScope
  scopeKey     String       // "" for GLOBAL, protocolName, or userId
  state        BreakerState @default(CLOSED)
  trippedRule  String?      // "abnormal_loss" | "depeg" | "oscillation" | "stale_data" | "manual"
  trippedAt    DateTime?
  detail       Json?        // the measurements that tripped it
  resetBy      String?      // admin identity on manual reset
  resetAt      DateTime?
  autoResetAt  DateTime?    // earliest time HALF_OPEN is allowed
  updatedAt    DateTime     @updatedAt
  @@unique([scope, scopeKey])
}

• Evaluation order in rebalanceCheckJob, before any batch executes: GLOBAL → PROTOCOL(from/to) → USER. An OPEN breaker at any applicable scope skips the affected batches (logged as BLOCKED with the breaker id — ties into the explainable-rebalance record).

2. Trip rules (src/agent/breakerRules.ts, new — pure, unit-tested)

• abnormal_loss: aggregate mark-to-market drawdown across the scope's active positions over a trailing window exceeds BREAKER_LOSS_PCT (e.g. portfolio down >X% in Y hours), computed from YieldSnapshot/Position.currentValue series using the robust conventions the risk-analytics stack already uses (period returns, not the smoothed cumulative column).
• depeg: a stablecoin the platform prices at $1 deviates beyond BREAKER_DEPEG_BPS from $1 on the DEX (uses the fee-oracle/routing price path or an oracle feed). Trips PROTOCOL breakers for every protocol holding that asset and blocks conversions into it.
• oscillation: the same batchKey has produced ≥ BREAKER_MAX_FLIPS rebalances within BREAKER_FLIP_WINDOW (detects A→B→A→B fee-burning). Reads recent RebalanceDecision/AgentLog history.
• stale_data: scanAllProtocols returned data older than BREAKER_STALE_MINUTES, or a scan failed N consecutive times — the agent must not trade on a stale APY table.
• Each rule returns { tripped: boolean, detail }; thresholds are config with sane defaults; rules are individually toggleable.

3. State transitions

• CLOSED → OPEN on any rule tripping; sets trippedRule, detail, autoResetAt = now + BREAKER_COOLDOWN.
• OPEN → HALF_OPEN automatically once autoResetAt passes and the tripping rule no longer evaluates true.
• HALF_OPEN → CLOSED after one clean evaluation cycle with no trips; HALF_OPEN → OPEN immediately if any rule trips again (cooldown doubles, capped).
• manual trips and resets: admin-only, always allowed, always audited.

4. Alerts, events, API

• On trip: alertingService.emit({ severity: 'critical', component: 'agent', ... }) + a agent.circuit_breaker_tripped event to affected users (USER/GLOBAL scope) with a plain-language reason.
• On reset: agent.circuit_breaker_reset event.
• GET /api/v1/admin/agent/breakers — list/inspect; POST /api/v1/admin/agent/breakers — manual trip; POST /api/v1/admin/agent/breakers/:id/reset — manual reset (requires a reason). All admin-scoped + in the admin audit log.
• GET /api/v1/agent/status (existing getAgentStatus) gains a breakers: { global, affectingYou } summary for the calling user.
• Prometheus: gauge of open breakers by scope/rule; alert rule for "GLOBAL breaker open > N minutes".

5. What the breaker does NOT stop

• User-initiated withdrawals (CRITICAL outbox ops) always proceed — the breaker only halts agent-initiated rebalances. This is explicit in code and tested.
• Snapshotting, scanning, metrics, and event delivery continue.

Edge Cases & Failure Modes

• Breaker evaluation itself fails (DB error): fail closed for GLOBAL (skip rebalancing that tick, alert) — never fail open and trade blind.
• De-peg recovery: the depeg breaker must require the price to be back within band for a sustained period (N consecutive checks), not a single tick, before HALF_OPEN.
• Legitimate market-wide drawdown: abnormal_loss will trip in a real crash — that is intended (stop churning fees while everything is falling); the runbook documents operator judgment for manual reset, and withdrawals are unaffected so users are never trapped.
• Oscillation false positive from a genuinely improving-then-reversing APY: the flip counter only counts rebalances that each individually passed the net-improvement threshold; document that this is a fee-protection heuristic, not a correctness guarantee.
• Per-user breaker interaction with compliance freeze (AML Transaction Monitoring & Sanctions Screening Pipeline #321): they are independent; a user can be breaker-open but not frozen and vice versa. The rebalance loop checks both.
• Cooldown doubling overflow: cap at BREAKER_MAX_COOLDOWN.
• Config change while OPEN: lowering a threshold must not auto-close an open breaker; only the transition rules do.

Security & Privacy Considerations

• Only admins can manually trip/reset; every manual action requires a reason and is audit-logged with identity.
• User-facing breaker info is limited to "agent rebalancing is paused for you / globally, reason: " — no other users' loss figures or thresholds.
• Trip detail blobs may contain position values — admin-scoped only; the user event carries a sanitized reason string.
• The breaker cannot be used to prevent a user withdrawing — enforced and tested.

Out of Scope

• Automated de-risking / moving to a safe asset when a breaker trips (this issue only halts; a follow-up could add a "flight to safety" action, gated by Approval Workflows & Multi-Signature Governance for High-Value Transactions #314 approvals).
• Market-data oracle selection/aggregation beyond what routing/fee-oracle already provides.
• Circuit breaking for non-agent flows (fiat, referral) — those have their own guards.

Suggested Implementation Plan

1. Schema: AgentCircuitBreaker + enums + migration/rollback.
2. src/agent/breakerRules.ts — the four pure rules + exhaustive unit tests (trip / no-trip / recovery).
3. Breaker evaluation + scope resolution in rebalanceCheckJob, before batch execution; fail-closed on error; BLOCKED decision records.
4. State machine (CLOSED/OPEN/HALF_OPEN, cooldown backoff) + unit tests.
5. Alerts, user events, admin endpoints, getAgentStatus summary, Prometheus gauge + alert rule.
6. Test: withdrawals proceed with a GLOBAL breaker open. Docs: docs/RUNBOOK.md operator procedures, docs/OBSERVABILITY.md.

Acceptance Criteria

• GLOBAL / PROTOCOL / USER breakers evaluated before any rebalance batch; an open breaker skips affected batches and records a BLOCKED decision
• Trip rules implemented as pure, unit-tested functions: abnormal_loss, depeg, oscillation, stale_data — each individually configurable
• CLOSED→OPEN→HALF_OPEN→CLOSED state machine with cooldown backoff; auto-reset only when the tripping rule has cleared for a sustained period
• Manual trip/reset is admin-only, requires a reason, and is audit-logged; breaker evaluation fails closed on error
• agent.circuit_breaker_tripped / _reset events to affected users with plain-language reasons; critical operator alert on trip
• User-initiated withdrawals are unaffected by any open breaker (explicit test)
• Admin endpoints + getAgentStatus breaker summary + Prometheus gauge/alert; docs/RUNBOOK.md + docs/OBSERVABILITY.md updated; tests green

[Adeiza15]

@Adeiza15 has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Been working for several hours and met this also what if you assign the job to me, so I can sought it also

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Adeiza15 to this issue.

[0xApana]

@0xApana has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Hi, maintainer i'll like to work on this.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @0xApana to this issue.

[BernardOnuh]

@BernardOnuh has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Hi, I've taken the time to go through the requirements and I'm confident I can put together a solution that satisfies the acceptance criteria. I'd be grateful if this could be assigned to me.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @BernardOnuh to this issue.

[drips-wave]

Congratulations, @BernardOnuh! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on August 31, 2026.

🧑‍💻 @BernardOnuh: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊

[grantfox-oss]

🎉 This issue has been marked as completed on GrantFox!

@BernardOnuh's PR #416 was approved and merged by @Abidoyesimze.

🏆 @BernardOnuh: You earned 40 FoxPoints for this contribution! Your current tier: Builder (1,172 total points). Track your full progress on GrantFox.

👏 Great work, @BernardOnuh! Keep contributing to Neurowealth.