Skip to content

fix(agent-actions): global kill-switch fails open on a D1 read error #2125

Description

@JSONbored

Parent: #1936

Problem

isGlobalAgentFrozen(env) (src/db/repositories.ts:2066) reads global_agent_controls.frozen inside a try/catch that returns false (not frozen) on any error — a thrown D1 exception, a missing table (a self-host instance that hasn't applied migration 0059 yet), a timeout, or a malformed row. This value feeds globalPaused at all 7 call sites that gate agent actuation (src/services/agent-action-executor.ts:63, src/github/client.ts:543, src/queue/processors.ts:1212/3376/7052, src/mcp/server.ts:2421, src/upstream/ruleset.ts:283, src/services/contributor-issue-draft.ts:263), and an identical fail-open copy exists in the /status health endpoint (src/review/ops.ts's defaultOpsHealthDeps.isFrozen).

The independent env-var backstop (AGENT_ACTIONS_PAUSED) does not auto-engage when the DB read fails, so it does not compensate for this specific failure mode.

Failure scenario: an operator sets global_agent_controls.frozen = 1 to halt the fleet during an incident. If D1 is simultaneously degraded, or a self-host instance hasn't applied migration 0059 yet, every read of the kill-switch throws, is swallowed, and returns false — every in-flight merge/close/approve continues to actuate during the exact window the operator tried to freeze the fleet, with no error surfaced anywhere.

This fail-open is an existing, explicitly-commented tradeoff (favoring fleet availability), not an oversight — but there's currently no way for an operator to distinguish "confirmed unfrozen" from "read failed, assumed unfrozen." That silence is the actual gap.

Requirements

  • A D1 read failure while checking the kill-switch must never be silently indistinguishable from a genuine unfrozen state.
  • The fix must not turn a routine transient D1 blip into a fleet-wide pause — availability still matters for the common case.
  • Must cover both isGlobalAgentFrozen and the duplicate implementation in src/review/ops.ts.

Deliverables

  • Emit a structured warning/audit event (or Sentry capture) from isGlobalAgentFrozen's catch block, distinguishing "confirmed unfrozen" from "read failed, assumed unfrozen."
  • Have the admin/ops route that sets frozen = 1 perform a read-after-write verification and surface an explicit error to the operator if the confirming read fails or doesn't reflect the write — closing the specific incident-window race.
  • Apply the same observability fix to src/review/ops.ts's defaultOpsHealthDeps.isFrozen.
  • Add a unit test asserting isGlobalAgentFrozen emits the new signal on a thrown D1 error (the existing test only asserts the fail-open value, not its observability).

Acceptance criteria

  • A D1 read failure while checking the kill-switch produces a distinguishable, alertable signal instead of silently resolving to "unfrozen."
  • Setting frozen = 1 during a degraded-D1 window either succeeds-and-is-confirmed or surfaces a clear error — it never silently no-ops.
  • A routine, non-incident D1 blip still does not pause the fleet.

Expected outcome

An operator who deliberately halts the fleet during an incident can trust the halt took effect, or gets an immediate, visible signal if it didn't — closing the one scenario where the emergency brake can silently fail exactly when it's needed most.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

Status
Done

Relationships

None yet

Development

No branches or pull requests

Issue actions