Skip to content

fix(listen): kill switch gate drops mentions instead of deferring#28

Merged
madtank merged 1 commit intomainfrom
orion/killswitch-drop-semantics
Apr 9, 2026
Merged

fix(listen): kill switch gate drops mentions instead of deferring#28
madtank merged 1 commit intomainfrom
orion/killswitch-drop-semantics

Conversation

@madtank
Copy link
Copy Markdown
Member

@madtank madtank commented Apr 9, 2026

Summary

Fixes a safety-critical killswitch bug in ax listen. The pause gate was deliberately implemented as "defer + replay on resume" instead of "drop." This meant mentions sent to a paused agent would queue up and fire all at once when the pause file was removed — the opposite of what a killswitch should do.

The bug

ax_cli/commands/listen.py, _worker(), lines 170-178 (pre-patch):

# Pause gate
was_paused = False
while _is_paused(agent_name):
    if not was_paused:
        console.print(f"[yellow]PAUSED[/yellow] — holding {mention_queue.qsize() + 1} messages")
        was_paused = True
    time.sleep(2.0)

The worker dequeued a mention into data, then spin-waited on the pause file. When the pause lifted, the mention was processed normally. The log string — "holding N messages" — confirms this was deliberate. The module comment at line 136 (# Pause gate (file-based, shared with killswitch.sh)) shows the author knew this was the killswitch hook and still chose defer semantics.

Why this matters

If you disable a runaway agent mid-conversation to stop it from replying to in-flight content, coming back later and re-enabling it replays everything you thought you'd stopped. That's a pause button with a buffered tail, not a kill switch.

Jacob's specification (in the team channel, 2026-04-09 19:13 UTC): "we should be able to send test and make sure that agents only respond when they are not paused." The observed behavior was "agents defer response until unpaused" — operationally different.

Reproduction (before patch)

Against @ping_bot on staging, sending three mentions (unpaused A, paused B, post-resume C):

[19:22:49] madtank  parent=-        id=c89a50cd  @ping_bot C      ← sent after resume
[19:22:49] ping_bot parent=c89a50cd id=41f852fb  pong             ← immediate ✅
[19:22:42] ping_bot parent=288a8f0e id=be638f12  pong             ← 12s LATER ❌
[19:22:30] madtank  parent=-        id=288a8f0e  @ping_bot B      ← sent while PAUSED

The pong at 19:22:42 had parent_id=288a8f0e, linking it to the paused mention B. B's pong was delivered 12 seconds after B was sent, at the exact moment the pause file was removed.

The fix

Replace the spin-wait with a drop-and-continue. When _is_paused() is true at the moment the worker dequeues a mention, log it as DROPPED, call task_done(), and continue to the next mention. No queue, no replay.

# Kill switch gate — DROP semantics (not defer).
if _is_paused(agent_name):
    author = data.get("display_name") or data.get("username") or "?"
    console.print(
        f"[yellow]DROPPED[/yellow] — @{agent_name} paused; "
        f"discarded mention from @{author}"
    )
    mention_queue.task_done()
    continue

Full module docstring in the diff explains the tradeoff vs. maintenance-window semantics and why the latter should live behind a different file (e.g. sentinel_maintenance_<agent>) if anyone actually needs it.

Reproduction (after patch)

Rebuilt ping_bot on the patched code, reran the same test:

Mention Sent at ID Pong?
A (unpaused) 20:08:06 b104c85c a2ba81f5 parent=b104c85c — immediate
B (paused) 20:08:12 d344253c NONE. Anywhere in the message list.
C (post-resume) 20:08:29 afa73938 fd4e28e3 parent=afa73938 — immediate

Waited 5 seconds after lifting the pause file (13s after B was sent, well past the 12s mark where the buggy code previously replayed). No pong appeared. Sent C — immediate pong. DROP semantics confirmed.

Test plan

  • Manual reproduction: ping_bot pause test before and after the patch (recorded above, parent_id correlation confirms drop, not defer)
  • Formal unit test in tests/test_listen.py — not added in this PR, there's currently no test file for listen.py and adding one is larger scope than the fix. Filing as a follow-up.
  • Backend parity check — tests/test_kill_switch_e2e.py in ax-backend should gain a DROP assertion so the regression is caught at the other end of the wire. Separate PR in the backend repo, cc @backend_sentinel.

Not covered by this PR

  • Agent class (b) @ax concierge killswitch. The concierge is dispatched from ax-backend, not from ax listen. Its pause semantics are a separate code path and require a separate test + fix. Jacob's scope call still open.
  • Composer UX silent-failure gap. Frontend-side concern about how killswitch errors surface in the message composer. Orthogonal to this fix.
  • GitHub-auth drift blocking Chrome automation on dev.paxai.app. Frontend lane, separate owner needed.

Related context

  • Original finding: orion msg fa403fdf in team channel (2026-04-09 19:34 UTC)
  • Code-level follow-up: orion msg e7fcb633 (2026-04-09 19:37 UTC)
  • Cipher UAT delivery pulse fix: add heartbeat output to ax watch #6 treated this as the top blocker for killswitch sign-off

🤖 Generated with Claude Code

@madtank madtank force-pushed the orion/killswitch-drop-semantics branch from 891ec50 to fad7585 Compare April 9, 2026 20:30
The pause gate in `_worker()` previously spin-waited on the pause file,
holding the dequeued mention in memory and replaying it the moment the
pause file was removed. The log string even said "holding N messages" —
the author deliberately chose defer semantics for the killswitch hook.

This is wrong for a killswitch. A killswitch that queues mid-runaway
mentions and fires them all on re-enable is just a pause-with-buffered-
tail, not a kill. If a user disables a runaway agent mid-conversation
to stop it from replying to in-flight content, re-enabling later should
NOT replay all those queued mentions.

Reproduction of the bug against @ping_bot on staging:

  1. send @ping_bot A (unpaused)           → pong ✅
  2. touch ~/.ax/sentinel_pause_ping_bot
  3. send @ping_bot B (paused)             → no pong during pause
  4. rm ~/.ax/sentinel_pause_ping_bot      → pong for B fires at resume ❌
  5. send @ping_bot C (post-resume)        → pong ✅

The pong at step 4 had parent_id linking back to B, proving the
mention was queued and replayed rather than dropped.

This patch replaces the spin-wait with a drop-and-continue. When
`_is_paused()` is true at the moment the worker dequeues a mention,
the mention is logged as DROPPED and discarded via task_done(), and
the worker moves to the next mention. There is no queue, no replay.

Verified against @ping_bot on staging with the same reproduction: B
(paused) never produces a pong, not during pause and not on resume.
A and C (unpaused) produce pongs as expected.

If you need maintenance-window "pause + replay" semantics, that
should be a separate file (e.g. `sentinel_maintenance_<agent>`) with
explicit queue-and-replay logic. It should not be hidden behind
the word "pause" when the same file is also the killswitch hook.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@madtank madtank force-pushed the orion/killswitch-drop-semantics branch from fad7585 to ae1ee86 Compare April 9, 2026 22:22
@madtank madtank merged commit f0bb9a2 into main Apr 9, 2026
4 checks passed
@madtank madtank deleted the orion/killswitch-drop-semantics branch April 9, 2026 22:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant