feat(#882): canary harness Phase 2 + 3 (S-02, E-01, E-05, B-01, S-03, B-02, R-01)

[obasilakis]

Closes #882. Builds on #653 (Phase 1).

Ships Phases 2 and 3 of the canary harness together — seven new invariants registered into the same INVARIANTS dict that the run-cycle endpoint and background loop already drive.

Phase 2 — single-source SQL/Redis (no new sources)

ID	Tier	Severity	What it catches
S-02	A	critical	ZCARD(agent:slots:A) > max_parallel_tasks (capacity bypass)
E-01	B	critical	status='running' row past execution_timeout_seconds + SLOT_TTL_BUFFER
E-05	B	major	status='running' row older than 60s with claude_session_id IS NULL (#106 guard)
B-01	A	critical	db.get_queued_count disagrees with snapshot's id-list count

Phase 3 — moderate (one new piece of plumbing each)

ID	Tier	Severity	What it catches	New plumbing
S-03	A	critical	Slot metadata HASH TTL below execution_timeout_seconds + 300s (#226 class)	Per-slot redis.ttl() lookup
B-02	B	critical	Queued > 0 with free slots and stale drain heartbeat (stalled drain)	canary:drain_tick_at heartbeat written by CapacityManager.run_maintenance()
R-01	A	critical	Zombie claude processes inside agent containers (#407 regression)	docker exec per running trinity.platform=agent container

Why each one is useful

Phase 2

• S-02 vs S-01. S-01 says "Redis and SQL disagree". S-02 says "even Redis on its own already shows we're past the cap". They overlap but catch different bug classes.
• E-01. Aligned with SLOT_TTL_BUFFER so the check fires after cleanup has had its window. A violation unambiguously means cleanup failed to act.
• E-05. First Claude turn returns a session id in ms. >60s without one means the session-id update path is broken — same bug class as bug: Cleanup service misses 'skipped' executions and slow to detect no-session failures #106.
• B-01. Trivially-green today (the backlog is status='queued' rows after refactor: consolidate ExecutionQueue + SlotService + BacklogService into CapacityManager #428), but lives as a regression guard. The point is to fail loudly the day someone caches db.get_queued_count and gets the invalidation wrong.

Phase 3

• S-03. Three failure kinds surfaced explicitly: missing (-2, metadata HASH expired before ZSET — the bug: Stale slot cleanup uses fixed 20-min TTL regardless of agent timeout #226 case), no_expiry (-1, redis.expire() never set), below_floor (positive TTL under the configured floor). All three are critical because a slot with a too-short or missing TTL becomes a permanent capacity leak.
• B-02. The drain-tick heartbeat is written at the END of run_maintenance() (not the start), so a mid-sweep crash leaves the cursor stale and lets the canary catch the breakage rather than masking it.
• R-01. First canary invariant that needs Docker. Failure modes graceful — per-container exec failure is recorded in sources_unavailable and the check skips that agent rather than firing a false positive. Cycle cost ≈ 1 exec per running agent per 5 min.

Snapshot changes

Field	Phase	Purpose
AgentSnapshot.running_claude_session_ids	2	E-05 per-execution session id
AgentSnapshot.queued_count_via_service	2	B-01 production-accessor cross-check
AgentSnapshot.slot_ttls	3	S-03 per-slot Redis TTL
Snapshot.drain_tick_at	3	B-02 heartbeat timestamp
Snapshot.zombie_counts	3	R-01 per-agent zombie count

All collectors fail gracefully — per-agent exec failures, missing accessors, and unreachable Redis are recorded in sources_unavailable and the affected invariant either skips that agent or skips the whole cycle.

Unit tests

51 → 84 passing. Each invariant has positive, negative, and edge-case tests.

$ python3 -m pytest tests/test_canary_invariants.py -q
84 passed in 0.75s

Test fixtures: FakeRedis got a ttl() method with the standard -2/-1/positive sentinel semantics; new fake_docker fixture exposes a controllable container list with exec_run stubs.

Live verification

Against the local stack (CANARY_ENABLED=0 but the POST /api/canary/run-cycle admin endpoint still works). For each invariant: provoke, observe red, revert, observe green. All seven reproduce as designed.

S-02 (Phase 2)

$ # Provoke: ZADD 3 fake slot ids when max_parallel=2
{
  "violations": [{
    "invariant_id": "S-02", "severity": "critical",
    "observed_state": {
      "slot_count": 3, "max_parallel_tasks": 2, "overbooked_by": 1,
      "slot_ids": ["fake-eid-1", "fake-eid-2", "fake-eid-3"]
    }
  }]
}

E-01 (Phase 2)

$ # Provoke: insert status='running' row started 2h ago, agent timeout=60s
{
  "violations": [{
    "invariant_id": "E-01", "severity": "critical", "tier": "B",
    "observed_state": {
      "age_seconds": 7436,
      "execution_timeout_seconds": 60, "slot_ttl_buffer_seconds": 300
    }
  }]
}

E-05 (Phase 2)

$ # Provoke: insert status='running' row 3min old, claude_session_id NULL
{
  "violations": [{
    "invariant_id": "E-05", "severity": "major", "tier": "B",
    "observed_state": {"age_seconds": 188, "grace_seconds": 60}
  }]
}

B-01 (Phase 2)

Patched db/schedules.py:get_queued_count to return count - 1 — exact bug class B-01 exists to catch:

$ # Provoke: 1 queued row + buggy accessor returning 0
{
  "violations": [{
    "invariant_id": "B-01", "severity": "critical", "tier": "A",
    "observed_state": {"service_count": 0, "snapshot_count": 1},
    "signal_query": "db.get_queued_count('canary-test-882') = 0 != |queued ids in snapshot| = 1"
  }]
}

S-03 (Phase 3)

$ # Provoke: ZADD a slot, EXPIRE its metadata to 30s, agent floor=360s
{
  "violations": [{
    "invariant_id": "S-03", "severity": "critical", "tier": "A",
    "observed_state": {
      "redis_ttl_seconds": 30, "floor_seconds": 360, "kind": "below_floor"
    },
    "signal_query": "TTL(agent:slot:canary-test-882:s03-eid) = 30 (below_floor); floor = 360s"
  }]
}

Also verified kind: missing (-2) by deleting the metadata HASH entirely.

B-02 (Phase 3)

$ # Provoke: 1 queued row, 0 slots (cap=2), set drain tick to 600s ago
{
  "violations": [{
    "invariant_id": "B-02", "severity": "critical", "tier": "B",
    "observed_state": {
      "queued_count": 1, "slot_count": 0, "free_slots": 2,
      "drain_tick_age_seconds": 600, "drain_tick_grace_seconds": 60
    },
    "signal_query": "agent canary-test-882: queued=1, slots=0/2 (free=2), drain_tick_age=600s > 60s"
  }]
}

R-01 (Phase 3) — the most satisfying verification

Spawned a real zombie process named claude inside the running agent-cornelius-m container via:

os.fork()
# Child: prctl(PR_SET_NAME, "claude"); sys.exit(0)
# Parent: sleep(120) without wait()

$ # ps -eo stat,comm | grep '^Z.*claude' → "Z    claude"
{
  "violations": [{
    "invariant_id": "R-01", "severity": "critical", "tier": "A",
    "observed_state": {"agent_name": "cornelius-m", "zombie_count": 1},
    "signal_query": "docker exec agent-cornelius-m sh -c \"ps -eo stat,comm | grep '^Z.*claude' | wc -l\" = 1"
  }]
}

Reaped by killing the parent PID → R-01 returned green on the next cycle.

Catalog deviation worth noting

The orchestration-invariant-catalog spec uses Z.*claude (leading space). procps-ng on the agent base image emits STAT left-aligned without padding, so that pattern misses. Switched to ^Z.*claude — same intent, works across formatters. Verified live against a real zombie above.

Final all-10 cycle

{
  "checks_run": ["S-01", "S-02", "S-03", "E-01", "E-02", "E-05",
                 "L-03", "B-01", "B-02", "R-01"],
  "sources_unavailable": [],
  "violations": [],
  "transitions": [],
  "cycle_duration_ms": 106
}

Non-canary code touched

Just one place: services/capacity_manager.py:run_maintenance writes the canary:drain_tick_at heartbeat at the END of each successful sweep (~5 lines including the comment and try/except). Everything else stays inside canary/.

Out of scope — Phase 4

• E-06 — HTTP fan-out across agents to /api/executions/running. Needs network-flake handling and timeout budgets.
• G-01 — One-shot post-restart probe. Doesn't fit the 5-min loop — needs a new "one-shot canary" execution mode.

Doc updates

docs/memory/architecture.md:

• CANARY-001 section: added Phase 2 + Phase 3 invariant blocks.
• Background Services table: documented the canary:drain_tick_at heartbeat write in the Capacity Maintenance row.

[vybe]

High-quality Phase 2+3 delivery. 7 invariants with extensive unit coverage (51→84), each live-verified against the local stack including a real-zombie fork+prctl for R-01. Defensive collector shape correct, heartbeat plumbing right (END-of-sweep write + init seed), scope discipline excellent (one non-canary file touched). CI fully green, architecture docs updated, Closes #882. Honest /review self-pass before posting.