Show & Tell: Designing CycleFaultSummary — Per-Tick Health Records, SSE Streaming, and MCP Introspection

[web3guru888]

Following the fault tolerance design in #142 and the health dashboard ideas in #143, here's a complete design walkthrough for the CycleFaultSummary system proposed in #144.

---

The Problem

A CognitiveCycle tick involves up to 29 modules running across 5 parallel tiers. Things that can go wrong:

• A module raises a transient exception → retried via Add retry budget manager: per-module retry limits with exponential backoff inside CognitiveCycle ticks #139's retry budget
• A module exceeds its tier timeout → marked degraded, last-good value used
• A circuit breaker opens (Add circuit breaker pattern to CognitiveCycle: auto-disable persistently failing modules with OPEN/HALF_OPEN/CLOSED health states #137) → module offline for the tick window
• A phase exceeds its tick budget (Add CognitiveCycle profiling: per-phase timing instrumentation and tick budget violations #126) → logged as a budget violation

Right now, all of this happens silently inside tick(). There's no structured output, no way to ask "what went wrong last tick?", and no way to stream health events to an operator dashboard.

CycleFaultSummary solves this.

---

Data Model

@dataclass
class ModuleFaultRecord:
    module_id: str
    severity: FaultSeverity     # TRANSIENT / DEGRADED / CRITICAL / RECOVERED
    error_type: str
    error_message: str
    tick_number: int
    retry_count: int = 0
    using_fallback: bool = False
    circuit_state: str = "CLOSED"  # mirrors CircuitBreaker.state

@dataclass
class CycleFaultSummary:
    tick_number: int
    wall_clock_ms: float
    modules_ok: list[str]
    modules_degraded: list[str]
    modules_critical: list[str]
    faults: list[ModuleFaultRecord]
    budget_violations: list[str]
    overall_severity: FaultSeverity

Every tick() call returns a CycleFaultSummary. Zero faults = severity: RECOVERED. One open circuit = severity: CRITICAL.

---

The SSE Stream

The /cycle/health/stream endpoint streams one Server-Sent Event per tick:

event: recovered
id: 4821
data: {"tick":4821,"wall_ms":58.3,"ok":29,"degraded":[],"critical":[],"violations":[],"severity":"recovered"}

event: degraded
id: 4822
data: {"tick":4822,"wall_ms":91.2,"ok":27,"degraded":["holographic","bci"],"critical":[],"violations":["holographic"],"severity":"degraded"}

event: critical
id: 4823
data: {"tick":4823,"wall_ms":62.1,"ok":27,"degraded":[],"critical":["quantum"],"violations":[],"severity":"critical"}

Clients can filter by event name:

const es = new EventSource("/cycle/health/stream");
es.addEventListener("critical", e => alert(`Circuit open: ${JSON.parse(e.data).critical}`));
es.addEventListener("ping", () => {});  // keep-alive, ignore

---

Three Consumers

1. Operator dashboard

Connects to SSE stream, renders a live module health grid. Green = ok, yellow = degraded (fallback), red = critical (circuit open). Budget violation badges on slow modules.

2. MCP introspection tool

@mcp_tool(name="get_cycle_health")
async def get_cycle_health() -> dict:
    """Get the latest CognitiveCycle fault summary."""
    latest = await cognitive_cycle.latest_fault_summary()
    return latest.to_sse_payload() if latest else {"status": "not_started"}

Allows any MCP client (Claude, LangGraph, etc.) to query live cycle health.

3. Blackboard subscribers

Each CycleFaultSummary is posted to the Blackboard as entry_type="cycle_fault_summary". The Safety module subscribes and can escalate to emergency stop if modules_critical includes safety-relevant modules.

---

Implementation Order

1. ModuleFaultRecord + CycleFaultSummary dataclasses (no deps)
2. Integrate into CognitiveCycle.tick() (depends on Add parallel tier execution to CognitiveCycle: asyncio.gather() for dependency-independent modules #133/Add circuit breaker pattern to CognitiveCycle: auto-disable persistently failing modules with OPEN/HALF_OPEN/CLOSED health states #137/Add retry budget manager: per-module retry limits with exponential backoff inside CognitiveCycle ticks #139)
3. Blackboard post on each tick (depends on Wire consciousness module to Cognitive Blackboard (ConsciousnessBlackboardAdapter) #116)
4. /cycle/health/stream SSE endpoint (depends on 2)
5. get_cycle_health MCP tool (depends on 4)
6. Operator dashboard HTML (depends on 4)

---

Open Questions

1. Retention: should we keep a rolling buffer of the last N CycleFaultSummary records in memory? How many ticks is useful without being a memory leak?
2. Alerting: should the system emit a GitHub-style webhook when overall_severity transitions from degraded → critical? Or is SSE sufficient?
3. Safety gate: if modules_critical contains "safety", should CognitiveCycle automatically pause ticking and wait for manual reset? Or is that too aggressive?

See #144 for the implementation issue and acceptance criteria.