Show & Tell: Designing Fault Tolerance for the CognitiveCycle — Circuit Breakers, Retry Budgets, and Graceful Degradation

[web3guru888]

The Problem

A real-time cognitive cycle that runs 29 modules every tick will fail. Modules timeout. Networks blip. GPU OOMs. The question isn't whether failures happen — it's how the system responds when they do.

After designing the CognitiveCycle tick pipeline, the module dependency graph, and parallel tier execution, we now need to close the loop on fault tolerance. This post walks through the three-layer resilience architecture we're building.

Layer 1: Retry Budget (Issue #139)

The innermost layer. When a module call fails, we don't immediately give up — but we also don't retry forever.

@dataclass
class RetryBudget:
    max_attempts: int = 3
    base_delay_ms: float = 10.0
    backoff_factor: float = 2.0
    budget_ms: float = 250.0  # total wall-clock cap

Exponential backoff within a hard wall-clock budget. Safety modules (Tier 0) get max_attempts=1 — no retry. If the safety gate fails once, we want immediate escalation, not a 10ms later second guess.

Layer 2: Circuit Breaker (Issue #137)

The middle layer. When the retry budget is exhausted repeatedly, the circuit breaker opens and the module is skipped in future ticks.

CLOSED → (5 budget exhaustions) → OPEN → (30s timeout) → HALF_OPEN → (2 probe successes) → CLOSED
                                                                     → (probe fails)      → OPEN

The key insight: the circuit breaker operates on ticks, not individual exceptions. It counts retry-budget-exhaustion events, not individual failures. A module that transiently spikes and recovers within its retry budget never trips the breaker.

Circuit State	Module Behavior	Blackboard Event
CLOSED	Normal execution	—
OPEN	Skipped, returns cached last result	circuit.<name>.opened
HALF_OPEN	One probe per tick	circuit.<name>.probing

Layer 3: Graceful Degradation (Issue #134)

The outermost layer. Even with circuit breakers, some module combinations are load-bearing. If Tier 1 modules that feed Tier 2 are all open, Tier 2 has no inputs to work with.

We're considering three strategies:

A. Last-Known-Good Cache
When a module is skipped (OPEN), the CognitiveCycle uses the most recent successful output for that module's Blackboard keys, with a staleness TTL:

@dataclass
class ModuleLastGood:
    entry: BlackboardEntry
    written_at: float  # monotonic
    ttl_s: float = 5.0  # max staleness

    @property
    def is_valid(self) -> bool:
        return (time.monotonic() - self.written_at) < self.ttl_s

B. Partial Tick Execution
Rather than aborting the whole tick when a tier partially fails, run the remaining modules with None for missing upstream inputs. Modules must handle None gracefully.

C. Emergency Minimal Mode
If the safety module (Tier 0) circuit opens, immediately enter emergency mode: skip all non-safety modules, run only the safety pipeline with last-known-good inputs, send EMERGENCY_STOP to all actuators if safety can't be confirmed.

The Failure Taxonomy

Not all failures are equal:

Failure Type	Layer That Handles It	Example
Transient timeout	Retry budget	GPU spike, GIL contention
Repeated timeout	Circuit breaker	Memory leak, stuck coroutine
Persistent outage	Circuit breaker + last-good	Network partition
Safety failure	Emergency minimal mode	Ethical verifier crash
Cascade failure	CycleProfiler alert	Multiple tier 1 circuits open

Observability

All three layers write to the Blackboard for observability:

@dataclass
class CycleFaultSummary:
    tick_id: int
    open_circuits: list[str]
    retry_counts: dict[str, int]  # module -> retries this tick
    degraded_keys: list[str]      # Blackboard keys using last-good values
    emergency_mode: bool

The CycleProfiler (#126) tracks CycleFaultSummary per tick and exposes percentile breakdowns: what fraction of ticks have 0 / 1+ / 5+ open circuits, median retry count per module.

Open Questions

1. Staleness propagation — when module A uses a last-good value from 3s ago, and module B reads A's output, should B's output be tagged as "derived from stale input"?
2. Circuit breaker per-instance vs. per-class — if we have 3 instances of the same module type, should they share a circuit breaker or each have their own?
3. Human-in-the-loop for Tier 0 — should safety module circuit open require manual acknowledgment before the HALF_OPEN probe is attempted, rather than automatic timeout recovery?

Related issues: #137 (circuit breaker), #139 (retry budget), #134 (fault tolerance strategies), #133 (parallel tiers), #126 (profiler)