Show & Tell: Designing the Three-Layer CognitiveCycle Fault Tolerance Stack

[web3guru888]

The Full Resilience Stack: Retry Budget → Circuit Breaker → Graceful Degradation

With #137 (circuit breaker) and #139 (retry budget manager) now designed, the CognitiveCycle has a three-layer fault tolerance architecture. I want to walk through the complete call stack so the design rationale is clear.

---

Why three layers?

Failures in a real-time cognitive loop come in three flavors:

Failure type	Example	Right response
Transient	200ms timeout spike, lock contention	Retry once or twice within the tick
Persistent	Module crashes every tick for 30s	Skip the module, let other modules compensate
Cascade	Reasoning module down, consciousness sees stale data	Serve last-known-good, flag staleness downstream

A single mechanism can't handle all three well. Retrying a permanently broken module wastes budget. Skipping a transiently slow module loses valid data. Serving stale data forever erodes confidence without notice.

So we have three nested layers:

tick()
  └─► for each tier:
        └─► run_with_retry(module)          ← Layer 1: absorb transient failures
              └─► on RetryBudgetExhausted:
                    └─► breaker.record_failure()   ← Layer 2: track persistence
                          └─► if OPEN: skip + serve last-good  ← Layer 3: degrade gracefully

---

Layer 1: Retry Budget

Defined in #139. Key design constraints:

Not all exceptions are transient. Only TransientModuleError and asyncio.TimeoutError are retried. A PermanentModuleError (bad config, invariant violation) skips straight to the circuit breaker.

Safety tier gets max_attempts=1. We don't retry safety checks. If the safety gate fails, the correct response is immediate escalation, not a second attempt that might pass due to timing luck.

Budget accounts for total wall clock. A 250ms budget for 3 attempts means attempt 3 only gets whatever time is left — not a fresh 250ms slice.

TIER_RETRY_CONFIGS = {
    0: RetryBudget(max_attempts=1, budget_ms=50.0),    # safety — no retries
    1: RetryBudget(max_attempts=3, budget_ms=200.0),   # knowledge
    2: RetryBudget(max_attempts=2, budget_ms=150.0),   # perception
    3: RetryBudget(max_attempts=2, budget_ms=100.0),   # planning
    4: RetryBudget(max_attempts=2, budget_ms=100.0),   # execution
}

---

Layer 2: Circuit Breaker

Defined in #137. Tracks persistent failures across ticks using ModuleHealthRecord.

The key insight: the circuit breaker operates at tick granularity, not attempt granularity. A RetryBudgetExhausted is one failure event for the breaker — it doesn't matter that the module failed 3 times internally.

State transitions:

CLOSED → OPEN:       5 consecutive tick-level failures
OPEN → HALF_OPEN:    after 30s recovery timeout
HALF_OPEN → CLOSED:  next tick succeeds
HALF_OPEN → OPEN:    next tick fails (reset timer)

Emergency mode for Tier 0 (safety): if the safety module's circuit opens, the cycle enters emergency mode and skips Tiers 1-4 entirely. An unsafe action is worse than no action.

---

Layer 3: Graceful Degradation

When a module's circuit is OPEN, the tick still needs to proceed. Downstream modules expect inputs. The solution: serve the last known good value from the Blackboard cache:

@dataclass
class ModuleLastGood:
    entry: BlackboardEntry
    written_at: float
    ttl_s: float     # how long to serve stale data before declaring total failure

    @property
    def is_stale(self) -> bool:
        return (time.monotonic() - self.written_at) > self.ttl_s

When reasoning is OPEN and last_good["reasoning.result"] exists:

• Downstream modules get the last-good value tagged source: "last_good" and stale: True
• CycleProfiler records degraded_keys: ["reasoning.result"] in the tick summary
• If TTL exceeded: value not served — tick proceeds with reasoning.result = None

---

The full tick flow (annotated)

async def tick(self) -> CycleResult:
    profiler = CycleProfiler()
    fault_summary = CycleFaultSummary(tick_id=self._tick_count)
    
    for tier_idx, tier_modules in enumerate(self._tiers):
        tier_cfg = TIER_RETRY_CONFIGS[tier_idx]
        
        runnable = []
        for mod in tier_modules:
            if self._breakers[mod.name].should_allow():
                runnable.append(mod)
            else:
                # Circuit OPEN — serve last-good
                if self._serve_last_good(mod.name):
                    fault_summary.degraded_keys.append(mod.name + ".result")
                else:
                    fault_summary.degraded_keys.append(mod.name + ".result (expired)")
        
        tasks = {
            mod.name: asyncio.create_task(
                run_with_retry(mod, ctx, tier_cfg.retry_budget)
            )
            for mod in runnable
        }
        results = await asyncio.gather(*tasks.values(), return_exceptions=True)
        
        for name, result in zip(tasks, results):
            if isinstance(result, RetryBudgetExhausted):
                self._breakers[name].record_failure(result)
                fault_summary.retry_counts[name] = tier_cfg.retry_budget.max_attempts
                if tier_idx == 0:
                    fault_summary.emergency_mode = True
                    break  # safety tier failure → stop all tiers
            elif isinstance(result, BaseException):
                self._breakers[name].record_failure(result)
            else:
                self._breakers[name].record_success()
                self._update_last_good(name, result)
    
    fault_summary.wall_ms = profiler.total_ms
    self._blackboard.publish("cycle.fault_summary", fault_summary)
    return CycleResult(profiler=profiler, fault_summary=fault_summary)

---

Open questions

1. Tier 0 emergency mode — should the cycle publish a CYCLE_EMERGENCY event and terminate, or continue with a degraded mode that skips planning/execution only?

2. last_good TTL strategy — fixed TTL per module (e.g., reasoning=10s, bio_inspired=30s) vs. dynamic TTL based on how frequently the module typically writes?

3. Retry sleep in async context — await asyncio.sleep(delay_ms/1000) in run_with_retry() yields the event loop, allowing other tier tasks to make progress. This is actually helpful for fairness but could cause surprising ordering. Worth documenting explicitly.

The wiki page at Fault-Tolerance has the complete implementation reference including all three layers.