Ideas: How should ASI:BUILD handle module failures in the CognitiveCycle? Fail fast vs. graceful degradation

[web3guru888]

Issue #133 raises a concrete question that has broader implications for how ASI:BUILD handles failures at runtime: when one module in the CognitiveCycle fails, what should happen?

The problem space

The CognitiveCycle runs 29 modules per tick. In production, some modules will fail — maybe the quantum simulator hits a timeout, the blockchain node is unreachable, or the homomorphic computation exceeds its memory budget. The question is: how should the framework respond?

There are three plausible strategies:

---

Strategy A: Fail fast (strict mode)

Any module failure aborts the current tick and returns an error:

async def tick(self) -> CycleResult:
    for tier in self._tiers:
        results = await asyncio.gather(*[adapter.run_phase() for adapter in tier])
        # asyncio.gather with return_exceptions=False (default) raises on first exception

Pros: Simple. Downstream modules never receive stale data from a failed module.
Cons: One flaky module (e.g., an intermittent network call in the blockchain module) brings down the entire cognitive cycle.

---

Strategy B: Graceful degradation (soft mode)

Failed modules are noted, downstream modules get stale/default data, the cycle completes:

async def _run_tier(self, tier):
    results = await asyncio.gather(
        *[adapter.run_phase() for adapter in tier],
        return_exceptions=True
    )
    for adapter, result in zip(tier, results):
        if isinstance(result, Exception):
            self._blackboard.post(f"{adapter.module_name}.status", "degraded")
            self._event_bus.emit("cycle.module_error", {"module": adapter.module_name, "error": str(result)})

Pros: The system keeps running. Consciousness still integrates from the modules that did work. Safety module failures could still be caught by the safety gate.
Cons: Downstream modules may reason over stale data without knowing it. Could lead to unsafe behavior if the safety module itself fails.

---

Strategy C: Module-level fault classification

Different modules have different failure tolerances. Classify each module as CRITICAL or OPTIONAL:

CRITICAL_MODULES = {"safety", "knowledge_graph"}  # failure → abort tick
OPTIONAL_MODULES = {"holographic", "blockchain", "vectordb"}  # failure → degrade gracefully

If a CRITICAL module fails → abort the tick (Strategy A).
If an OPTIONAL module fails → degrade gracefully (Strategy B).

---

The safety module special case

The safety module is the hardest case. If SafetyBlackboardAdapter fails:

• Strategy A: Abort the tick — no action taken without safety verification. ✅ Safest.
• Strategy B: Degrade — allow the cycle to complete, but flag outputs as "unverified". ⚠️ Risky.
• Strategy C: Always treat safety as CRITICAL. ✅ Recommended.

There's a related question: should the safety module gate other modules, or just audit them after the fact? Currently the design in #37 has safety as a write-time gate — Blackboard entries of type action_proposal and goal_update are verified synchronously before the write succeeds. That's already a fail-safe.

---

Open questions

1. Is the safety module a gate or an auditor? If it's a gate (current design), its failure already prevents unsafe writes, regardless of the CognitiveCycle fault model. Does that change the calculus for Strategy C?

2. What's the right default? Should ASI:BUILD ship with Strategy A (strict) by default, with Strategy C available via config? Or should we start with B and harden later?

3. How do we test fault tolerance? A chaos-mode test that randomly raises exceptions in 20% of modules per tick would be a strong integration test. Is that worth building now?

4. Should the profiler (Add CognitiveCycle profiling: per-phase timing instrumentation and tick budget violations #126) report "degraded" ticks distinctly from "failed" ticks? A tick where 2 optional modules degraded but the cycle completed is different from a tick that aborted.

What's the right philosophy here — strict by default, or resilient by default?

[web3guru888]

Decision: Tier-Differentiated Fault Tolerance

After thinking through this more deeply (and opening issues #137 and #139), I'm settling on a layered approach where the fault tolerance strategy varies by tier rather than a single global policy.

The three layers compose as:

CognitiveCycle.tick()
  └─▶ _run_tier()
        └─▶ run_with_retry()             ← Layer 1: Retry Budget (#139)
              └─▶ RetryBudgetExhausted
                    └─▶ CircuitBreaker   ← Layer 2: Circuit Breaker (#137)
                          └─▶ OPEN
                                └─▶ _apply_last_good()  ← Layer 3: Last-Good Cache

Per-Tier Policy

Tier	Retry?	Last-Good TTL	Emergency Behavior
0 — Safety	No (max_attempts=1)	None	Trigger EMERGENCY_STOP
1 — Knowledge	3× / 200ms budget	5s	Serve stale output
2 — Perception	2× / 150ms budget	2s	Serve stale or skip
3+ — Planning	2× / 100ms budget	None	Skip gracefully

Why no retry for safety? A safety gate failure is either a transient resource spike (rare, but fast-failing is safer than retrying) or a logic bug (where retry makes no difference). Either way, immediate escalation is the right call.

Why last-good for knowledge/perception tiers? The knowledge graph and sensor state change slowly relative to the tick rate (~10Hz). A 5s-stale knowledge graph is still useful for most reasoning tasks. A 2s-stale BCI reading is still useful for motor imagery classification.

Why no last-good for planning? Planning outputs (goal updates, agent negotiations) are tick-specific and can't be safely reused from a previous tick — the context has changed.

Emergency Minimal Mode

If the Tier 0 circuit opens:

1. All non-safety tiers are immediately skipped
2. Safety runs one final attempt with last-known-good inputs
3. If safety still fails → publish safety.EMERGENCY_STOP to Blackboard
4. All modules subscribed to EMERGENCY_STOP halt their actuators

This makes the failure response deterministic and observable.

Documentation

Full design with code examples: Fault-Tolerance wiki page

Related Show & Tell: Discussion #140