Ideas: Should CognitiveCycle tiers have asyncio timeouts? Per-tier vs per-module vs asyncio.wait() strategies

[web3guru888]

Discussion #136 (Show & Tell on tier-parallel execution) raised this as an open question. It deserves its own thread.

The problem

Once CognitiveCycle uses asyncio.gather() to run modules concurrently within a tier, what happens if one module hangs? A single slow I/O call — a VectorDB timeout, a blockchain RPC retry, a federated learning round that stalls — blocks the entire asyncio.gather() for that tier. Subsequent tiers cannot start until the gather resolves. If the hang exceeds the tick budget (100ms), the CognitiveCycle misses its real-time constraint.

Option A: per-tier asyncio.wait_for() timeout

Wrap the entire asyncio.gather() in a timeout:

async def _run_tier_with_timeout(self, tier_idx: int, tier: list[str]) -> dict:
    try:
        return await asyncio.wait_for(
            self._run_tier(tier_idx, tier),
            timeout=TIER_BUDGET_MS[tier_idx] / 1000,
        )
    except asyncio.TimeoutError:
        # All tasks in the tier are cancelled
        self._blackboard.publish("cycle.tier_timeout", {"tier": tier_idx})
        return {}  # downstream tiers get stale Blackboard data

Pros: Simple, hard deadline enforced.
Cons: asyncio.wait_for() cancels all tasks in the gather when the timeout fires — even the ones that were about to complete. A slow blockchain module could cancel a fast vectordb result that was 1ms away from finishing.

Option B: per-module asyncio.wait_for() with individual fallbacks

Timeout each module independently:

async def _run_module_with_timeout(self, name: str, budget_ms: float):
    try:
        return await asyncio.wait_for(
            self._adapters[name].run_phase(self._blackboard),
            timeout=budget_ms / 1000,
        )
    except asyncio.TimeoutError:
        self._blackboard.publish("cycle.module_timeout", {"module": name})
        return PhaseResult(module=name, success=False, timed_out=True)

Pros: Slow modules don't cancel fast ones. Each module gets its own budget.
Cons: A module that is consistently timing out still slows the tier to its individual timeout. Need to combine with circuit breaker (#137) to eventually skip it.

Option C: asyncio.wait() with FIRST_EXCEPTION + cancellation

done, pending = await asyncio.wait(
    tasks.values(),
    timeout=TIER_BUDGET_MS[tier_idx] / 1000,
    return_when=asyncio.ALL_COMPLETED,
)
for task in pending:
    task.cancel()
    await asyncio.gather(task, return_exceptions=True)

Pros: Clean cancellation of only the slow tasks. Done tasks are preserved.
Cons: More complex; cancelled tasks should ideally have cancellation cleanup handlers.

Key design questions

1. Should the safety module ever be timed out? If safety hangs, should the tick continue (unsafe) or block until it completes (potentially miss real-time)? Argument for special-casing: safety is Tier 0, so if it times out we arguably should not run Tier 1+ at all.

2. What is the right fallback for a timed-out module? Options: (a) stale Blackboard data from previous tick, (b) a sentinel timed_out=True PhaseResult, (c) skip Blackboard write entirely. How do downstream modules detect stale data?

3. Should tier timeout be the same as sum(individual module budgets) in the tier, or max(individual module budgets)? The point of tier-parallel execution is that the tier should complete in max(T_i), not sum(T_i). So the tier timeout should be roughly max(PHASE_BUDGETS_MS[m] for m in tier) plus a small margin, not the sum.

4. Interaction with circuit breaker (Add circuit breaker pattern to CognitiveCycle: auto-disable persistently failing modules with OPEN/HALF_OPEN/CLOSED health states #137): A module that repeatedly times out should eventually trip its circuit breaker (OPEN state) so it is skipped entirely. The timeout itself should count as a "failure" for the health record.

What is your intuition on the right strategy? Option A (tier-level), Option B (per-module), or Option C (asyncio.wait)?