Q&A: My CognitiveCycle tick is exceeding 100ms — how do I diagnose which module is slow?

[web3guru888]

Question

I'm running the CognitiveCycle and seeing occasional ticks that take 180ms+ (well over the 100ms target). The logs show cycle.module_error events but I can't tell which tier or module is the bottleneck. What's the right debugging workflow?

---

Answer

There are three layers of instrumentation you can use, from quickest to most thorough.

Step 1: Read the CycleFaultSummary

Every tick writes a CycleFaultSummary to the Blackboard (Issue #140). Start here:

# Stream fault summaries in real-time
import asyncio
from asi_build.integration import CognitiveCycle, CognitiveBlackboard

async def monitor():
    bb = CognitiveBlackboard()
    async for event in bb.subscribe("cycle.fault_summary"):
        s = event.payload
        if not s.is_healthy:
            print(f"Tick {s.tick_id}: {s.status}")
            print(f"  open circuits: {s.open_circuits}")
            print(f"  degraded keys: {s.degraded_keys}")
            print(f"  retry counts:  {s.retry_counts}")
            print(f"  tier timings:  {s.tier_wall_ms}")
            print(f"  total: {s.wall_ms:.1f}ms")

asyncio.run(monitor())

Or use the SSE endpoint:

curl -N http://localhost:8080/cycle/health/stream

Step 2: Inspect TierTiming from the CycleProfiler

The CycleProfiler (Issue #126) records per-tier and per-module timing. If you see one tier taking much longer than expected, drill down:

# Access profiler from CycleResult
result = await cycle.tick()
for tier_idx, tier_timing in result.profiler.tier_timings.items():
    print(f"Tier {tier_idx}: {tier_timing.total_wall_ms:.1f}ms")
    for module, attempts in tier_timing.module_timings.items():
        for attempt in attempts:
            print(f"  {module} attempt {attempt.attempt}: {attempt.duration_ms:.1f}ms ({attempt.outcome})")

Common patterns:

• One module in Tier 2 takes 80ms+: That module is your bottleneck. Check if it's doing blocking I/O.
• All modules in a tier take 60ms: The tier itself may be CPU-bound — check for GIL contention.
• Multiple retry attempts visible: A transient failure is eating into the tick budget (retry budget from Add retry budget manager: per-module retry limits with exponential backoff inside CognitiveCycle ticks #139).

Step 3: Check circuit breaker state

If a module is consistently slow, it may already be in HALF_OPEN state (recovering) and the probe is slow:

for name, breaker in cycle.breakers.items():
    if breaker.state != "CLOSED":
        print(f"{name}: {breaker.state}")
        print(f"  consecutive failures: {breaker.consecutive_failures}")
        print(f"  last failure: {breaker.last_failure_time:.1f}s ago")

Step 4: Bisect module loading

If you can't reproduce with the full 29-module stack, use the minimal module subset guide (Discussion #135) to isolate which modules are needed for your workload. A 5-module tick should complete in <20ms on most hardware.

---

Common root causes

Symptom	Likely cause	Fix
Tier 2 consistently >60ms	reasoning module doing synchronous file I/O	Wrap in asyncio.to_thread()
Intermittent spikes after long runs	Memory allocation pressure → GC pause	Enable GC metrics in bench_cycle.py
Tier 0 >30ms	safety formal_verify taking too long	Reduce proof depth or add timeout
All tiers slow	Event loop blocked somewhere	Add asyncio.set_event_loop_debug(True) to find slow callbacks

---

Related resources

• Fault-Tolerance wiki — full architecture including retry budget, circuit breaker, last-good serving
• Parallel-Execution wiki — tier structure and expected timing targets
• Issue #126 — CycleProfiler implementation
• Issue #127 — per-phase timing constraints discussion
• Discussion #141 — health dashboard options for production monitoring