Show & Tell: Inside Phase5MetricsCollector — sliding windows, Blackboard namespacing, and circuit-breaker integration

[web3guru888]

Overview

Phase 5 brings ASI:BUILD's first real-time self-modification loop: STDP weight updates during live CognitiveCycle ticks, federated hot-reload across mesh agents, and KG transactional writes during SLEEP_PHASE. But self-modification is only safe if you can measure whether it's helping.

Issue #215 defines the Phase 5 evaluation framework — this post walks through the key design decisions in Phase5MetricsCollector.

---

Architecture in one diagram

CognitiveCycle tick
       │
       │ (end of ONLINE_LEARNING tier)
       ▼
Phase5MetricsCollector.on_tick(tick_id)
       ├── reads learning.weight_updates.*    → updates delta_norms deque
       ├── reads learning.hot_reload.events.* → updates reload_success deque
       ├── reads coordination.coalitions.*    → updates coalition_size deque
       ├── reads memory.consolidation.*       → updates consolidation_rate deque
       └── reads consciousness.goal_proposals.*→ updates phi_acceptance deque
              │
              │ every tick
              ▼
      Blackboard: metrics.phase5.*
              │
              │ every 1000 ticks (wall-clock: every 10s)
              ▼
      Prometheus: asi_build_* gauges + histograms
              │
              │ threshold crossing?
              ▼
      Phase5RollbackManager.on_metrics_tick()

---

Design decision 1: Deque-based sliding windows

Every rate and percentile metric uses collections.deque(maxlen=1000):

self._delta_norms: deque[float] = deque(maxlen=1000)
self._rejection_flags: deque[bool] = deque(maxlen=1000)
self._reload_success: deque[bool] = deque(maxlen=1000)

At 100Hz, maxlen=1000 gives a 10-second rolling view. Why 10 seconds?

• Long enough to catch a burst of bad weight updates (e.g., a runaway STDP episode)
• Short enough that a fix (circuit open → weights revert → circuit reset) shows up in metrics within one 30-second reset window
• Short enough to avoid false alarms from a single noisy tick

For latency histograms, we use deque(maxlen=10_000) — a longer window because latency outliers matter more than rates.

---

Design decision 2: Blackboard key namespace

All Phase 5 metrics go under metrics.phase5.<subsystem>.<name> with a 5-tick TTL:

metrics.phase5.online_learning.weight_delta_norm_p95        TTL=5
metrics.phase5.online_learning.safety_gate_rejection_rate   TTL=5
metrics.phase5.coordination.coalition_size_mean             TTL=5
metrics.phase5.coordination.task_allocation_latency_p99     TTL=5
metrics.phase5.memory.episodic_consolidation_rate           TTL=5
metrics.phase5.consciousness.phi_weighted_goal_acceptance_rate TTL=5

The 5-tick TTL means a stale metric disappears rather than staying at its last-seen value. A consumer that reads None knows to treat it as "no data", not "last known good."

---

Design decision 3: P95 without hot-path numpy

Calling numpy.percentile every tick on 1000 elements is ~0.05ms. At 100Hz that's 5ms/second of pure metric overhead — unacceptable for a real-time loop. Instead:

def _p95(self, window: deque[float]) -> float | None:
    if len(window) < 20:  # not enough data yet
        return None
    # Amortize: only re-sort every 100 ticks
    if self._tick_count % 100 == 0:
        self._sorted_delta_norms = sorted(window)
    if not self._sorted_delta_norms:
        return None
    idx = int(0.95 * len(self._sorted_delta_norms))
    return self._sorted_delta_norms[min(idx, len(self._sorted_delta_norms) - 1)]

This amortizes the sort cost over 100 ticks, giving a "snapshot P95" updated at 1Hz rather than 100Hz. For alerting purposes 1Hz is more than sufficient.

---

Design decision 4: Prometheus push in background asyncio.Task

async def start(self) -> None:
    self._prometheus_task = asyncio.create_task(self._prometheus_push_loop())

async def _prometheus_push_loop(self) -> None:
    while True:
        await asyncio.sleep(10.0)  # push every 10 seconds wall-clock
        snapshot = await self._build_snapshot()
        self._export_to_prometheus(snapshot)

Pushing every 10 seconds decouples the metric push from tick rate. If the cognitive cycle slows to 50Hz during a heavy coordination round, Prometheus still gets data on a wall-clock schedule.

---

Design decision 5: Circuit breaker reuse

Rather than inventing Phase 5-specific circuit breakers, we reuse the CircuitState enum and Blackboard key convention from Phase 4 (#137):

# Phase 4 circuit breaker pattern (existing)
bb.write("circuit.<module_id>.state", CircuitState.OPEN)

# Phase 5 learning circuit (new, same pattern)
bb.write("circuit.online_learning.state", CircuitState.OPEN)
bb.write("circuit.online_learning.reason", "weight_delta_norm_p95 exceeded 0.8")
bb.write("circuit.online_learning.reset_at", utc_now() + timedelta(seconds=30))

STDPOnlineLearner.compute_update() already reads circuit.<module_id>.state as part of the standard module health check. Adding circuit.online_learning.state to that check requires zero new infrastructure.

---

Open questions

1. Metric cold start: On system boot, deques are empty. Should Phase5MetricsCollector suppress alerts for the first 1000 ticks (10 seconds), or emit a NO_DATA state to downstream consumers?

2. Multi-agent aggregation: In a 4-agent mesh, each agent runs its own Phase5MetricsCollector. Should per-agent metrics be aggregated in MeshCoordinator before alerting, or should each agent alert independently (risking alert storms)?

3. SLEEP_PHASE sampling: episodic_consolidation_rate is only meaningful during SLEEP_PHASE ticks. Should the collector skip non-SLEEP_PHASE ticks for that metric, or track the phase flag via Blackboard read?

Related: #215 | #216 | #210 | #213