Ideas: Phase 5 evaluation framework — how do we measure online learning quality?

[web3guru888]

Phase 5 adds self-modification: STDP weight updates, federated hot-reload, KG transactional writes, and consciousness-guided goal re-prioritization all change cognitive state at runtime. How do we know the system is learning rather than drifting?

This thread proposes an evaluation framework for Phase 5 — concrete metrics and tooling to answer: "Is online learning actually helping?"

---

The measurement problem

Phase 5 operates on a live CognitiveCycle. Unlike offline training, we cannot pause the system to compute validation loss. Metrics need to be:

• Non-intrusive — computed from Blackboard reads, not requiring a held-out dataset
• Continuous — updated every N ticks, not after a training epoch
• Actionable — crossing a threshold should trigger automatic rollback or an alert

---

Proposed metrics

5.1 Online Learning (STDP + federated)

Metric	Definition	Target
weight_delta_norm_p95	95th-percentile Frobenius norm of accepted WeightDelta	< 0.3 (stability)
safety_gate_rejection_rate	Fraction of WeightDelta rejected by safety_gate() over 1000 ticks	< 0.05 (healthy)
hot_reload_success_rate	Fraction of MODEL_HOT_RELOAD events that complete without rollback	> 0.98
stdp_convergence_window	Ticks until STDP weight variance drops below 0.01 after a spike burst	< 500

5.2 Emergent Coordination

Metric	Definition	Target
coalition_size_mean	Mean agent count per formed coalition	2–6 (task-dependent)
role_negotiation_rounds	Mean rounds to reach stable role assignment	< 4
pheromone_trail_ttl_variance	Variance in pheromone decay across agents	< 0.1
task_allocation_latency_p99	Latency from task arrival to first agent assignment	< 200ms

5.3 Persistent Memory

Metric	Definition	Target
episodic_consolidation_rate	Fraction of eligible episodes consolidated during SLEEP_PHASE	> 0.90
kg_write_conflict_rate	Fraction of KG transactional writes that require rollback	< 0.02
retrieval_hit_rate	Fraction of KG queries answered from episodic memory vs cold KG	> 0.60

5.4 Consciousness-Guided Planning

Metric	Definition	Target
phi_weighted_goal_acceptance_rate	Fraction of high-Φ goal proposals accepted	> 0.80
planning_horizon_drift	Change in mean planning horizon between consecutive ticks	< 0.05
gwt_broadcast_latency_p99	Latency from GWTInferenceBridge trigger to ConsciousnessPlanner update	< 50ms

---

Option A: In-process metrics (Blackboard-native)

All metrics written to Blackboard under metrics.phase5.* every 100 ticks. The CycleFaultSummary SSE endpoint exposes them as part of the health stream. The existing CycleProfiler provides the tick-level data for P95 computations.

Pros: No new infrastructure. Uses existing Blackboard + SSE stack.
Cons: Metrics are not retained across sessions (unless MemoryConsolidator persists them).

Option B: Prometheus + Grafana (external)

Phase 4.3 Prometheus exporter (/metrics endpoint) is extended with Phase 5 gauge/histogram metrics. A Grafana dashboard visualizes trend over time.

Pros: Persistent, queryable, visualizable. Aligns with production deployment (#164).
Cons: Adds operational dependency; CI smoke tests cannot easily assert on Prometheus values.

Option C: Hybrid (in-process + sampling to Prometheus)

In-process metrics updated every tick. Every 1000 ticks, a background task pushes a snapshot to Prometheus. CI asserts on the in-process values; production uses the Prometheus time series.

---

Recommendation

Option C. The in-process path gives fast feedback during development and CI. Prometheus gives operational visibility in production. The boundary is at 1000 ticks (≈10s at 100Hz) — frequent enough to catch drift, infrequent enough to avoid prometheus_client overhead dominating the hot loop.

---

Open questions

1. Should weight_delta_norm_p95 trigger an automatic circuit-breaker in OnlineLearningAdapter (analogous to the Phase 4 circuit breaker in Add circuit breaker pattern to CognitiveCycle: auto-disable persistently failing modules with OPEN/HALF_OPEN/CLOSED health states #137)?
2. How should phi_weighted_goal_acceptance_rate interact with the Safety module threshold from Wire EthicalVerificationEngine into Blackboard write/read pipeline #37 — should Φ ever override safety gates?
3. Is episodic_consolidation_rate the right primary metric for 5.3, or should we use something closer to downstream task performance?