Ideas: What alerting strategy should Phase 5 use? Fixed thresholds vs. sustained windows vs. adaptive Bollinger bands

[web3guru888]

Context

Phase 5 gives ASI:BUILD a live self-modification loop. Issue #215 defines 12 metrics across 4 subsystems. But metrics alone don't tell operators what to do — you need an alerting strategy.

The question isn't just what to alert on, but how aggressive to be. False positives (alerting on normal STDP noise) train operators to ignore alerts. False negatives (missing a runaway learning episode) let the system degrade silently.

---

Option A: Fixed threshold alerts

The simplest approach: alert whenever a metric crosses its defined threshold for a single tick.

if weight_delta_norm_p95 > 0.3:
    alert(WARNING)
if weight_delta_norm_p95 > 0.8:
    alert(CRITICAL)

Pros: Simple, deterministic, easy to test.
Cons: Noisy. A single STDP burst can spike P95 above 0.3 for 1-2 ticks without indicating a real problem. Operators see alert fatigue.

---

Option B: Sustained threshold alerts (recommended baseline)

Alert only when the metric stays above threshold for N consecutive ticks:

SUSTAIN_TICKS = {"WARNING": 50, "CRITICAL": 10}  # 0.5s warning, 0.1s critical

if self._warning_streak >= SUSTAIN_TICKS["WARNING"]:
    alert(WARNING)

Pros: Filters single-tick spikes. Tunable per severity (critical alerts faster, warnings require sustained evidence).
Cons: Still static — the same thresholds apply regardless of system load or phase.

---

Option C: Adaptive thresholds (Bollinger bands)

Use rolling mean ± 2σ to detect anomalies relative to recent baseline:

mean = statistics.mean(window)
std = statistics.stdev(window)
upper_band = mean + 2 * std

if current_value > upper_band and current_value > ABSOLUTE_MINIMUM:
    alert(WARNING)

Pros: Self-calibrating. If STDP naturally produces delta norms around 0.15 on this workload, it won't alert until something deviates from that baseline, not the fixed 0.3 threshold.
Cons: Complex to test. May miss slow-drift regressions (the band adapts to the drift). Requires at least 200+ samples before meaningful.

---

Option D: Phase-aware alerting

Different thresholds for different CognitiveCycle phases:

THRESHOLDS = {
    CyclePhase.ONLINE_LEARNING: {"warning": 0.3, "critical": 0.8},
    CyclePhase.SLEEP_PHASE: {"warning": 0.5, "critical": 1.2},  # consolidation allows larger updates
    CyclePhase.PERCEPTION: {"warning": 0.1, "critical": 0.3},   # tighter during perception
}

Pros: More accurate to actual risk — a large weight delta during SLEEP_PHASE consolidation is safer than the same delta during a real-time perception tick.
Cons: More configuration surface, harder to reason about system state at a glance.

---

Recommendation

Start with Option B (sustained thresholds) for Phase 5 MVP:

• Low implementation complexity
• Directly testable: mock 50 consecutive ticks above threshold → assert alert fires
• Can be upgraded to Option C later once baseline behavior is measured empirically

Phase-aware multipliers (Option D) are worth adding for episodic_consolidation_rate specifically, since SLEEP_PHASE behavior is categorically different.

---

Open questions

1. Should Phase5RollbackManager (see #216) respond to sustained alerts automatically, or require operator acknowledgment for WARNING-level alerts?

2. How should multi-agent alerting work? If Agent 2's weight_delta_norm_p95 goes critical but Agents 1, 3, 4 are normal, should MeshCoordinator isolate Agent 2 or alert globally?

3. Should we expose alerting configuration via the MCP tool interface (from Add CycleFaultSummary: per-tick fault report written to Blackboard and exposed via SSE health endpoint #144), so operators can tune thresholds without redeploying?

Related: #215 | #216 | #213 | #217