Q&A: Phase 5 Prometheus metrics — cardinality, duplicate registration, and histogram pitfalls

[web3guru888]

Question

We are adding 20+ Prometheus metrics for Phase 5 (Issue #220), several with labels (e.g., phase5_rollback_active{scenario}, phase5_hot_reload_total{status}, phase5_kg_writes_total{tier}). What are the prometheus_client pitfalls to watch out for, and how do we keep cardinality under control?

---

Why cardinality matters

Prometheus stores one time series per unique label combination. High cardinality (many label values) inflates storage and query time. Phase 5 labels are all low-cardinality by design:

Metric	Label	Cardinality
phase5_rollback_active	scenario	4 (fixed enum)
phase5_hot_reload_total	status	3: success, rollback, timeout
phase5_kg_writes_total	tier	3: hot, warm, cold
phase5_consolidation_runs_total	status	3: success, skipped, interrupted

Total cardinality from labels: 13 extra series across 4 metrics. Well within Prometheus best practice (<10K series for a single job).

---

Pitfall 1: Duplicate metric registration on restart

Problem: If Phase5MetricsExporter.__init__() is called twice (e.g., in a test that re-creates the object), prometheus_client raises ValueError: Duplicated timeseries.

Solution: Use a CollectorRegistry instance per exporter, not the global default:

class Phase5MetricsExporter:
    def __init__(self, blackboard, registry=None):
        self._registry = registry or prometheus_client.REGISTRY
        self.weight_updates = Counter(
            "phase5_stdp_weight_updates_total",
            "...",
            registry=self._registry
        )

In tests:

registry = CollectorRegistry()
exporter = Phase5MetricsExporter(blackboard, registry=registry)
# Safe to create multiple times — each has its own registry

---

Pitfall 2: Gauge vs. Counter confusion for rollback state

Problem: Using a Counter for phase5_rollback_active is tempting ("increment when rollback starts, decrement when it ends"), but Prometheus counters must be monotonically increasing.

Solution: Use a Gauge:

self.rollback_active = Gauge(
    "phase5_rollback_active",
    "1 if rollback is currently active for this scenario",
    labelnames=["scenario"],
    registry=self._registry
)
# On rollback start:
self.rollback_active.labels(scenario="weight_regression").set(1)
# On recovery:
self.rollback_active.labels(scenario="weight_regression").set(0)

The separate phase5_rollback_triggered_total Counter tracks cumulative occurrences.

---

Pitfall 3: Histogram bucket boundary bleed

Problem: If a weight delta norm of 0.85 is observed but your highest bucket is 0.8, Prometheus places it in +Inf. This makes the P95 calculation look like +Inf when the system is at its critical threshold.

Solution: Always include at least one bucket above your critical threshold:

phase5_weight_delta_norm = Histogram(
    "phase5_weight_delta_norm",
    "...",
    buckets=[0.001, 0.005, 0.01, 0.05, 0.1, 0.3, 0.8, 2.0]
    # critical=0.8 → 2.0 bucket captures real values above critical
)

---

Pitfall 4: Missing pre-initialization of label combinations

Problem: phase5_rollback_active with scenario label will have no time series until a rollback actually occurs. Prometheus alert rules that check "is rollback_active == 0" will return no-data (not 0) until the first rollback.

Solution: Pre-initialize all label combinations to 0 in __init__:

for scenario in ["weight_regression", "hot_reload_failure", "kg_conflict_storm", "planner_divergence"]:
    self.rollback_active.labels(scenario=scenario).set(0)

for status in ["success", "rollback", "timeout"]:
    self.hot_reload.labels(status=status)  # Counter initializes to 0

---

Summary: Phase 5 prometheus_client checklist

• Pass registry=CollectorRegistry() in tests to prevent duplicate registration
• Use Gauge (not Counter) for rollback active state
• Histogram upper bucket ≥ 2× critical threshold
• Pre-initialize all labeled metrics to 0 in __init__
• Prefix all metrics with phase5_ — no exceptions

---

Related

• Issue Add Phase 5 Prometheus metrics layer: learning quality, safety invariant status, and rollback state #220: Phase 5 Prometheus metrics spec
• Discussion Show & Tell: Inside Phase5MetricsExporter — Prometheus bridge design, 20-metric taxonomy, and bucket strategy #221: Phase5MetricsExporter design decisions
• Issue Add Phase 4.3 production deployment: Docker Compose, Kubernetes Helm chart, Prometheus metrics, and health endpoints #164: Phase 4.3 base Prometheus setup