fix(rook-ceph): stop CephMonitorQuorumAtRisk firing critical on healthy clusters

[Aviator-Coding]

Intent

Fix the CephMonitorQuorumAtRisk PrometheusRule so it stops firing critical forever on a healthy cluster.

The rule's expression 'ceph_mon_quorum_status < 3' has a gauge-vs-count bug: the metric is a per-mon gauge (1 = in quorum, 0 = not), not a count. On a healthy 3-mon cluster, each monitor evaluates to '1 < 3' = true, causing the critical alert to fire once per healthy monitor permanently.

Changed to 'count(ceph_mon_quorum_status == 1) < 3' to properly count monitors in quorum and alert only when fewer than 3 are in quorum.

Verified that no other identical gauge-vs-count mistakes exist in the file. The metric has been firing continuously since 16:02:53Z 2026-08-23 when the mgr module resumed exporting after a prior disable.

What Changed

• Fixed the CephMonitorQuorumAtRisk alert expression in kubernetes/apps/rook-ceph/rook-ceph/cluster/prometheusrules.yaml from ceph_mon_quorum_status < 3 to sum(ceph_mon_quorum_status) < 3, since the metric is a per-mon gauge (1 = in quorum, 0 = not) rather than a count, and comparing it directly to 3 made every healthy mon evaluate 1 < 3 = true, firing the alert as critical permanently on a healthy 3-mon cluster.
• sum() was chosen over an earlier count(ceph_mon_quorum_status == 1) < 3 fix to avoid an empty-vector blind spot, where a matcher with zero matching series (e.g. total quorum loss) would return no series and never fire.
• Documented the root cause, fix, risk assessment, and rollback/verification steps in docs/ceph-cluster-changelog.md.

Risk Assessment

✅ Low: The fix-round change is a single-line PromQL expression edit (count-with-filter to sum) that correctly resolves both the original gauge-vs-count false-positive and the round-1 empty-vector-at-zero-quorum blind spot, with no other files touched.

Testing

Using a locally-fetched promtool (Prometheus v3.14.0, downloaded to a scratch /tmp dir and removed afterward) I wrote unit-test scenarios simulating healthy, degraded, and total-quorum-loss cluster states against the actual rule expressions from each commit: the base-commit buggy expr fires on all three healthy monitors (reproducing the reported bug exactly), an intermediate count(==1) variant silently misses the total-outage case (empty-vector blind spot), and the target commit's shipped sum(ceph_mon_quorum_status) < 3 correctly stays silent when healthy and fires in both degraded scenarios. A kustomize build of the cluster directory also confirms the fix renders correctly into the deployed PrometheusRule. Note the shipped expression differs from the literal text in the user intent (count(...==1) vs sum(...)) because a same-branch follow-up review commit (c1431311) improved on the first fix (649869b0) to close the exact blind-spot demonstrated above - the final code still fully satisfies and exceeds the stated requirement. No issues found; all transient test files and the downloaded promtool binary were removed and the worktree is clean.

Evidence: promtool unit-test transcript: pre-fix bug repro, blind-spot regression, and passing fix

### promtool test rules -- target commit fix: sum(ceph_mon_quorum_status) < 3
$ promtool test rules test_after.yml
  SUCCESS

exit: 0

### promtool test rules -- base commit (pre-fix) expr: ceph_mon_quorum_status < 3
$ promtool test rules test_before.yml
  FAILED:
    name: healthy 3-mon cluster must NOT alert,
    alertname: CephMonitorQuorumAtRisk, time: 5m, 
        exp:[], 
        got:[
            0:
              Labels:{alertname="CephMonitorQuorumAtRisk", ceph_daemon="mon-a", severity="critical"}
              Annotations:{summary="Ceph monitor quorum has fewer than 3 members"},
            1:
              Labels:{alertname="CephMonitorQuorumAtRisk", ceph_daemon="mon-b", severity="critical"}
              Annotations:{summary="Ceph monitor quorum has fewer than 3 members"},
            2:
              Labels:{alertname="CephMonitorQuorumAtRisk", ceph_daemon="mon-c", severity="critical"}
              Annotations:{summary="Ceph monitor quorum has fewer than 3 members"}
            ]

    name: one mon down (2 of 3 in quorum) must alert,
    alertname: CephMonitorQuorumAtRisk, time: 5m, 
        exp:[
            0:
              Labels:{alertname="CephMonitorQuorumAtRisk", severity="critical"}
              Annotations:{summary="Ceph monitor quorum has fewer than 3 members"}
            ], 
        got:[
            0:
              Labels:{alertname="CephMonitorQuorumAtRisk", ceph_daemon="mon-a", severity="critical"}
              Annotations:{summary="Ceph monitor quorum has fewer than 3 members"},
            1:
              Labels:{alertname="CephMonitorQuorumAtRisk", ceph_daemon="mon-b", severity="critical"}
              Annotations:{summary="Ceph monitor quorum has fewer than 3 members"},
            2:
              Labels:{alertname="CephMonitorQuorumAtRisk", ceph_daemon="mon-c", severity="critical"}
              Annotations:{summary="Ceph monitor quorum has fewer than 3 members"}
            ]

    name: total quorum loss (all mons down) must still alert, no blind spot,
    alertname: CephMonitorQuorumAtRisk, time: 5m, 
        exp:[
            0:
              Labels:{alertname="CephMonitorQuorumAtRisk", severity="critical"}
              Annotations:{summary="Ceph monitor quorum has fewer than 3 members"}
            ], 
        got:[
            0:
              Labels:{alertname="CephMonitorQuorumAtRisk", ceph_daemon="mon-a", severity="critical"}
              Annotations:{summary="Ceph monitor quorum has fewer than 3 members"},
            1:
              Labels:{alertname="CephMonitorQuorumAtRisk", ceph_daemon="mon-b", severity="critical"}
              Annotations:{summary="Ceph monitor quorum has fewer than 3 members"},
            2:
              Labels:{alertname="CephMonitorQuorumAtRisk", ceph_daemon="mon-c", severity="critical"}
              Annotations:{summary="Ceph monitor quorum has fewer than 3 members"}
            ]


exit: 1

### promtool test rules -- intermediate variant: count(ceph_mon_quorum_status == 1) < 3
$ promtool test rules test_count_variant.yml
  FAILED:
    name: total quorum loss (all mons down) must still alert, no blind spot,
    alertname: CephMonitorQuorumAtRisk, time: 5m, 
        exp:[
            0:
              Labels:{alertname="CephMonitorQuorumAtRisk", severity="critical"}
              Annotations:{summary="Ceph monitor quorum has fewer than 3 members"}
            ], 
        got:[]


exit: 1

Evidence: kustomize build of kubernetes/apps/rook-ceph/rook-ceph/cluster showing rendered rule

- alert: CephMonitorQuorumAtRisk
annotations:
summary: Ceph monitor quorum has fewer than 3 members
expr: sum(ceph_mon_quorum_status) < 3
for: 1m
labels:
severity: critical

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed ✅

• ⚠️ kubernetes/apps/rook-ceph/rook-ceph/cluster/prometheusrules.yaml:39 - The new expr count(ceph_mon_quorum_status == 1) < 3 fixes the stated gauge-vs-count false-positive on a healthy cluster, but introduces a false-negative blind spot in the exact worst-case scenario the alert exists to catch: total quorum loss. ceph_mon_quorum_status == 1 filters the instant vector to only series currently equal to 1; if zero monitors are in quorum (all three report 0), that filtered vector is empty. PromQL aggregations (count, sum, etc.) over an empty vector return an empty result, not the literal number 0 - so count(...) < 3 never evaluates to true and CephMonitorQuorumAtRisk silently fails to fire precisely when the whole mon quorum is down. Partial loss (1 or 2 of 3 mons in quorum) is unaffected and fires correctly. A more robust expression that doesn't drop zero-valued series - e.g. sum(ceph_mon_quorum_status) < 3 (sums the 0/1 gauge directly without filtering) or (count(ceph_mon_quorum_status == 1) or vector(0)) < 3 - would keep the comparison well-defined at zero mons in quorum.

🔧 Fix: fix(rook-ceph): use sum() to avoid empty-vector blind spot at zero quorum
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

• promtool check rules on the exact rule text from prometheusrules.yaml
• promtool test rules against a healthy 3-mon cluster (all ceph_mon_quorum_status=1): fix produces no alert
• promtool test rules against a degraded cluster (1 of 3 mons out of quorum): fix correctly fires the alert
• promtool test rules against total quorum loss (0 of 3 mons in quorum): fix correctly fires, unlike the count(==1) intermediate variant which returns an empty vector and silently misses this case
• promtool test rules against the base-commit (pre-fix) expr ceph_mon_quorum_status < 3: reproduces the reported bug, firing on all 3 healthy monitors
• kubectl kustomize kubernetes/apps/rook-ceph/rook-ceph/cluster --load-restrictor LoadRestrictionsNone to confirm the PrometheusRule still renders correctly with the new expression
• Manual review of the full prometheusrules.yaml file to confirm no other alert repeats the same per-instance-gauge-vs-count-threshold mistake

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[mortyops]

--- kubernetes/apps/rook-ceph/rook-ceph/cluster Kustomization: rook-ceph/rook-ceph-cluster PrometheusRule: rook-ceph/ceph-custom-alerts

+++ kubernetes/apps/rook-ceph/rook-ceph/cluster Kustomization: rook-ceph/rook-ceph-cluster PrometheusRule: rook-ceph/ceph-custom-alerts

@@ -38,13 +38,13 @@

       for: 10m
       labels:
         severity: warning
     - alert: CephMonitorQuorumAtRisk
       annotations:
         summary: Ceph monitor quorum has fewer than 3 members
-      expr: ceph_mon_quorum_status < 3
+      expr: sum(ceph_mon_quorum_status) < 3
       for: 1m
       labels:
         severity: critical
     - alert: CephClusterNearFull
       annotations:
         description: Cluster is {{ $value | humanizePercentage }} full. Consider adding