CP-44601: Aggregate CloudZeroWebhookNoEvents with sum() to stop per-series misfires

[evan-cz]

Problem

The CloudZeroWebhookNoEvents PrometheusRule alert fires on healthy webhooks. Its expression

rate(czo_webhook_types_total{job="..."}[30m]) == 0

is evaluated per series. czo_webhook_types_total is a counter split by kind_group/kind_version/kind_resource/operation (and per pod/instance by the scrape), so the alert fires whenever any single series is flat for 30m — a rarely-seen resource/operation (e.g. storageclass creates) or one quiet replica — even while the webhook is healthy and busy overall.

It also never fires in the case it is meant to catch: the counter has no series until the first admission event, so a webhook that is up but receiving nothing produces an absent metric and an empty result.

Fix

(sum(rate(czo_webhook_types_total{job="..."}[30m])) or vector(0)) == 0

• sum() collapses the per-label and per-pod series into a single fleet-wide rate, so the alert fires only when no replica sees any admission event of any type.
• or vector(0) substitutes 0 when the metric is absent, so genuine silence still fires.

The template comment and helm/docs/monitoring-infrastructure.md are updated to match.

Verification

• helm template renders the corrected expression.
• helm-unittest webhookserver_gating_test.yaml passes 36/36.

This PR was created with AI assistance (Claude Code by Anthropic).

[greptile-apps]

Greptile Summary

Fixes the CloudZeroWebhookNoEvents PrometheusRule alert so it no longer fires spuriously on quiet-but-healthy individual metric series, and now correctly fires when the webhook is genuinely silent from startup.

• PromQL fix: replaces the bare rate(...) == 0 (evaluated per series) with (sum(rate(...)) or vector(0)) == 0, collapsing all label and pod dimensions into a single fleet-wide rate and substituting 0 when the counter is absent.
• Comment: adds a detailed inline explanation in the template justifying both the sum() and or vector(0) requirements.
• Docs: updates helm/docs/monitoring-infrastructure.md to reflect the new sum(rate) == 0 condition and its intent.

Confidence Score: 5/5

• Safe to merge — the change narrows a noisy alert to fire only on genuine fleet-wide silence, and the absent-metric case is correctly handled with or vector(0).
• Both the spurious-fire bug (per-series evaluation) and the never-fires bug (absent metric returns empty vector) are addressed correctly. The alert's explicit labels: block (severity: critical, component: webhook) preserves all Alertmanager routing context that the bare sum() would otherwise strip. The template comment and documentation are thorough and accurate.
• No files require special attention.

Important Files Changed

Filename	Overview
helm/templates/prometheusrule.yaml	Fixes CloudZeroWebhookNoEvents alert expression by wrapping with sum() to collapse per-label/per-pod series into a single fleet-wide rate, and adding or vector(0) to handle the absent-metric case. The updated comment thoroughly documents both problems and why each part of the fix is required. Routing labels (severity: critical, component: webhook) are explicitly set in the alert labels: block, so no label context is lost from sum() stripping metric labels.
helm/docs/monitoring-infrastructure.md	Documentation updated to reflect the corrected sum(rate) == 0 condition and adds a brief explanation of the fleet-wide aggregation and or vector(0) fallback. Formatting-only change otherwise (column width adjustments).

_{Reviews (2): Last reviewed commit: "CP-44601: Aggregate CloudZeroWebhookNoEv..." | Re-trigger Greptile}