feat(observability): deploy policy-reporter for PolicyReport metrics + dashboards (P12, closes #93) - #1085
Merged
Merged
Conversation
…+ dashboards (P12, #93) New observability-policy-reporter module wraps kyverno/policy-reporter (3.7.4): watches the PolicyReport/ClusterPolicyReport CRs Kyverno already produces (ADR-014, both validate/mutate and cosign verifyImages/ verifyAttestations) and exposes them as Prometheus metrics (policy_report_result / cluster_policy_report_result, detailed label mode) + ships its own bundled Grafana dashboards (Overview, PolicyReport details, ClusterPolicyReport details) via the same sidecar ConfigMap convention every other add-on here uses. Verified locally with `helm template` before wiring: the chart needs `monitoring.enabled = true` as a separate top-level gate alongside the nested serviceMonitor/grafana.dashboards toggles (every monitoring template checks it first) — confirmed both the hub (dashboards rendered) and spoke (ServiceMonitor only, no ConfigMaps) value shapes produce the expected resources before ever touching real state. Deployed on both clusters (hub renders dashboards, preprod spoke only emits metrics — same split as OpenCost/P11): every dashboard panel gets a $cluster filter via the chart's own multicluster option, matching the house dashboard convention (#151), since Prometheus's externalLabels.cluster (ADR-043/044) is already on every series regardless of source. New enable_policy_reporting cost_profile-style toggle in _base.hcl, on for both platform and preprod in env.hcl — this is cheap (one small watcher pod, no cloud spend) and the PolicyReport volume that matters lives on preprod (tenant environments under Kyverno Enforce). Curated alerting deferred to a follow-up commit once live metric/job names are verified against the real deployment, per house convention (verify PromQL live before committing an alert). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
💰 Infracost reportMonthly estimate generatedEstimate details (includes details of unsupported resources) |
💰 Infracost reportThis pull request is aligned with your company's FinOps policies and the Well-Architected Framework. Monthly estimate generatedEstimate details (includes details of unsupported resources)This comment will be updated when code changes. |
3 tasks
gangster
added a commit
that referenced
this pull request
Jul 2, 2026
…up) (#1086) 3 alerts (PolicyReporterDown, PolicyReportNewViolation, ClusterPolicyReportNewViolation) verified against the real deployment applied in #1085 — job="policy-reporter", label set (cluster/exported_namespace/ policy/severity/source/status), and health=ok on both clusters (currently zero fail/error results; 61 series total, all pass, confirming the label names are right rather than just accidentally matching nothing). policy_report_result / cluster_policy_report_result are GAUGES (current report state, re-emitted each cycle) — used a level check (sum(...) > 0, for: 15m) rather than increase(), which would be wrong for a gauge. Added the matching docs/runbooks/observability-alerts.md section per house convention. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
1 task
gangster
added a commit
that referenced
this pull request
Jul 2, 2026
#151 status (#1087) Updates the P12 status in the plan doc + current-state doc to reflect what's now live and verified: policy-reporter on both clusters (#1085), curated alerts (#1086), real PolicyReport/ClusterPolicyReport data flowing through the federated Mimir view. Curated-alert count bumped 28->31 / 7->8 components. Notes the one real limitation (curated alerts only evaluate against the hub's local Prometheus — preprod's PolicyReport data is dashboarded but not yet directly alerted on, same gap every other curated alert already has pending the Mimir ruler). Also reflects #595/#151 already being closed. Most of the diff in 102-observability-stack.md is the repo's markdown formatter normalizing every "-" bullet in the file to "+" on save — a pre-existing house style the formatter enforces, unrelated to this change; the actual content edit is the status header + the P12 phase entry. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#102 P12 — closes #93. New
observability-policy-reportermodule wrapskyverno/policy-reporter (3.7.4): watches the
PolicyReport/ClusterPolicyReportCRs Kyverno already produces (ADR-014, both validate/mutate andcosign verifyImages/verifyAttestations) and exposes them as Prometheus metrics
(
policy_report_result/cluster_policy_report_result,detailedlabel mode: namespace, policy, rule,kind, name, status, severity, category, source) + ships its own bundled Grafana dashboards (Overview,
PolicyReport details, ClusterPolicyReport details) via the same sidecar ConfigMap convention every
other add-on here already uses.
create_dashboards = true) renders the dashboards, the preprod spoke (create_dashboards = false) only emits metrics —the hub's federated
Mimir (all clusters)datasource renders the spoke's data too.$clusterfilter via the chart's ownmulticlusteroption, matchingthe house dashboard convention (Dashboards: filter by cluster/environment and by tenant (template variables + convention) #151) —
externalLabels.cluster(ADR-043/044) is already on everyseries regardless of source, so this works with zero extra wiring.
enable_policy_reportingcost_profile-style toggle in_base.hcl, explicitly on for bothplatform and preprod in
env.hcl— cheap (one small watcher pod, no cloud spend), and thePolicyReport volume that matters lives on preprod (tenant environments under Kyverno Enforce; already
confirmed real data exists there:
alpha-checkout-dev/alpha-conformance-dev/etc. PolicyReports +ClusterPolicyReports).
Curated alerting ("spikes / new violations" per #93's scope) is deferred to a follow-up commit once
live metric/job names are confirmed against the real deployment — per house convention (verify PromQL
live before committing an alert), not guessed up front.
Test plan
helm templatedry-run against both the hub and spoke value shapes before wiring any Terraform —caught that the chart needs
monitoring.enabled = trueas a separate top-level gate alongsidethe nested
serviceMonitor/grafana.dashboardstoggles (every monitoring template checks itfirst); confirmed the hub renders 3 dashboard ConfigMaps + a ServiceMonitor, the spoke renders
only the ServiceMonitor
tofu fmt -check/tofu validateclean on the new moduleterragrunt validatepasses on both new units (platform + preprod)terragrunt hcl fmt --checkclean on all touched.hclterraform-docsgenerated for the new module;markdownlint-cli2clean🤖 Generated with Claude Code