Skip to content

feat(observability): deploy policy-reporter for PolicyReport metrics + dashboards (P12, closes #93) - #1085

Merged
gangster merged 1 commit into
mainfrom
policy-reporter-p12-93
Jul 1, 2026
Merged

feat(observability): deploy policy-reporter for PolicyReport metrics + dashboards (P12, closes #93)#1085
gangster merged 1 commit into
mainfrom
policy-reporter-p12-93

Conversation

@gangster

@gangster gangster commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

#102 P12closes #93. New observability-policy-reporter module wraps
kyverno/policy-reporter (3.7.4): watches the
PolicyReport/ClusterPolicyReport CRs Kyverno already produces (ADR-014, both validate/mutate and
cosign verifyImages/verifyAttestations) and exposes them as Prometheus metrics
(policy_report_result/cluster_policy_report_result, detailed label mode: namespace, policy, rule,
kind, name, status, severity, category, source) + ships its own bundled Grafana dashboards (Overview,
PolicyReport details, ClusterPolicyReport details) via the same sidecar ConfigMap convention every
other add-on here already uses.

  • Deployed on both clusters, same hub/spoke split as OpenCost (P11): the hub (create_dashboards = true) renders the dashboards, the preprod spoke (create_dashboards = false) only emits metrics —
    the hub's federated Mimir (all clusters) datasource renders the spoke's data too.
  • Every dashboard panel gets a $cluster filter via the chart's own multicluster option, matching
    the house dashboard convention (Dashboards: filter by cluster/environment and by tenant (template variables + convention) #151) — externalLabels.cluster (ADR-043/044) is already on every
    series regardless of source, so this works with zero extra wiring.
  • New enable_policy_reporting cost_profile-style toggle in _base.hcl, explicitly on for both
    platform and preprod in env.hcl — cheap (one small watcher pod, no cloud spend), and the
    PolicyReport volume that matters lives on preprod (tenant environments under Kyverno Enforce; already
    confirmed real data exists there: alpha-checkout-dev/alpha-conformance-dev/etc. PolicyReports +
    ClusterPolicyReports).

Curated alerting ("spikes / new violations" per #93's scope) is deferred to a follow-up commit once
live metric/job names are confirmed against the real deployment — per house convention (verify PromQL
live before committing an alert), not guessed up front.

Test plan

  • helm template dry-run against both the hub and spoke value shapes before wiring any Terraform —
    caught that the chart needs monitoring.enabled = true as a separate top-level gate alongside
    the nested serviceMonitor/grafana.dashboards toggles (every monitoring template checks it
    first); confirmed the hub renders 3 dashboard ConfigMaps + a ServiceMonitor, the spoke renders
    only the ServiceMonitor
  • tofu fmt -check / tofu validate clean on the new module
  • terragrunt validate passes on both new units (platform + preprod)
  • terragrunt hcl fmt --check clean on all touched .hcl
  • terraform-docs generated for the new module; markdownlint-cli2 clean
  • Not applied to any live cluster from this PR

🤖 Generated with Claude Code

…+ dashboards (P12, #93)

New observability-policy-reporter module wraps kyverno/policy-reporter
(3.7.4): watches the PolicyReport/ClusterPolicyReport CRs Kyverno already
produces (ADR-014, both validate/mutate and cosign verifyImages/
verifyAttestations) and exposes them as Prometheus metrics
(policy_report_result / cluster_policy_report_result, detailed label mode) +
ships its own bundled Grafana dashboards (Overview, PolicyReport details,
ClusterPolicyReport details) via the same sidecar ConfigMap convention every
other add-on here uses.

Verified locally with `helm template` before wiring: the chart needs
`monitoring.enabled = true` as a separate top-level gate alongside the nested
serviceMonitor/grafana.dashboards toggles (every monitoring template checks
it first) — confirmed both the hub (dashboards rendered) and spoke
(ServiceMonitor only, no ConfigMaps) value shapes produce the expected
resources before ever touching real state.

Deployed on both clusters (hub renders dashboards, preprod spoke only emits
metrics — same split as OpenCost/P11): every dashboard panel gets a $cluster
filter via the chart's own multicluster option, matching the house dashboard
convention (#151), since Prometheus's externalLabels.cluster (ADR-043/044) is
already on every series regardless of source.

New enable_policy_reporting cost_profile-style toggle in _base.hcl, on for
both platform and preprod in env.hcl — this is cheap (one small watcher pod,
no cloud spend) and the PolicyReport volume that matters lives on preprod
(tenant environments under Kyverno Enforce).

Curated alerting deferred to a follow-up commit once live metric/job names
are verified against the real deployment, per house convention (verify PromQL
live before committing an alert).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown

💰 Infracost report

Monthly estimate generated

Estimate details (includes details of unsupported resources)
──────────────────────────────────
72 projects have no cost estimate changes.
Run the following command to see their breakdown: infracost breakdown --path=/path/to/code

──────────────────────────────────
162 cloud resources were detected:
∙ 34 were estimated
∙ 124 were free
∙ 4 are not supported yet, see https://infracost.io/requested-resources:
  ∙ 1 x aws_athena_workgroup
  ∙ 1 x aws_ce_anomaly_monitor
  ∙ 1 x aws_ce_anomaly_subscription
  ∙ 1 x aws_cur_report_definition
This comment will be updated when code changes.

@infracost

infracost Bot commented Jul 1, 2026

Copy link
Copy Markdown

💰 Infracost report

This pull request is aligned with your company's FinOps policies and the Well-Architected Framework.

Monthly estimate generated
Estimate details (includes details of unsupported resources)
Key: * usage cost, ~ changed, + added, - removed

──────────────────────────────────
Key: * usage cost, ~ changed, + added, - removed
2 projects have no cost estimate changes.
Run the following command to see their breakdown: infracost breakdown --path=/path/to/code

──────────────────────────────────

*Usage costs can be estimated by updating [Infracost Cloud settings](https://dashboard.infracost.io/org/reference-platform/settings/usage-cost-defaults), see [docs](https://www.infracost.io/docs/features/usage_based_resources/#infracost-usageyml) for other options.

605 cloud resources were detected:
∙ 76 were estimated
∙ 463 were free
∙ 66 are not supported yet, see https://infracost.io/requested-resources:
  ∙ 36 x aws_subnet
  ∙ 9 x aws_eks_access_entry
  ∙ 4 x aws_identitystore_group
  ∙ 4 x aws_identitystore_user
  ∙ 3 x aws_ecr_lifecycle_policy
  ∙ 2 x aws_ce_cost_allocation_tag
  ∙ 2 x aws_flow_log
  ∙ 2 x aws_vpc
  ∙ 1 x aws_athena_workgroup
  ∙ 1 x aws_ce_anomaly_subscription
  ∙ 1 x aws_cur_report_definition
  ∙ 1 x aws_organizations_organization

This comment will be updated when code changes.

@gangster
gangster merged commit a9cc2e5 into main Jul 1, 2026
24 checks passed
gangster added a commit that referenced this pull request Jul 2, 2026
…up) (#1086)

3 alerts (PolicyReporterDown, PolicyReportNewViolation,
ClusterPolicyReportNewViolation) verified against the real deployment applied
in #1085 — job="policy-reporter", label set (cluster/exported_namespace/
policy/severity/source/status), and health=ok on both clusters (currently
zero fail/error results; 61 series total, all pass, confirming the label
names are right rather than just accidentally matching nothing).

policy_report_result / cluster_policy_report_result are GAUGES (current
report state, re-emitted each cycle) — used a level check (sum(...) > 0,
for: 15m) rather than increase(), which would be wrong for a gauge.

Added the matching docs/runbooks/observability-alerts.md section per house
convention.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
gangster added a commit that referenced this pull request Jul 2, 2026
#151 status (#1087)

Updates the P12 status in the plan doc + current-state doc to reflect what's
now live and verified: policy-reporter on both clusters (#1085), curated
alerts (#1086), real PolicyReport/ClusterPolicyReport data flowing through
the federated Mimir view. Curated-alert count bumped 28->31 / 7->8 components.
Notes the one real limitation (curated alerts only evaluate against the hub's
local Prometheus — preprod's PolicyReport data is dashboarded but not yet
directly alerted on, same gap every other curated alert already has pending
the Mimir ruler). Also reflects #595/#151 already being closed.

Most of the diff in 102-observability-stack.md is the repo's markdown
formatter normalizing every "-" bullet in the file to "+" on save — a
pre-existing house style the formatter enforces, unrelated to this change;
the actual content edit is the status header + the P12 phase entry.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kyverno PolicyReport observability (Prometheus/Grafana + policy-reporter)

1 participant