Publish metrics to Prometheus and Grafana

[realshuting]

Currently policy metrics are shown in the status. It would be good to expose a /metrics endpoint that can be used by Prometheus to collect and aggregate metrics. A Grafana dashboard can then be used to display policy execution state and trends.

[realshuting]

These could be high-level considerations:

• Total count of violations
• Number of violations on audit mode
• Number of violations of blocked resources (enforce mode)
• Total count of successful policy application
• Number of audit policies
• Time to evaluate the request
• Time to process a rule (mutate/validate/generate)
• Time to add a new policy
• Time to process policy vs resource

This is just an initial proposal. @JimBugwadia @shivdudhani Please add to the list if you have valuable cases.

[dirien]

hi, i would suggest to support ServiceMonitor / PodMonitor of the prometheus operator.

[yashvardhan-kukreja]

@realshuting , I don't think, as a part of this issue, we'd have to programme anything for grafana per se.
The way it seems to me, we just have to code a new custom exporter in kyverno which exposes the metrics you listed above on /metrics

Then, while using grafana, we just have to make it use the prometheus datasource which would be storing kyverno's metrics (exported in the first place by the custom exporter)

Please correct me if I am wrong :)

[phoenixking25]

I think this task will also include thanos to make metrics globally available at a central place to present on Grafana.

[JimBugwadia]

We should think about the use cases for different metric sets. I see at least 2 user stories:

• I am a cluster admin and want to see if policy rules are executing correctly and efficiently. I need to provide these reports for compliance and also report things like how many API requests were blocked.
• I am a cluster admin and want to see violation trends across namespaces.
• I am a workload / namespace admin and want to see metrics and trends for my namespace.

There are likely others...

Based on these concerns / use cases we can determine the appropriate metrics and dashboards / reports.

[JimBugwadia]

See: https://github.com/fjogeleit/policy-reporter - this may help address some of the use cases mentioned.

[realshuting]

It would be good to have these metrics https://kubernetes.slack.com/archives/CLGR9BJU9/p1614050069016200?thread_ts=1614025217.009600&cid=CLGR9BJU9 to help tune the memory usage.

[yashvardhan-kukreja]

@realshuting I guess we can close this one as well now

[realshuting]

Closed via #1877.