ValidatingAdmissionPolicy: fixes to metrics

[jiahuif]

What would you like to be added?

As of 1.3, ValidatingAdmissionPolicy metrics are in alpha stability [1]. Currently the metrics have following problems:

• there is no way to count total errors, the current implementation counts only "error but failurePolicy=Ignore" but not denials caused by errors;
• there is no way to differentiate errors, especially, between transient errors like "params not ready" and persistent errors like runtime errors;
• the current implementation does not record the evaluation duration for acceptance or denial due to errors. This issue makes any analysis of policy performance impossible.

This issue purpose the following enhancements to the metrics:

• add a label of error to check_total and check_duration_seconds
  • the values are enum'ed with each possible error, namely RUNTIME, PARAMS, INTERNAL. The label can be empty for no errors;
• add error handling to ObserveRejection, to record any denials coming from errors;
• add error reporting to all places where ObserveRejection is called;

References:

1. https://github.com/kubernetes/kubernetes/blob/v1.30.0-beta.0/staging/src/k8s.io/apiserver/pkg/admission/plugin/policy/validating/metrics/metrics.go

Why is this needed?

resolve the issues mentioned above.

Let's discuss the beta graduation of the metrics in another issue.

[jiahuif]

/assign
/sig api-machinery
cc @cici37

[jiahuif]

/sig instrumentation

[seans3]

/triage accepted

[dgrisonnet]

cc @logicalhan

[jiahuif]

cc @fedebongio