Controller Group Metrics

[asauber]

Controller Group Metrics

Add a ControllerGroup for all controllers. ControllerGroup is used to
aggregate controllers which currently have a dynamic name, for example,
per-Endpoint and per-CiliumNetworkPolicy controllers.

ControllerGroups are used to expose a success/failure counter
Prometheus metric for each group. Each ControllerGroup also has a flag
which indicates if the metric should be enabled by default.

A config option is added which allows the user to specify a set of
controller group name strings, which override the default set of
controller metrics which are enabled. The special values of "all" and
"none" are supported.

Example Usage

With controller-group-metrics: "all"

root@kind-control-plane:/home/cilium# curl localhost:9968/metrics 2>/dev/null | grep controllers_group
# HELP cilium_controllers_group_runs_total Number of times that a controller group was run labeled by completion status and controller name
# TYPE cilium_controllers_group_runs_total counter
cilium_controllers_group_runs_total{group_name="bpf-map-sync",status="success"} 29
cilium_controllers_group_runs_total{group_name="cilium-health",status="success"} 3
cilium_controllers_group_runs_total{group_name="dns-garbage-collector-job",status="success"} 3
cilium_controllers_group_runs_total{group_name="endpoint-gc",status="success"} 1
cilium_controllers_group_runs_total{group_name="initial-global-identities",status="success"} 3
cilium_controllers_group_runs_total{group_name="ipcache-inject-labels",status="failure"} 12
cilium_controllers_group_runs_total{group_name="ipcache-inject-labels",status="success"} 14
cilium_controllers_group_runs_total{group_name="k8s-heartbeat",status="success"} 5
cilium_controllers_group_runs_total{group_name="link-cache",status="success"} 10
cilium_controllers_group_runs_total{group_name="metricsmap-bpf-prom-sync",status="success"} 30
cilium_controllers_group_runs_total{group_name="resolve-identity",status="success"} 1
cilium_controllers_group_runs_total{group_name="resolve-labels",status="success"} 3
cilium_controllers_group_runs_total{group_name="restore-endpoint-identity",status="success"} 4
cilium_controllers_group_runs_total{group_name="sync-host-ips",status="success"} 3
cilium_controllers_group_runs_total{group_name="sync-lb-maps-with-k8s-services",status="success"} 1
cilium_controllers_group_runs_total{group_name="sync-policymap",status="success"} 15
cilium_controllers_group_runs_total{group_name="sync-to-k8s-ciliumendpoint",status="success"} 76
cilium_controllers_group_runs_total{group_name="sync-utime",status="success"} 3
cilium_controllers_group_runs_total{group_name="write-cni-file",status="success"} 1

With controller-group-metrics: "sync-policymap sync-host-ips restore-endpoint-identity"

root@kind-worker:/home/cilium# curl localhost:9968/metrics 2>/dev/null | grep group_runs_total
# HELP cilium_controllers_group_runs_total Number of times that a controller group was run labeled by completion status and controller name
# TYPE cilium_controllers_group_runs_total counter
cilium_controllers_group_runs_total{group_name="restore-endpoint-identity",status="success"} 1
cilium_controllers_group_runs_total{group_name="sync-host-ips",status="success"} 4
cilium_controllers_group_runs_total{group_name="sync-policymap",status="success"} 8

With controller-group-metrics: "none all sync-policymap sync-host-ips restore-endpoint-identity"

root@kind-worker:/home/cilium# curl localhost:9968/metrics 2>/dev/null | grep group_runs_total
root@kind-worker:/home/cilium#

Release note:

Add per-controller success/failure count metrics and a config option for these

[asauber]

/test

[learnitall]

There was one s/option/flag that was missed, but other than that lgtm. Thanks!

[asauber]

/test

[asauber]

Hit known flake #27431

[asauber]

/test

[asauber]

rebased again to pick up fix for #27431

[asauber]

/test