pkg/endpoint: calculate Kube API-Server lag from pod events

[aanm]

Since Cilium receives CNI events when a pod is created, Cilium can
calculate the lag for kube-apiserver events by checking the time an
ADD event for that Pod was received and subtracting by the time the CNI
event for that pod was received.

Signed-off-by: André Martins andre@cilium.io

Fixes: #13679

Add metric 'cilium_k8s_event_lag_seconds' for calculated lag of Kubernetes events

[aanm]

test-me-please

[christarazi]

LGTM, a few minor changes.

[christarazi]

Have you run this through promtool?

[christarazi]

Also, I think we can add more context to the description of this metrics. Since this is the lag between receiving the CNI ADD and then getting it in the cache, let's add that so it is clear for readers.

[aanm]

I haven't but our smoke tests do :)

[aanm]

test-me-please

[christarazi]

Let's update this description per my other comment as soon as this passes CI.

[christarazi]

(Actually we might want to rename the variable if we update the description to be specific to CNI ADD -> pod cache lag time.)

[joestringer]

Yeah, I think it would be nice to make it more specific than just k8s_event_lag_seconds, for example k8s_pod_create_latency_seconds or something?

Also, how exactly do restored endpoints impact the metric? I get the impression that the time is set during restore but I don't follow exactly how the metric might be updated/influenced by restored endpoints.

Otherwise LGTM.

[aanm]

Yeah, I think it would be nice to make it more specific than just k8s_event_lag_seconds, for example k8s_pod_create_latency_seconds or something?

Also, how exactly do restored endpoints impact the metric? I get the impression that the time is set during restore but I don't follow exactly how the metric might be updated/influenced by restored endpoints.

Otherwise LGTM.

@joestringer I think it's still fair to assume the latency of Kubernetes starts to "count" as soon we restore an endpoint.

[pchaigno]

But the K8s watcher won't receive anything for the restored endpoints because the corresponding pods already exist, right? So the create time for restored endpoints probably doesn't matter unless we have a very serious lag and Cilium manages to restart before the k8s event is received.

[aanm]

But the K8s watcher won't receive anything for the restored endpoints because the corresponding pods already exist, right? So the create time for restored endpoints probably doesn't matter unless we have a very serious lag and Cilium manages to restart before the k8s event is received.

The watchers receives all pods regardless of the state of the local endpoints. I didn't check but I think the watcher receives the pods before we restore.

[pchaigno]

But the K8s watcher won't receive anything for the restored endpoints because the corresponding pods already exist, right? So the create time for restored endpoints probably doesn't matter unless we have a very serious lag and Cilium manages to restart before the k8s event is received.

The watchers receives all pods regardless of the state of the local endpoints. I didn't check but I think the watcher receives the pods before we restore.

Indeed, we are blocking on the initial update of pods from K8s in InitK8sSubsystem() before continuing with the restoration of endpoints.