Adds metric for monitoring the service programming duration

[ovidiutirla]

Adds a new metric (service_implementation_delay ) for monitoring the service programming duration. The metrics measures the execution time of the service event handler reflecting the time it took to program the service. From the time the service or pod was changed to the time the change was propagated.

The metric also take into consideration the duration to program the in-cluster network (Network programming latency SLIs) which is useful to monitor and understand network bottlenecks in large clusters.

The metric should not impact the current performance of the handler allowing us to have it enabled by default.

Adds `service_implementation_delay` metric accounting the duration in seconds to propagate the data plane programming of a service, its network and endpoints from the time the service or the service pod was changed excluding the event queue latency

[dlapcevic]

Looks good. Thanks Ovidiu!

[derailed]

@ovidiutirla Thank you for this PR! I think we should perhaps revisit the naming here as service_programming is a little confusion since we are measure cilium k8s service event handling time.

[qmonnet]

/test

[qmonnet]

/test

[ovidiutirla]

Looks like all tests passed, I'll squash the commits and rebase on main.

[tommyp1ckles]

Nice! just one question

[squeed]

The documentation says, "Duration in seconds to propagate the programming of a service [...] from the time the service was changed." The code as-written ignores queue latency. Is that desired? I assume not. I'd suggest setting the "start" timestamp in ServiceCache.emitEvent() instead. Or, update the documentation to indicate it is dataplane programming latency, not end-to-end latency.

Ideally we could determine the original enqueue time in the Resource queue itself (thoughts, @joamaki), but that's overkill for this PR.

[qmonnet]

/test

[qmonnet]

Nit: the update to pkg/metrics/metrics.go could all go into the first commit, rather than having the description fixed in the second commit. But that's not blocking.

Looks good from my side, thanks!

[qmonnet]

Sorry, there was a bit of fat-fingering on the phone UI on my end for the auto-merge. The close/reopen was to re-trigger the image build.
/test

[derailed]

@ovidiutirla Thank you for the updates!

[aanm]

/test

[aanm]

/test