feat: Add the http return code to metric api_processed_total

[vipul-21]

Adding the return code to the cilium_api_limiter_processed_requests_total based on the error/success of the API call. It helps in filtering the metric based on http code as labeloutcome of fail is generic.
Tested the changes via grafana and prometheus:

[vipul-21]

Couple of things I need feedback on:

1. In case of upgrade of cilium image, the old metrics will not have return_code. So during the transition phase, we will be return_code as empty for old cilium agents. Should we return a default code (unknown) in case of that ? or filter based on the labeloutcome similar to what I added for etcd package.
2. Error code for failure is 500 in the PR based on labeloutcome. Is that okay ?

[lambdanis]

One request, can you document the change in the Changed Metrics in Documentation/operations/upgrade.rst?

In case of upgrade of cilium image, the old metrics will not have return_code. So during the transition phase, we will be return_code as empty for old cilium agents. Should we return a default code (unknown) in case of that ?

I'm not sure I understand. You mean to backport the default code to older Cilium versions? I think it's fine if the new agents just start exposing this new label.

[chancez]

Seems reasonable.

[lambdanis]

docs good, thanks!

[vipul-21]

/test

[marseel]

Looks reasonable for kvstore.

I think that it was actually a wrong decision for us to reuse api_limiter for kvstore package, but it was the closest thing that we had to implement logic we needed at that time 😿

[julianwiedmann]

/test

[vipul-21]

@julianwiedmann I want to backport this to cilium version 1.13 and above.
I have a draft PR for v1.13, but can you help adding the backport labels for 1.14 and 1.15 ?
slack conversation: https://cilium.slack.com/archives/C2B917YHE/p1712076138237509

[julianwiedmann]

As @joestringer noted in the slack thread:

the backporting process and critera are here: https://docs.cilium.io/en/stable/contributing/release/backports/#backport-process . At a glance it looks like this is typically something we wouldn't bother backporting and rather include in a newer release.

[joestringer]

@tamilmani1989 pointed out that this could be useful for debugging, so in line with the "Debug tool improvements" reason in backporting I think it's reasonable to at least backport to the current stable release.