gateway-api: Expose metric port setting

[sayboras]

Description

This is to make sure that users can disable and set port number for Gateway API controller accordingly.

Fixes: #23082
Reported-by: Karsten Nielsen karsten.nielsen@ingka.ikea.com
Signed-off-by: Tam Mach tam.mach@cilium.io

Tasks

• Provide the way to configure metric port
  • sensible default number
  • disable metric server by convention (e.g. using value as 0)
• Update related docs https://docs.cilium.io/en/latest/operations/system_requirements/

Testing

Testing was done locally as per below

Installation with default value

docker@minikube:~$ sudo lsof -i :9965   
COMMAND      PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
cilium-op 206673 root   10u  IPv6 2432329      0t0  TCP *:9965 (LISTEN)

Installation with metric port disabled

docker@minikube:~$ sudo lsof -i TCP | grep cilium-op
cilium-op 213975   root    3u  IPv4 2457566      0t0  TCP localhost:9891 (LISTEN)
cilium-op 213975   root    7u  IPv4 2457570      0t0  TCP minikube:44918->minikube:8443 (ESTABLISHED)
cilium-op 213975   root    8u  IPv4 2466675      0t0  TCP localhost:9234 (LISTEN)
cilium-op 213975   root    9u  IPv4 2474711      0t0  TCP minikube:51888->minikube:8443 (ESTABLISHED)
cilium-op 213975   root   10u  IPv6 2477908      0t0  TCP *:37571 (LISTEN)

[qmonnet]

Looks good, thank you!

[squeed]

These are hostNetwork ports, which are a somewhat limited resource. Is there a reason the gateway controller uses a separate metrics port despite being in the same process?

FWIW the PR looks fine, just curious.

[gandro]

While I think that metricsPort is the better name, all most other components (agent, operator, proxy, relay) use prometheus.port for the name, e.g. gatewayAPI.prometheus.port. Something to consider, I don't have strong feelings about this (and there are other agent metrics like Hubble (not relay) which do not use prometheus.port either).

[gandro]

Please add this port to the metrics service as well, so they can be discovered via the ServiceMonitor infrastructure (though it seems that for the proxy, the infrastructure might also be incomplete).

https://github.com/cilium/cilium/blob/d914e6cd50965446fca04e71d4e199f3887c5872/install/kubernetes/cilium/templates/cilium-agent/service.yaml

[tklauser]

LGTM for @cilium/operator owned files.

[youngnick]

LGTM in service mesh terms, but I also wondered about why it needs to be a separate port.

[gandro]

Regarding the multiple ports - my understanding here is that the metrics are exposed by a third-party library ("sigs.k8s.io/controller-runtime"), thus we are just exposing that. There is precedent for doing that: The envoy metrics of the Cilium pod are also exposed in a separate port.

The Hubble and Cilium-Agent metrics are also exposed on a different port, but the reason there is not technical, but more use-case driven: Having them on separate ports allows them to be scraped by different consumers (the Hubble metrics will be more interesting to app teams, while the Cilium metrics will be interesting to platform teams).

Having said that, the gateway-api and envoy metrics will likely have the same consumers as the regular cilium-agent metrics, i.e. platform teams. So I think it's worth having a discussion if it makes sense to expose then under a common port. But it would likely require us to implement some kind of gather functionality inside the agent itself, which I don't think I think is a larger undertaking that will not make it into v1.13.

[squeed]

While the controller-runtime supports starting up an independent metrics server, there's no need to use that. You can also access the registry separately via sigs.k8s.io/controller-runtime/pkg/metrics.Registry and attach it to the existing metrics server.

Or just overwrite it with a separate registry.

[dylandreimerink]

LGTM from a datapath perspective.

I would personally prefer to fold these metrics into the main Cilium metrics registry, having multiple ports for metrics seems overkill. But I will let the other people already in this deal with that.

[sayboras]

While the controller-runtime supports starting up an independent metrics server, there's no need to use that. You can also access the registry separately via sigs.k8s.io/controller-runtime/pkg/metrics.Registry and attach it to the existing metrics server. Or just overwrite it with a separate registry.

This is good point, I didn't know that Registry is global variable in controller package.

[sayboras]

Closed in favour of #23501