metrics: add a metric for max observed endpoint ifindex

[asauber]

Adds a metric endpoint_max_ifindex, which is the current maximum
interface index for all endpoints. The metric is updated during the
periodic invocation of syncHostIPs.

This metric can be used to determine if the interface index for the next
Pod will exceed Cilium's limit of max(uint16) on older kernels. On
kernels which do not provide ifindex via the FIB, Cilium must store the
ifindex in the CT map, and 16 bits are reserved per entry for this
purpose. This presents a limit of 65535 lifetime interfaces, which can
be approached quickly with sufficient pod churn.

Users who are subject to this limitation (typically a kernel version
less than 5.10), are advised to reboot or roll the host in this case.

Its default-enabled status is dynamic. On kernels which provide ifindex
via the FIB, the metric is disabled (since Cilium's ifindex limits are
not subject to max(uint16) in this case).

It can be enabled with the following Helm values:

prometheus:
  enabled: true
  metrics:
    - +cilium_endpoint_max_ifindex

On kernels which do not provide ifindex via the FIB, the metric is enabled by default.

Dynamic default based on Kernel version

On Linux 4.19 without metric explicitly enabled

$ cat contrib/testing/kind-values.yaml | grep -A 1 prom
prometheus:
  enabled: true

$ curl 2>/dev/null 172.18.0.4:9962/metrics | grep ifindex
# HELP cilium_endpoint_max_ifindex Maximum interface index observed for existing endpoints
# TYPE cilium_endpoint_max_ifindex gauge
cilium_endpoint_max_ifindex 7

$ uname -r
kind-4.19

On Linux 6.4 without metric explicitly enabled

$ cat contrib/testing/kind-values.yaml | grep -A 1 prom
prometheus:
  enabled: true
  
$ curl 2>/dev/null 172.19.0.3:9962/metrics | grep ifindex

$ 

On Linux 6.4 with metric explicitly enabled

$ cat contrib/testing/kind-values.yaml | grep -A 3 prom
prometheus:
  enabled: true
  metrics:
    - +cilium_endpoint_max_ifindex

$ curl 2>/dev/null 172.19.0.3:9962/metrics | grep ifindex
# HELP cilium_endpoint_max_ifindex Maximum interface index observed for existing endpoints
# TYPE cilium_endpoint_max_ifindex gauge
cilium_endpoint_max_ifindex 11

$ uname -r
6.4.12-arch1-1

Longer-running example

Roll a pod or two

$ k delete pod -n kube-system          coredns-5d78c9869d-5w8pc
pod "coredns-5d78c9869d-5w8pc" deleted

and the ifindex increases

$ curl 172.19.0.2:9962/metrics 2>/dev/null | grep ifindex
# HELP cilium_endpoint_max_ifindex Maximum interface index observed for existing endpoints
# TYPE cilium_endpoint_max_ifindex gauge
cilium_endpoint_max_ifindex 43

Addresses: #17259
Addresses: #16260

metrics: add a metric for max observed endpoint ifindex

[maintainer-s-little-helper]

Commit 5ec7915 does not match "(?m)^Signed-off-by:".

Please follow instructions provided in https://docs.cilium.io/en/stable/contributing/development/contributing_guide/#developer-s-certificate-of-origin

[asauber]

The metric was tested on Linux 4.19 interactively using an lvh VM:
https://docs.cilium.io/en/stable/contributing/testing/e2e/#running-tests-in-vm

After bringing up the VM:

On the local machine

# build new container images for kind
make kind-image

# get our recently built cilium image onto the VM
docker save -o ciliumdev.tar localhost:5000/cilium/cilium-dev:local
scp -P 2224 ciliumdev.tar root@localhost:/host/cilium

# get our recently built cilium operator image onto the VM
docker save -o ciliumop.tar localhost:5000/cilium/operator-generic:local

# get cilium CLI on to the VM
scp -P 2224 $(which cilium) root@localhost:/host/ciliumcli

On the VM

alias cilium=/host/ciliumcli
alias k=kubectl

# load Cilium image
docker load -i ciliumdev.tar 
kind load docker-image localhost:5000/cilium/cilium-dev:local

# load Cilium operator image
docker load -i ciliumop.tar
kind load docker-image localhost:5000/cilium/operator-generic:local

cilium uninstall
cilium install \
	--chart-directory=/host/cilium/install/kubernetes/cilium \
	--helm-values=/host/cilium/contrib/testing/kind-values.yaml \
	--version=

k get pods -A -owide

[asauber]

Upon the first run of CI for this PR, an import cycle was detected. This commit fixes that import cycle, with details in the commit message:

9c4a431

[asauber]

/test

[borkmann]

@asauber could we add backport 1.14 label given the ifindex fix went also in there?

[asauber]

@derailed yes, this can be used to implement an alert to determine if the cluster is nearing the max ifindex of 65535 (if on an affected kernel version)

The current suggested mitigation in that case is to roll or reboot the Node.

[christarazi]

Thanks for the PR!

A few comments below. One other thing is that I don't see any commit descriptions for the commits and all the context is instead in the PR. As a project, we are not opposed to a PR description, but we state in our contrib guide that PRs should contain commits with a description. Plus, this is beneficial when bisecting / git blame.

[learnitall]

Looks great on docs! Thank you for adding the description as to the dynamic default status, awesome stuff.

[asauber]

/test