metric: Avoid memory leak/increase in cilium-agent

[sayboras]

Description

This commit is to make sure that the processed item in pod deletion queue is removed by explicitly call Done() function as per suggestion in godoc¹ .

The impact of not having this change will be increasing of memory in cilium agent when the hubble metrics are enabled. This might take days (if not weeks) to observe in a normal Cilium deployment due to low number of Pod deletion events (i.e. in high churn environment, the memory will be increasing in a faster pace).

Testing is done before and after the changes as per below.

Sample workload to simulate high number of pod deletion events

apiVersion: batch/v1
kind: Job
metadata:
  name: pod-churn-job
spec:
  completions: 50000000
  parallelism: 100
  template:
    metadata:
      labels:
        app: pod-churn-job
    spec:
      containers:
      - name: churn-app
        image: sandeshkv92/highpodchurn:linux_amd64
      restartPolicy: Never

Before this change, the cilium agent memory keeps increasing from 150MB to ~500MB in less than 3 hours, while with the same workload configured and this change, the memory is quite stable for a longer period (e.g. 5 hours).

Fixes: 782f934

Testing

Testing was done with the above workload, please refer to the below dashboard for visualization.

Before

After

Footnotes

1. https://pkg.go.dev/k8s.io/client-go@v0.29.3/util/workqueue#Type.Get ↩

[sayboras]

/test

[sayboras]

/test

[michi-covalent]

After dashboard seems to be missing 👀

[sayboras]

After dashboard seems to be missing 👀

Thanks, I have updated the PR description, will let it run for some more time before marking this ready to review.