Alert when a single pod is using up the majority of CPU or memory of a node.

[vijay-veeranki]

Related to:

#4491

Create an alert when a single pod is using up the majority of CPU or memory of a node.

The following query returns per-container average number of CPUs used during the last 5 minutes:

rate(container_cpu_usage_seconds_total{container!~"POD|"}[5m])
The lookbehind window in square brackets (5m in the case above) can be changed to the needed value. See possible time duration values here.

The container!~"POD|" filter removes metrics related to cgroups hierarchy (see this answer for more details) and metrics for e.g. pause containers (see these docs).

Since each pod can contain multiple containers, then the following query can be used for returning per-pod average number of CPUs used during the last 5 minutes:

sum(
rate(container_cpu_usage_seconds_total{container!~"POD|"}[5m])
) by (namespace,pod)

[sj-williams]

Findings doc:
https://docs.google.com/document/d/1qAxCYFzDQta00l4v3IZ1CUyjOECuWAXDh3oAqNF0UtA/edit#heading=h.sdps58w53s1e

[sj-williams]

TODO:
Pair with AP team to get rshiny app stood up in test cluster to simulate memory leak issues

[SteveMarshall]

We're going to try to recreate the cause of this in #5251. Once that's done, we'll revisit this.