kubelet: expose OOM metrics

[jonkerj]

What type of PR is this?

/kind feature

What this PR does / why we need it:

cAdvisor has code to expose the OOM metric container_oom_events_total since 0.40.0, which is included in kubelet since 1.23.0, but this was not activated in Kubelet coede.

Which issue(s) this PR fixes:

Fixes # (n/a, if needed, let me know and I'll raise a ticket)

Special notes for your reviewer:

My test environment is Talos 0.14.x, with k8s 1.23.3, so I tested it by:

• cherry-picking this commit onto v1.23.3
• make WHAT=cmd/kubelet from golang:1.17-buster
• monkey-patch resulting binary into an image based on talos-systems/kubelet
• Result is docker.io/jonkerj/talos-kubelet:0.0.2

The nodes running that specific Kubelet expose the metric, and I see the metric increasing if I go beyond memory resource limits (using stress-ng).

Does this PR introduce a user-facing change?

Add the metric `container_oom_events_total` to kubelet's cAdvisor metric endpoint.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

Welcome @jonkerj!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @jonkerj. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wzshiming]

/ok-to-test
Can we have a test that this works?

[jonkerj]

/test pull-kubernetes-e2e-gce-ubuntu-containerd

[jonkerj]

Can we have a test that this works?

I'll amend my PR to expand test/e2e_node/resource_metrics_test.go to include tests for the existence of this metric. I think testing for actual OOMs getting counted (ie, launch a well crafted pod eating memory) will be very convoluted and even error prone (it should use a well known image and it should consistently trip OOM after the init phase), so I will write a check for container_oom_events_total yielding 0 .. 1e6.

[jonkerj]

/test pull-kubernetes-node-e2e-containerd

[jonkerj]

@wzshiming is there anything else I can do?

[wzshiming]

#79002
#84331

kubernetes/vendor/github.com/google/cadvisor/metrics/prometheus.go

Lines 1760 to 1769 in c6452be

	if includedMetrics.Has(container.OOMMetrics) {
	c.containerMetrics = append(c.containerMetrics, containerMetric{
	name: "container_oom_events_total",
	help: "Count of out of memory events observed for the container",
	valueType: prometheus.CounterValue,
	getValues: func(s *info.ContainerStats) metricValues {
	return metricValues{{value: float64(s.OOMEvents), timestamp: s.Timestamp}}
	},
	})
	}

I looked at PR of the metric that was added last time, and also looked at the context and it does work, look be fine.

/lgtm

/assign @klueska
for kubelet approval

[wzshiming]

Can we have a test in e2e?

[jonkerj]

Can we have a test in e2e?

I think testing for actual OOMs getting counted (ie, launch a well crafted pod eating memory) will be very convoluted and even error prone (it should use a well known image and it should consistently trip OOM after the init phase), so I included a test for metric presence in e2e_node before (see earlier comment).

If more is needed, I'd like to have suggestions on what to do, because I honestly would not know where to start.

[ehashman]

/priority backlog
/triage accepted
/lgtm

/assign @dashpole
does this seem reasonable?

[dashpole]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jonkerj, mrunalp, wzshiming

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/kubelet/OWNERS [mrunalp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-triage-robot]

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

• The PR does have any do-not-merge/* labels
• The PR does not have the needs-ok-to-test label
• The PR is mergeable (does not have a needs-rebase label)
• The PR is approved (has cncf-cla: yes, lgtm, approved labels)
• The PR is failing tests required for merge

You can:

• Review the full test history for this PR
• Prevent this bot from retesting with /lgtm cancel or /hold
• Help make our tests less flaky by following our Flaky Tests Guide

/retest

[krmayankk]

i think cadvisor exposes a ton of metrics, not all are exposed through kubelet. should we in future consider exposing cadvisor metrics using a config file passed to kubelet ?