fix container_oom_events_total always returns 0.

[chengjoey]

fix #3015
In a Kubernetes pod, if a container is OOM-killed, it will be deleted and a new container will be created. Therefore, the container_oom_events_total metric will always be 0. this pr refactor the collector of oom events, and retain the deleted container oom information for a period of events. And add flag oom_event_retain_time to decide how long the oom event will be keep, default is 5 minutes

[k8s-ci-robot]

Hi @chengjoey. Thanks for your PR.

I'm waiting for a google member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[chengjoey]

/assign @iwankgb @kragniz

Is it feasible to keep the container metrics with oomkilled without deleting them?
please take a look

[chengjoey]

/kind bug

[iwankgb]

What happens when PID 1 forks another process and the forked process get OOM-killed?

[chengjoey]

What happens when PID 1 forks another process and the forked process get OOM-killed?

The forked process that was OOM-killed can still read relevant log information from /dev/kmsg. It should still be possible to associate with the corresponding container.

[szuecs]

@chengjoey what happens if a container is killed every second?
Do I understand correctly that we keep creating new container metrics or would this counter increase to >1?
In case we would create new container metrics, then I would call this feature a memory leak, because every container start will create a new set of metrics and now we would store them forever as far as I understand. (I am not very familiar with the code)

[ishworgurung]

memory leak [ ... ]

@szuecs In what way would it be a memory leak ?

[szuecs]

@ishworgurung maybe the wording is not correct, but it will increase memory overtime, which is never GCed and finally cadvisor get oom. As far as I understand.
Increasing the counter is great, though. Having old metrics forever is likely an issue.
Maybe it's also not part of this PR, feel free to ignore this.

[chengjoey]

@ishworgurung maybe the wording is not correct, but it will increase memory overtime, which is never GCed and finally cadvisor get oom. As far as I understand. Increasing the counter is great, though. Having old metrics forever is likely an issue. Maybe it's also not part of this PR, feel free to ignore this.

hi @szuecs @ishworgurung , I have made modifications in this PR, putting the oom event metric information in a separate map, and adding the flag oom_event_retain_time to configure the retention time. Oom metric that exceeds this time will still be deleted to prevent memory leaks.

@iwankgb could you please task a review when you have time

[dims]

/ok-to-test

[dims]

@chengjoey please resolve merge conflicts:

[chengjoey]

/test pull-cadvisor-e2e

[chengjoey]

@chengjoey please resolve merge conflicts:

Thanks @dims , pr has been rebased

[chengjoey]

/test pull-cadvisor-e2e

[chengjoey]

/test pull-cadvisor-e2e

[chengjoey]

/test pull-cadvisor-e2e

[k8s-ci-robot]

@chengjoey: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cadvisor-e2e	0b6dfeb	link	true	/test pull-cadvisor-e2e

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[nlamirault]

Hi, any news on this bugfix?
We're waiting for an alert on OOMKilled event (kubernetes-monitoring/kubernetes-mixin#822)
Thanks.

[taraspos]

Hi!
just wanted to bump this issue again. Would be great to get it fixed.

[tsipo]

Hi @pschichtel and others.
I was testing this issue on Kubernetes using a small image I have built - see here. I have noticed that when I use that tool in a forked mode - this and this use-cases, I go get container_oom_events_total == 1 as the container continues to live enough time after the OOMKill for cAdvisor to be scraped. If I change the AFTER_FORK_INTERVAL env var to 0, and the container exits immediately, I gon't get container_oom_events_total == 1 as the container is de-registered immediately from cAdvisor.
FYI.