-
Notifications
You must be signed in to change notification settings - Fork 18.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
{{.State.OOMKilled}} not being correctly set for kernel 4.19+ #38352
Comments
Thanks for reporting; I think the actual detection of the OOM is performed by containerd, which should send this information back to moby/libcontainerd/client_daemon.go Lines 820 to 826 in 46368fd
moby/libcontainerd/client_daemon.go Lines 864 to 868 in 46368fd
@crosbymichael @dmcgowan any ideas? |
Are you also able to reproduce this on Docker 18.09 (just in case it got addressed) ? |
Thanks for taking a look! I just upgraded after your message and docker-ce 18.09 shows the same behavior:
This OOM killer patch was fairly recent (went into the kernel released in Oct. 2018 if I am not mistaken), so I wasn't expecting 18.09 to have the fix anyway. |
Thanks for checking! |
Had a quick chat with the containerd maintainers, and likely some changes will have to be made in the containerd/cgroups package, so I copied the information from this issue there; containerd/cgroups#74 |
This has been fixed in version 4.20.2-1. The issue can be closed. Thanks! |
Thanks! Closing 🤗 |
Description
Hello,
Apologies if the reported bug is a duplicate of another issue. I tried searching through the issues but didn't find anything similar.
There were some improvements made to OOM killer in kernel 4.19 to make it more 'cgroup aware'. You can find the relevant commits under [1] in the additional information section. After the change went in, Docker isn't setting the
.State.OOMKilled
flag correctly despite OOM killer being invoked.To reproduce this, I have created a sample image named
fakhan/sl7:oom-test
. The entrypoint of the image is a program that consumes around 1024MB of memory. Creating this container with anything lower should trigger OOM killer. I have provided more information below on the bug.BUG REPORT INFORMATION shown below:
Steps to reproduce the issue:
I have posted the snippets from
/var/log/messages
for both these instances under the additional information section.Describe the results you received:
As shown above, OOMKilled evaluates to 'False' on kernel 4.19
Describe the results you expected:
OOMKilled should have evaluated to 'True' on kernel 4.19 just like the on the previous versions
Additional information you deem important (e.g. issue happens only occasionally):
[1]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=dc0b58643aff8b378086f25cce6789ccba68cbcb
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5989ad7b5ede38d605c588981f634c08252abfc3
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3d8b38eb81cac81395f6a823f6bf401b327268e6
[2]
/var/log/messages
snippets are as follows. First for the older kernelThen for the newer kernel
Output of
docker version
:Docker version on both machines is the same (18.06)
Output of
docker info
:Additional environment details (AWS, VirtualBox, physical, etc.):
Nothing special about the environment. I am running these commands on two bare metal boxes with different kernels.
The text was updated successfully, but these errors were encountered: