Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pause containers getting oom-killed #1975

Open
mcluseau opened this issue Dec 19, 2018 · 7 comments
Open

pause containers getting oom-killed #1975

mcluseau opened this issue Dec 19, 2018 · 7 comments

Comments

@mcluseau
Copy link

@mcluseau mcluseau commented Dec 19, 2018

Hi,

We see issues with pause container's processes being killed by the oom_reaper when the pod limits are low:

[ 6690.590220] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[ 6690.590797] [157294]     0 157294    21430      478      12       4        0             0 conmon
[ 6690.590802] [157486]     0 157486      256        1       4       2        0          -998 pause
[ 6690.591368] [387940]     0 387940    21430       39      12       3        0             0 conmon
[ 6690.591370] [387941]     0 387941   137779     3339      25       4        0             0 runc
[ 6690.591373] [387951]     0 387951   119281     1724      22       4        0          -998 runc:[2:INIT]
[ 6690.591376] Memory cgroup out of memory: Kill process 387941 (runc) score 398 or sacrifice child
[ 6690.591403] Killed process 387951 (runc:[2:INIT]) total-vm:477124kB, anon-rss:3000kB, file-rss:3896kB, shmem-rss:0kB
[ 6690.591992] oom_reaper: reaped process 387951 (runc:[2:INIT]), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Steps to reproduce the issue:

kubectl create -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  containers:
  - image: alpine
    imagePullPolicy: IfNotPresent
    name: test
    stdin: true
    tty: true
    resources:
      limits:
        memory: 10Mi
      requests:
        memory: 10Mi
EOF

Additional information you deem important (e.g. issue happens only occasionally):

It also means that previous limits set on pods, when they are tight, will be triggered when migrating to cri-o.

Output of crio --version:

crio version 1.13.0
commit: 56d7d9a0750d7deb06182361837b690683f13dfe

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical.

@mrunalp

This comment has been minimized.

Copy link
Member

@mrunalp mrunalp commented Dec 31, 2018

Is this behavior different than what you see with docker?

@mcluseau

This comment has been minimized.

Copy link
Author

@mcluseau mcluseau commented Dec 31, 2018

we were on containerd before, and tight limits like that were set and working. They don't work anymore. Example use case: serving single-page apps with nginx uses very little memory. We used to set the limit at 20MiB, now we set it at 32MiB.

I wonder what's best thought, between "hiding" this cost to be compatible or making the cost explicit and have that documented in some migration notes.

@rhatdan

This comment has been minimized.

Copy link
Contributor

@rhatdan rhatdan commented Mar 18, 2019

@mcuseau @mrunalp is this still an issue? Is it something we need to fix?

@mcluseau

This comment has been minimized.

Copy link
Author

@mcluseau mcluseau commented Mar 19, 2019

@rhatdan I can't confirm or infirm that as I had to switch the cluster back to containerd for now. Is there something that could have solved this issue as a side effect?

@rhatdan

This comment has been minimized.

Copy link
Contributor

@rhatdan rhatdan commented Mar 19, 2019

@mcluseau I am not sure. I am just trying to clean up issues and see if they have been fixed or bring them to the attention of developers.

@mrunalp

This comment has been minimized.

Copy link
Member

@mrunalp mrunalp commented Mar 20, 2019

I opened #2154 to partly help with this. @mcluseau do you know if your container shim process is running in a cgroup or not with containerd? We always run conmon under pod cgroup so it gets accounted to pod.

@mcluseau

This comment has been minimized.

Copy link
Author

@mcluseau mcluseau commented Mar 21, 2019

Hi @mrunalp , no the shim is not accounted in the pod with containerd. It's what I called the "hidden cost" earlier.

ps aux (+manual filtering):

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      752231  0.0  0.0 107360  8516 ?        Sl   Mar20   0:02 containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/34e...494 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd

pstree (+ manual filtering) :

        |-dkl(1946)-+-containerd(1966)-+-containerd-shim(752231)-+-pause(752271)
        |           |                  |                         |-{containerd-shim}(752232)
        |           |                  |                         |-{containerd-shim}(752233)
        |           |                  |                         |-{containerd-shim}(752234)
        |           |                  |                         |-{containerd-shim}(752235)
        |           |                  |                         |-{containerd-shim}(752236)
        |           |                  |                         |-{containerd-shim}(752237)
        |           |                  |                         |-{containerd-shim}(752238)
        |           |                  |                         |-{containerd-shim}(752240)
        |           |                  |                         |-{containerd-shim}(752458)
        |           |                  |                         `-{containerd-shim}(761890)

find /sys/fs/cgroup/ -name tasks |xargs grep 752231 (+manual filtering):

/sys/fs/cgroup/openrc/dkl-user-services/tasks:752231
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.