Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

application crash due to k8s 1.9.x open the kernel memory accounting by default #61937

Open
wzhx78 opened this Issue Mar 30, 2018 · 76 comments

Comments

Projects
None yet
@wzhx78
Copy link

wzhx78 commented Mar 30, 2018

when we upgrade the k8s from 1.6.4 to 1.9.0, after a few days, the product environment report the machine is hang and jvm crash in container randomly , we found the cgroup memory css id is not release, when cgroup css id is large than 65535, the machine is hang, we must restart the machine.

we had found runc/libcontainers/memory.go in k8s 1.9.0 had delete the if condition, which cause the kernel memory open by default, but we are using kernel 3.10.0-514.16.1.el7.x86_64, on this version, kernel memory limit is not stable, which leak the cgroup memory leak and application crash randomly

when we run "docker run -d --name test001 --kernel-memory 100M " , docker report
WARNING: You specified a kernel memory limit on a kernel older than 4.0. Kernel memory limits are experimental on older kernels, it won't work as expected and can cause your system to be unstable.

k8s.io/kubernetes/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs/memory.go

-		if d.config.KernelMemory != 0 {
+			// Only enable kernel memory accouting when this cgroup
+			// is created by libcontainer, otherwise we might get
+			// error when people use `cgroupsPath` to join an existed
+			// cgroup whose kernel memory is not initialized.
 			if err := EnableKernelMemoryAccounting(path); err != nil {
 				return err
 			}

I want to know why kernel memory open by default? can k8s consider the different kernel version?

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT

Uncomment only one, leave it on its own line:

/kind bug
/kind feature

What happened:
application crash and cgroup memory leak

What you expected to happen:
application stable and cgroup memory doesn't leak

How to reproduce it (as minimally and precisely as possible):
install k8s 1.9.x on kernel 3.10.0-514.16.1.el7.x86_64 machine, and create and delete pod repeatedly, when create more than 65535/3 times , the kubelet report "cgroup no space left on device" error, when the cluster run a few days , the container will crash.

Anything else we need to know?:

Environment: kernel 3.10.0-514.16.1.el7.x86_64

  • Kubernetes version (use kubectl version): k8s 1.9.x
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
  • Kernel (e.g. uname -a): 3.10.0-514.16.1.el7.x86_64
  • Install tools: rpm
  • Others:
@qkboy

This comment has been minimized.

Copy link

qkboy commented Mar 30, 2018

Use below test case can reproduce this error:
first, make cgroup memory to be full:

# uname -r
3.10.0-514.10.2.el7.x86_64
# kubelet --version
Kubernetes 1.9.0
# mkdir /sys/fs/cgroup/memory/test
# for i in `seq 1 65535`;do mkdir /sys/fs/cgroup/memory/test/test-${i}; done
# cat /proc/cgroups |grep memory
memory  11      65535   1

then release 99 cgroup memory that can be used next to create:

# for i in `seq 1 100`;do rmdir /sys/fs/cgroup/memory/test/test-${i} 2>/dev/null 1>&2; done 
# mkdir /sys/fs/cgroup/memory/stress/
# for i in `seq 1 100`;do mkdir /sys/fs/cgroup/memory/test/test-${i}; done 
mkdir: cannot create directory ‘/sys/fs/cgroup/memory/test/test-100’: No space left on device <-- notice number 100 can not create
# for i in `seq 1 100`;do rmdir /sys/fs/cgroup/memory/test/test-${i}; done <-- delete 100 cgroup memory
# cat /proc/cgroups |grep memory
memory  11      65436   1

second, create a new pod on this node.
Each pod will create 3 cgroup memory directory. for example:

# ll /sys/fs/cgroup/memory/kubepods/pod0f6c3c27-3186-11e8-afd3-fa163ecf2dce/
total 0
drwxr-xr-x 2 root root 0 Mar 27 14:14 6d1af9898c7f8d58066d0edb52e4d548d5a27e3c0d138775e9a3ddfa2b16ac2b
drwxr-xr-x 2 root root 0 Mar 27 14:14 8a65cb234767a02e130c162e8d5f4a0a92e345bfef6b4b664b39e7d035c63d1

So when we recreate 100 cgroup memory directory, there will be 4 item failed:

# for i in `seq 1 100`;do mkdir /sys/fs/cgroup/memory/test/test-${i}; done    
mkdir: cannot create directory ‘/sys/fs/cgroup/memory/test/test-97’: No space left on device <-- 3 directory used by pod
mkdir: cannot create directory ‘/sys/fs/cgroup/memory/test/test-98’: No space left on device
mkdir: cannot create directory ‘/sys/fs/cgroup/memory/test/test-99’: No space left on device
mkdir: cannot create directory ‘/sys/fs/cgroup/memory/test/test-100’: No space left on device
# cat /proc/cgroups 
memory  11      65439   1

third, delete the test pod. Recreate 100 cgroup memory directory before confirm all test pod's container are already destroy.
The correct result that we expected is only number 100 cgroup memory directory can not be create:

# cat /proc/cgroups 
memory  11      65436   1
# for i in `seq 1 100`;do mkdir /sys/fs/cgroup/memory/test/test-${i}; done 
mkdir: cannot create directory ‘/sys/fs/cgroup/memory/test/test-100’: No space left on device

But the incorrect result is all cgroup memory directory created by pod are leaked:

# cat /proc/cgroups 
memory  11      65436   1 <-- now cgroup memory total directory
# for i in `seq 1 100`;do mkdir /sys/fs/cgroup/memory/test/test-${i}; done    
mkdir: cannot create directory ‘/sys/fs/cgroup/memory/test/test-97’: No space left on device
mkdir: cannot create directory ‘/sys/fs/cgroup/memory/test/test-98’: No space left on device
mkdir: cannot create directory ‘/sys/fs/cgroup/memory/test/test-99’: No space left on device
mkdir: cannot create directory ‘/sys/fs/cgroup/memory/test/test-100’: No space left on device

Notice that cgroup memory count already reduce 3 , but they occupy space not release.

@wzhx78

This comment has been minimized.

Copy link
Author

wzhx78 commented Mar 30, 2018

/sig container
/kind bug

@wzhx78

This comment has been minimized.

Copy link
Author

wzhx78 commented Mar 30, 2018

@kubernetes/sig-cluster-container-bugs

@feellifexp

This comment has been minimized.

Copy link

feellifexp commented Mar 30, 2018

This bug seems to be related: opencontainers/runc#1725

Which docker version are you using?

@qkboy

This comment has been minimized.

Copy link

qkboy commented Mar 30, 2018

@feellifexp with docker 1.13.1

@frol

This comment has been minimized.

Copy link

frol commented Mar 30, 2018

There is indeed a kernel memory leak up to 4.0 kernel release. You can follow this link for details: moby/moby#6479 (comment)

@wzhx78

This comment has been minimized.

Copy link
Author

wzhx78 commented Mar 31, 2018

@feellifexp the kernel log also have this message after upgrade to k8s 1.9.x

kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x8020)

@wzhx78

This comment has been minimized.

Copy link
Author

wzhx78 commented Mar 31, 2018

I want to know why k8s 1.9 delete this line if d.config.KernelMemory != 0 { in k8s.io/kubernetes/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs/memory.go

@feellifexp

This comment has been minimized.

Copy link

feellifexp commented Mar 31, 2018

I am not an expert here, but this seems to be change from runc, and the change was introduced to k8s since v1.8.
After reading the code, it seems it impacts cgroupfs cgroup driver, while systemd driver is not changed. But I did not test the theory yet.
Maybe experts from kubelet and container can chime in further.

@kevin-wangzefeng

This comment has been minimized.

Copy link
Member

kevin-wangzefeng commented Mar 31, 2018

/sig node

@k8s-ci-robot k8s-ci-robot added sig/node and removed needs-sig labels Mar 31, 2018

@kevin-wangzefeng

This comment has been minimized.

Copy link
Member

kevin-wangzefeng commented Mar 31, 2018

I want to know why k8s 1.9 delete this line if d.config.KernelMemory != 0 { in
k8s.io/kubernetes/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs/memory.go

I guess opencontainers/runc#1350 is the one you are looking for, which is actually an upstream change.

/cc @hqhq

@wzhx78

This comment has been minimized.

Copy link
Author

wzhx78 commented Mar 31, 2018

thanks @kevin-wangzefeng , the runc upstream had changed, I know why now , the change is `hqhq/runc@fe898e7 , but enable kernel memory accounting on root by default , the child cgroup will enable also, this will cause cgroup memory leak on kernel 3.10.0, @hqhq , is there any way to let us enable or disable kernel memory by ourself? or get the warning log when the kernel < 4.0

@hqhq

This comment has been minimized.

Copy link

hqhq commented Apr 1, 2018

@wzhx78 The root cause is there are kernel memory limit bugs in 3.10, if you don't want to use kernel memory limit because it's not stable on your kernel, the best solution would be to disable kernel memory limit on your kernel.

I can't think of a way to workaround this on runc side without causing issues like opencontainers/runc#1083 and opencontainers/runc#1347 , unless we add some ugly logic like do different things for different kernel versions, I'm afraid that won't be an option.

@wzhx78

This comment has been minimized.

Copy link
Author

wzhx78 commented Apr 1, 2018

@hqhq it's exactly kernel 3.10's bug, but we spent more time to found it and it brought us big trouble on production environment, since we only upgrade k8s version from 1.6.x to 1.9.x. In k8x 1.6.x version , it doesn't open the kernel memory by default since runc had if condition. but after 1.9.x, runc open it by default. we don't want others who upgrade the k8s 1.9.x version had this big trouble. And runc is popular container solution, we think it need to consider different kernel version, at least, if runc can report the error message in kubelet error log when the kernel is not suitable for open kernel memory by default

@wzhx78

This comment has been minimized.

Copy link
Author

wzhx78 commented Apr 2, 2018

@hqhq any comments ?

@hqhq

This comment has been minimized.

Copy link

hqhq commented Apr 2, 2018

Maybe you can add an option like --disable-kmem-limit for both k8s and runc to make runc disable kernel memory accounting.

@warmchang

This comment has been minimized.

Copy link
Contributor

warmchang commented Apr 3, 2018

v1.8 and all later versions will be affected by this.
e5a6a79#diff-17daa5db16c7d00be0fe1da12d1f9165L39

image

@wzhx78

This comment has been minimized.

Copy link
Author

wzhx78 commented Apr 3, 2018

@warmchang yes.

Is this reasonable to add --disable-kmem-limit flag in k8s ? anyone can discuss this with us ?

@like-inspur

This comment has been minimized.

Copy link

like-inspur commented Apr 13, 2018

I don't find there is a config named disable-kmem-limit for k8s. How to add this flag? @wzhx78

@wzhx78

This comment has been minimized.

Copy link
Author

wzhx78 commented Apr 14, 2018

k8s doesn't support now, we need to discuss with community whether is reasonable to add this flag in kubelet start option

@gyliu513

This comment has been minimized.

Copy link
Member

gyliu513 commented Apr 16, 2018

Not only 1.9, but also 1.10 and master have same issue. This is a very serious issue for production, I think providing a parameter to disable kmem limit is good.

/cc @dchen1107 @thockin any comments for this? Thanks.

@wzhx78

This comment has been minimized.

Copy link
Author

wzhx78 commented Apr 23, 2018

@thockin @dchen1107 any comments for this?

@gyliu513

This comment has been minimized.

Copy link
Member

gyliu513 commented May 21, 2018

@dashpole any reason to update memory.go as follows in e5a6a79#diff-17daa5db16c7d00be0fe1da12d1f9165L39 , this is seriously impacting Kubernetes 1.8, 1.9, 1.10, 1.11 etc.

-		if d.config.KernelMemory != 0 {
+			// Only enable kernel memory accouting when this cgroup
+			// is created by libcontainer, otherwise we might get
+			// error when people use `cgroupsPath` to join an existed
+			// cgroup whose kernel memory is not initialized.
 			if err := EnableKernelMemoryAccounting(path); err != nil {
 				return err
 			}

kolyshkin added a commit to kolyshkin/runc that referenced this issue Oct 31, 2018

libcontainer/cgroups: do not enable kmem on broken kernels
Commit fe898e7 (PR opencontainers#1350) enables kernel memory accounting
for all cgroups created by libcontainer even if kmem limit is
not configured.

Kernel memory accounting is known to be broken in RHEL7 kernels,
including the latest RHEL 7.5 kernel. It does not support reclaim
and can lead to kernel oopses while removing cgroup (merging it
with its parent). Unconditionally enabling kmem acct on RHEL7
leads to bugs:

* opencontainers#1725
* kubernetes/kubernetes#61937
* moby/moby#29638

I am not aware of any good way to figure out whether the kernel
memory accounting in the given kernel is working or broken.
For the lack of a better way, let's check if the running kernel
is RHEL7, and disable initial setting of kmem.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

kolyshkin added a commit to kolyshkin/runc that referenced this issue Nov 1, 2018

libcontainer: enable to compile without kmem
Commit fe898e7 (PR opencontainers#1350) enables kernel memory accounting
for all cgroups created by libcontainer -- even if kmem limit is
not configured.

Kernel memory accounting is known to be broken in some kernels,
specifically the ones from RHEL7 (including RHEL 7.5). Those
kernels do not support kernel memory reclaim, and are prone to
oopses. Unconditionally enabling kmem acct on such kernels lead
to bugs, such as

* opencontainers#1725
* kubernetes/kubernetes#61937
* moby/moby#29638

This commit gives a way to compile runc without kernel memory setting
support. To do so, use something like

	make BUILDTAGS="seccomp nokmem"

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

kolyshkin added a commit to kolyshkin/runc that referenced this issue Nov 1, 2018

libcontainer/cgroups: do not enable kmem on broken kernels
Commit fe898e7 (PR opencontainers#1350) enables kernel memory accounting
for all cgroups created by libcontainer even if kmem limit is
not configured.

Kernel memory accounting is known to be broken in RHEL7 kernels,
including the latest RHEL 7.5 kernel. It does not support reclaim
and can lead to kernel oopses while removing cgroup (merging it
with its parent). Unconditionally enabling kmem acct on RHEL7
leads to bugs:

* opencontainers#1725
* kubernetes/kubernetes#61937
* moby/moby#29638

I am not aware of any good way to figure out whether the kernel
memory accounting in the given kernel is working or broken.
For the lack of a better way, let's check if the running kernel
is RHEL7, and disable initial setting of kmem.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

kolyshkin added a commit to kolyshkin/runc that referenced this issue Nov 1, 2018

libcontainer: ability to compile without kmem
Commit fe898e7 (PR opencontainers#1350) enables kernel memory accounting
for all cgroups created by libcontainer -- even if kmem limit is
not configured.

Kernel memory accounting is known to be broken in some kernels,
specifically the ones from RHEL7 (including RHEL 7.5). Those
kernels do not support kernel memory reclaim, and are prone to
oopses. Unconditionally enabling kmem acct on such kernels lead
to bugs, such as

* opencontainers#1725
* kubernetes/kubernetes#61937
* moby/moby#29638

This commit gives a way to compile runc without kernel memory setting
support. To do so, use something like

	make BUILDTAGS="seccomp nokmem"

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

thaJeztah added a commit to thaJeztah/runc that referenced this issue Nov 13, 2018

libcontainer: ability to compile without kmem
Commit fe898e7 (PR opencontainers#1350) enables kernel memory accounting
for all cgroups created by libcontainer -- even if kmem limit is
not configured.

Kernel memory accounting is known to be broken in some kernels,
specifically the ones from RHEL7 (including RHEL 7.5). Those
kernels do not support kernel memory reclaim, and are prone to
oopses. Unconditionally enabling kmem acct on such kernels lead
to bugs, such as

* opencontainers#1725
* kubernetes/kubernetes#61937
* moby/moby#29638

This commit gives a way to compile runc without kernel memory setting
support. To do so, use something like

	make BUILDTAGS="seccomp nokmem"

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 6a2c155)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

clnperez added a commit to clnperez/runc that referenced this issue Nov 13, 2018

libcontainer: ability to compile without kmem
Commit fe898e7 (PR opencontainers#1350) enables kernel memory accounting
for all cgroups created by libcontainer -- even if kmem limit is
not configured.

Kernel memory accounting is known to be broken in some kernels,
specifically the ones from RHEL7 (including RHEL 7.5). Those
kernels do not support kernel memory reclaim, and are prone to
oopses. Unconditionally enabling kmem acct on such kernels lead
to bugs, such as

* opencontainers#1725
* kubernetes/kubernetes#61937
* moby/moby#29638

This commit gives a way to compile runc without kernel memory setting
support. To do so, use something like

	make BUILDTAGS="seccomp nokmem"

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
@ribx

This comment has been minimized.

Copy link

ribx commented Nov 19, 2018

I have no clue about cgroups and even less about docker and kubernetes internals. What I did to our k8s dev cluster was:

find /sys/fs/cgroup/ -iname "run-*" | while read line ; do grep -q . $line/tasks || rmdir $line ;done

This reduced the cgroups from ~65000 to ~3900.

Maybe someone can elaborate, if the absence of empty cgroups could harm docker/k8s runtime environment.

@apatil

This comment has been minimized.

Copy link
Contributor

apatil commented Nov 29, 2018

@kolyshkin is there documentation for how to use opencontainers/runc@6a2c155 in k8s?

@lcnsir

This comment has been minimized.

Copy link

lcnsir commented Dec 27, 2018

Is this RHEL update https://access.redhat.com/articles/3411331

Prior to this update, removing a memory cgroup sometimes resulted in a kernel warning or a kernel panic. This happened due to kmem caches being handled in the memory resource controller (memcg) without taking into account whether they are shared with the parent. This update introduces a fix that ensures delayed kmem cache removals and aliased kmem caches are handled properly. As a result, memory cgroups can now be removed without warnings or crashes. (BZ#1546734)

fixed the issue in RHEL OS part ?

@kolyshkin

This comment has been minimized.

Copy link
Contributor

kolyshkin commented Dec 27, 2018

@lcnsir not quite. What was fixed is a single kernel issue resulting in a kernel oops. A bunch of issues still remain, the main one being that kmem ctrl in this kernel is not trying to reclaim memory (try to shrink caches etc) once it's tight.

@kolyshkin

This comment has been minimized.

Copy link
Contributor

kolyshkin commented Dec 27, 2018

@apatil it should be clear from the commit message (a build tag should be added when building for rhel7 -- this means it should be a separate build unfortunately); let me know if you have any specific questions

@lcnsir

This comment has been minimized.

Copy link

lcnsir commented Dec 28, 2018

@kolyshkin thanks for your clarification. Changing the docker Cgroup Driver on rhel from cgroupfs tosystemd and related configurations in k8s (maybe also need an older version of docker) will lighten the problem or be a workaround ? Since as I know the OpenShift did this by default.

houming-wang added a commit to houming-wang/kubernetes that referenced this issue Jan 8, 2019

Disable Kernel Memory Accounting by default
Kernel memory accounting is known to be broken in some kernels like RHEL/CentOS,
this feature should be disabled to make pod/container cgroup works.
Detail info:
kubernetes#61937

houming-wang added a commit to houming-wang/kubernetes that referenced this issue Jan 8, 2019

Disable Kernel Memory Accounting by default
Kernel memory accounting is known to be broken in some kernels like RHEL/CentOS,
this feature should be disabled to make pod/container cgroup works.
Detail info: kubernetes#61937

houming-wang added a commit to houming-wang/kubernetes that referenced this issue Jan 9, 2019

Disable Kernel Memory Accounting by default
Kernel memory accounting is known to be broken in some kernels like RHEL/CentOS,
this feature should be disabled to make pod/container cgroup works.
Detail info: kubernetes#61937

chestack added a commit to es-container/kubernetes that referenced this issue Jan 9, 2019

Disable Kernel Memory Accounting by default
Kernel memory accounting is known to be broken in some kernels like RHEL/CentOS,
this feature should be disabled to make pod/container cgroup works.
Detail info: kubernetes#61937
@pires

This comment has been minimized.

Copy link
Member

pires commented Jan 16, 2019

It seems there is now a way to compile the kubelet without kmem accounting #72114

@dims

This comment has been minimized.

Copy link
Member

dims commented Jan 16, 2019

@pires looks like we don't have a way to specify tags in our build harness, we may need something like this #72998

@pires

This comment has been minimized.

Copy link
Member

pires commented Jan 18, 2019

@dims indeed, thanks for sharing.

@j-mccloskey

This comment has been minimized.

Copy link

j-mccloskey commented Jan 18, 2019

I think I could be seeing this issue in my cluster (v1.11.6) with AMIs from k8s-1.11-debian-stretch-amd64-hvm-ebs-2018-08-17. (Kernel seems to be 4.9.0-7-amd64).

Most of my pods are fine, but one in particular performs more writes to disk and it has a memory leak but behaves fine when running outside the cluster. Any advice on if its related to this issue or how to investigate further? Any help would be really appreciated.

Here is the output from cat /sys/fs/cgroup/memory/memory.stat on the node the pod is running, but I'm not really sure what I'm looking for tbh:

cache 82735104
rss 0
rss_huge 0
mapped_file 5402624
dirty 135168
writeback 0
pgpgin 1159186
pgpgout 1138987
pgfault 2212539
pgmajfault 97
inactive_anon 24576
active_anon 8192
inactive_file 1646592
active_file 81055744
unevictable 0
hierarchical_memory_limit 9223372036854771712
total_cache 4558540800
total_rss 2807050240
total_rss_huge 31457280
total_mapped_file 563404800
total_dirty 851968
total_writeback 0
total_pgpgin 124157768
total_pgpgout 122367193
total_pgfault 195377003
total_pgmajfault 1471
total_inactive_anon 3477504
total_active_anon 2808090624
total_inactive_file 3347120128
total_active_file 1206837248
total_unevictable 12288

Compared to the same command run on an instance (4.14.88-88.76.amzn2.x86_64) outside the cluster:

cache 777666560
rss 291532800
rss_huge 0
shmem 425984
mapped_file 160505856
dirty 0
writeback 0
swap 0
pgpgin 2447050
pgpgout 2186015
pgfault 2290225
pgmajfault 712
inactive_anon 282624
active_anon 291643392
inactive_file 664309760
active_file 112930816
unevictable 8192
hierarchical_memory_limit 9223372036854771712
hierarchical_memsw_limit 9223372036854771712
total_cache 777666560
total_rss 291532800
total_rss_huge 0
total_shmem 425984
total_mapped_file 160505856
total_dirty 0
total_writeback 0
total_swap 0
total_pgpgin 2447050
total_pgpgout 2186015
total_pgfault 2290225
total_pgmajfault 712
total_inactive_anon 282624
total_active_anon 291643392
total_inactive_file 664309760
total_active_file 112930816
total_unevictable 8192
@fenixshadow2005

This comment has been minimized.

Copy link

fenixshadow2005 commented Mar 5, 2019

May I ask if it's fixed or not in k8s 1.13?
How can I prevent this issue in my CentOS 7.5 with 3.10 kernel?

Thanks!

@qkboy

This comment has been minimized.

Copy link

qkboy commented Mar 5, 2019

May I ask if it's fixed or not in k8s 1.13?
How can I prevent this issue in my CentOS 7.5 with 3.10 kernel?

Thanks!

follow this recommend:Known Issue - KMEM - MSPH-2018-0006
We recommend that all Mesosphere DC/OS users on RHEL or CentOS 7.X upgrade to Docker 18.09.1, which was released on 01/09/2019. Docker 18.09.1 is a patch from Docker that does not enable kernel memory accounting within the Engine on the RHEL 7.X kernel.

Be attention: you must reboot your machine to disable kernel memory accounting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.