Disk Read is too high when memory is exhausted

[shsjshentao]

What happened:
High disk read
What you expected to happen:
low disk read

How to reproduce it (as minimally and precisely as possible):
use polinux/stress image
run stress --vm 1 --vm-bytes 125M --vm-hang 1 in many pods or docker containers over max host memory
then look iotop or iostat
Anything else we need to know?:
none
Environment:

• Kubernetes version (use kubectl version):
  k8s 1.13.1
• Cloud provider or hardware configuration:
  docker 1.13.1
• OS (e.g. from /etc/os-release):
  centos 7.2
• Kernel (e.g. uname -a):
  4.4
• Install tools:
  iotop
• Others:
  none

[shsjshentao]

I think it is the same with https://github.com/kubernetes/kubernetes/issues/61999
@dashpole

[neolit123]

/sig node

[anarchistHH1983]

/sig windows

…

On Mon, Feb 18, 2019, 1:51 PM Lubomir I. Ivanov ***@***.*** wrote: /sig node — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#74128 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Atif5QYdiq-YimC5XxUDbau2WrDyE3klks5vOvYugaJpZM4a9jgT> .

[dashpole]

It probably isn't 100% the same as #61999, since that was fixed all the way back in 1.10.

Is there a reason why you are running 4 kubelets?

Can you tell me more about the memory exhaustion? The image looks like it is just showing high I/O

[shsjshentao]

I only run one kubelet，and many things （not only kubelet） will happen high read io if memory exhausted.
And kubelet du is very slow
Feb 20 03:13:34 10-23-134-252 hyperkube[1549]: I0220 03:10:47.027142 1549 fsHandler.go:135] du and find on following dirs took 36m30.243286852s: [/var/lib/docker/overlay2...

[shsjshentao]

sorry, it's not k8s bug, when i ran containers without k8s, it still happened this issue. Maybe a kernel issue.