Many WC processes

[daniel-yavorovich]

Dear Sira/Madams,

I use Kubernetes v1.4.6 and see many wc processes. Some processes are running for a month.

# ps aux | grep -c '[wc]'
3790 
# ps auxffww | grep '[wc]' | head -125 | tail -10
root     29274  0.9  0.4 5705996 147268 ?      Ssl  Nov24 505:52 /usr/bin/dockerd -H fd:// --mtu=1472 --bip=10.1.42.1/24
root     29281  0.0  0.0 3388080 20264 ?       Ssl  Nov24   4:53  \_ docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc
root     29533  0.0  0.0 282564   936 ?        Sl   Nov24   5:16  |   \_ docker-containerd-shim 43862be3c4f16a0df9258806dff9bb3b4870d72e32b644442d2a57d5c9466a61 /var/run/docker/libcontainerd/43862be3c4f16a0df9258806dff9bb3b4870d72e32b644442d2a57d5c9466a61 docker-runc
root     29549  4.7  0.3 815028 119500 ?       Ssl  Nov24 2536:05  |   |   \_ /hyperkube kubelet --allow-privileged --api-servers=http://localhost:8080 --config=/etc/kubernetes/manifests-multi --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --hostname-override=10.42.233.138 --v=2
root     29585  0.0  0.0 379220 15696 ?        S    Nov24   0:39  |   |       \_ journalctl -k -f
root     32761  0.0  0.0      0     0 ?        Z    Nov26   0:00  |   |       \_ [wc] <defunct>
root      9058  0.0  0.0      0     0 ?        Z    Nov27   0:00  |   |       \_ [wc] <defunct>
root     25592  0.0  0.0      0     0 ?        Z    Nov30   0:00  |   |       \_ [wc] <defunct>
root     12461  0.0  0.0      0     0 ?        Z    Dec01   0:00  |   |       \_ [wc] <defunct>
root     27458  0.0  0.0      0     0 ?        Z    Dec12   0:00  |   |       \_ [wc] <defunct>

You can answer what are generated by these processes, and is not this a bug?

Thank you.

[Typositoire]

I just restarted Kubelet (after a week) and I had 9000 wc zombies... I run on 1.5.1

[fraenkel]

This might be the same as #38894

[BugRoger]

It's not necessarily the same cause as in #38894. We're experiencing the same problem but have a full featured GNU find. It is related though.

[0xmichalis]

@kubernetes/sig-node-misc

[euank]

The fixes for #38894 should be in master, just not in any release branch.

@calebamiles wanna see about getting the two commits referenced in #38894 (comment) into the 1.5 release branch?

[dashpole]

The change in cAdvisor: #1558 still needs to be vendored into kubernetes, and cherrypicked to 1.4

[calebamiles]

I actually don't know how cAdvisor is vendored into Kubernetes. @dashpole if you ping me on the PR that vendors into Kubernetes I can try and get it cherry picked.

cc: @euank, @saad-ali

[MartinPyka]

+1 for adding the fix for #38894 into the next release

[Bregor]

@daniel-yavorovich, @Typositoire do you use weave-net?

[daniel-yavorovich]

@Bregor no, I use flannel only.

Thank you.

[Bregor]

@daniel-yavorovich ok, thank you.

[stuszynski]

We're running onv1.5.3 and we also have exact wc processes leak issue. After the few days since the last restart of kubelet, we have hundreds of defunct wc processes with hyperkube kubelet as a parent.

$ ps ax | grep wc 
31902  0.0  0.0      0     0 ?        Z    10:51   0:00 [wc] <defunct>
32042  0.0  0.0      0     0 ?        Z    11:47   0:00 [wc] <defunct>
...

$ ps ax | grep wc | wc -l
1038

I also found some mentions in kubelet log:

Mar 14 21:51:19 kubernetes2 hyperkube[16528]: I0314 21:51:19.284652   16528 fs.go:495] killing cmd [find /var/lib/docker/overlay/4ea252565a33e2f5763648f95eeaa44d9d614fdb212b3d442d1915b1cba06393 -xdev -printf .], and cmd [wc -c] due to timeout(2m0s)

It looks like it has already been solved by this commit: c03ec46 (#39477 ). But it wasn't backported to v1.5 nor v1.4 yet.

[dashpole]

Working on getting it to 1.5, and 1.4 #43113

[golfreeze]

I've faced this bug also on v.1.5.3 ,
To fixed it , Shall I need to upgrade kubernetes , Right ?
If "YES" , Could you show the step ?
Thank you.

[dashpole]

The fix is in v1.5.6

[golfreeze]

Thank you dashpole!!

[calebamiles]

@dashpole are you still planning on fixing this for 1.4? Otherwise could probably close this issue.

[dashpole]

I am still planning on it. I ran into a roadblock cherrypicking it to 0.24.2 in the cadvisor branch, but I dont remember what it was... Ill try and do that soon

[calebamiles]

Thanks for the update, @dashpole.

[calebamiles]

any news, @dashpole?

[dashpole]

Oh, sorry. I dont think ill be able to get to this soon. For those affected, 1.5 and 1.6 both have the fix for this. We can probably just close this.

[0xmichalis]

/sig node

[euank]

/close

[k8s-ci-robot]

@euank: you can't close an issue unless you authored it or you are assigned to it.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[euank]

Monthly reminder; I think this can be closed as fixed in 1.5+ if someone with the power to close it wishes to do so.

[varsharaja]

I am seeing the same issue in 1.5.5. What exactly should I do to fix this? Our kubernetes was installed with the old setup-aws scripts and not with kops.

[dashpole]

@varsharaja as I mentioned here, upgrading to 1.5.6 will fix this.

[varsharaja]

Thank you. Will do.

…

On 10-Jan-2018 23:20, "David Ashpole" ***@***.***> wrote: @varsharaja <https://github.com/varsharaja> as I mentioned here <#39334 (comment)>, upgrading to 1.5.6 will fix this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39334 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ARHDwyh_I84ExsvbhrcS-8O8wmaR3n2Yks5tJPhugaJpZM4LYk1y> .