Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
kubelet-wrapper leaves behind Orphaned Pods #1831
Container Linux Version
Cloud Provider and VirtualBox VMs. But issue should be the same in all environments
If Pods are going to be deleted, they should go away and don't leave behind Orphaned Pods.
Orphaned Pods stay on the system
You should see a new kubelet container in rkt list, and the old one stopped
I did not test this with Persistent Storage, but this may be the same issue. This could be a real problem, because the Volume may not be attachable to another node.
While with Secrets and ConfigMaps this is not as relevant, because the Orphaned Pods are cleaned up on the next reboot anyway.
Copy & paste from: kubernetes/kubernetes#38498 (comment)
TL;DR: It seeoms to be a problem when running kubelet in rkt fly on CoreOS
Currently happens on the system:
But it can't be moved, since it is still mounted. So like you mentioned, kubelet does not consider the volume to be a tmpfs.
Hmm. There was a crashed kubelet (out of space on this node)
Some mounts are gone.
Turn on more logging
Okey, let's drill this one down, according to https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/empty_dir/empty_dir_linux.go#L37, just to be sure, this works like expected.
Trying this on the affected node, with the real file
Surprise, this is a tmpfs like expected. So what else could this be? I noticed that Type:61267 shows us, that we are a ext4 mount point. So likely kubelets hits /.
For sure, kubelet is running as a rkt fly container.
Well, this would have been discoverable without the lines of code. But anyway.
It is quite simple to reproduce this behavior. Just start a Pod with a Secret. Stop kubelet on the node, I left it off until the api server recognised. Started it again (with help kubelet-wrapper, of course) and waited until the api server showed this node as Ready. I made sure, the pod was still running on this node. After this, I just deleted it with kubectl. Voila, one more Orphand Pod with the same symptoms.
referenced this issue
Feb 25, 2017
Because of my old setup, there was still a kubelet.service without the rm, but I tested it this morning with the recommended kubelet.service and the problem still exists. kubernetes/kubernetes#38498 (comment)
One more thing, I just realized. One mount counts up when restarting kubelet.
Unmounting all of them by hand, and waiting for kubelet to recreate it:
No Orphaned Pod!
@lucab I did another test with hyperkube "no node" without running it in rkt. Without rkt, the problem does not exist.
While still exists when using rkt.
As per kubernetes/kubernetes#38498 (comment), the
The only bit I'd like to double-check in order to turn this on by default is ensuring that the recursive option doesn't propagate back an additional mount to the host-ns.