This repository has been archived by the owner on Oct 16, 2020. It is now read-only.
kubelet-wrapper leaves behind Orphaned Pods #1831
Labels
Comments
You should not see the old exited one, as the example service unit does a rm in |
|
Because of my old setup, there was still a kubelet.service without the rm, but I tested it this morning with the recommended kubelet.service and the problem still exists. kubernetes/kubernetes#38498 (comment) One more thing, I just realized. One mount counts up when restarting kubelet. Unmounting all of them by hand, and waiting for kubelet to recreate it: No Orphaned Pod! |
|
@lucab I did another test with hyperkube "no node" without running it in rkt. Without rkt, the problem does not exist. While still exists when using rkt. |
|
/cc @pbx0 |
|
As per kubernetes/kubernetes#38498 (comment), the The only bit I'd like to double-check in order to turn this on by default is ensuring that the recursive option doesn't propagate back an additional mount to the host-ns. /cc @euank @aaronlevy |
lucab
added a commit
to lucab/coreos-overlay
that referenced
this issue
Apr 5, 2017
… mount So far `/var/lib/kubelet` was mounted as an implicit non-recursive mount. This changes the wrapper to an explicit recursive mount. As shown in kubernetes/kubernetes#38498 (comment), current non-recursive behavior seems to confuse the kubelet which is incapable of cleaning up resources for orphaned pods, as the extisting mountpoints for them are not available inside kubelet chroot. With `recursive=true`, those mounts are made available in the chroot and can be unmounted on the host-side from kubelet chroot via shared back-propagation. Fixes coreos/bugs#1831
euank
pushed a commit
to euank/coreos-overlay
that referenced
this issue
May 30, 2017
… mount So far `/var/lib/kubelet` was mounted as an implicit non-recursive mount. This changes the wrapper to an explicit recursive mount. As shown in kubernetes/kubernetes#38498 (comment), current non-recursive behavior seems to confuse the kubelet which is incapable of cleaning up resources for orphaned pods, as the extisting mountpoints for them are not available inside kubelet chroot. With `recursive=true`, those mounts are made available in the chroot and can be unmounted on the host-side from kubelet chroot via shared back-propagation. Fixes coreos/bugs#1831
euank
pushed a commit
to euank/coreos-overlay
that referenced
this issue
May 30, 2017
… mount So far `/var/lib/kubelet` was mounted as an implicit non-recursive mount. This changes the wrapper to an explicit recursive mount. As shown in kubernetes/kubernetes#38498 (comment), current non-recursive behavior seems to confuse the kubelet which is incapable of cleaning up resources for orphaned pods, as the extisting mountpoints for them are not available inside kubelet chroot. With `recursive=true`, those mounts are made available in the chroot and can be unmounted on the host-side from kubelet chroot via shared back-propagation. Fixes coreos/bugs#1831
ChrisMcKenzie
pushed a commit
to ChrisMcKenzie/coreos-overlay
that referenced
this issue
Dec 9, 2017
… mount So far `/var/lib/kubelet` was mounted as an implicit non-recursive mount. This changes the wrapper to an explicit recursive mount. As shown in kubernetes/kubernetes#38498 (comment), current non-recursive behavior seems to confuse the kubelet which is incapable of cleaning up resources for orphaned pods, as the extisting mountpoints for them are not available inside kubelet chroot. With `recursive=true`, those mounts are made available in the chroot and can be unmounted on the host-side from kubelet chroot via shared back-propagation. Fixes coreos/bugs#1831
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Issue Report
Bug
Container Linux Version
Environment
Cloud Provider and VirtualBox VMs. But issue should be the same in all environments
Expected Behavior
If Pods are going to be deleted, they should go away and don't leave behind Orphaned Pods.
Actual Behavior
Orphaned Pods stay on the system
Reproduction Steps
You should see a new kubelet container in rkt list, and the old one stopped
I did not test this with Persistent Storage, but this may be the same issue. This could be a real problem, because the Volume may not be attachable to another node.
While with Secrets and ConfigMaps this is not as relevant, because the Orphaned Pods are cleaned up on the next reboot anyway.
Other Information
Copy & paste from: kubernetes/kubernetes#38498 (comment)
TL;DR: It seeoms to be a problem when running kubelet in rkt fly on CoreOS
Currently happens on the system:
But it can't be moved, since it is still mounted. So like you mentioned, kubelet does not consider the volume to be a tmpfs.
Hmm. There was a crashed kubelet (out of space on this node)
Some mounts are gone.
Turn on more logging
Okey, let's drill this one down, according to https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/empty_dir/empty_dir_linux.go#L37, just to be sure, this works like expected.
This works:
Trying this on the affected node, with the real file
Surprise, this is a tmpfs like expected. So what else could this be? I noticed that Type:61267 shows us, that we are a ext4 mount point. So likely kubelets hits /.
For sure, kubelet is running as a rkt fly container.
Well, this would have been discoverable without the lines of code. But anyway.
It is quite simple to reproduce this behavior. Just start a Pod with a Secret. Stop kubelet on the node, I left it off until the api server recognised. Started it again (with help kubelet-wrapper, of course) and waited until the api server showed this node as Ready. I made sure, the pod was still running on this node. After this, I just deleted it with kubectl. Voila, one more Orphand Pod with the same symptoms.
The text was updated successfully, but these errors were encountered: