-
Notifications
You must be signed in to change notification settings - Fork 887
"sharing" the rkt data directory across a shared bindmount breaks rkt rm #3181
Comments
The "remount unshare" (MS_PRIVATE) is done in MountGC. It was done initially for I can reproduce the error "device or resource busy" without rkt with the following:
/cc @steveej |
@euank You're right, it is the same issue as the thing with systemd-udevd again:
Not only systemd-udevd, but any services that is started with
|
Just to add, this is the current work around we add to our rktnetes configuration in coreos-kuberetes: coreos/coreos-kubernetes@78acfe9 Any extra configuration steps required just make rktnetes more fragile to install by people not using that repo's install scripts. |
My previous comment was wrong: this is not related to systemd-udevd, bluetoothd or others. I tested again without services like systemd-udevd that "captures" the mount and the same issue still happens. This is caused by the way vfs_rmdir() and __is_local_mountpoint() are implemented: rmdir is allowed only when the dentry is not used as a mountpoint in the same mount namespace.
It might be possible to fix this in the kernel by checking the path in addition to the dentry, but it's not easy because __is_local_mountpoint() does not have access to the struct path, and it would involve lots of changes in vfs (rmdir, unlink, rename), p9, nfsd... |
@alban what's the reason we needed to remount with |
IIRC, the |
Pinging this issue since I was reminded of it over here coreos/coreos-kubernetes#768 (comment) and it still has real impact. I think we should definitely make non-fly stage1s not unshare mounts during rm (and that's an easy / not very scary fix). |
is there an update on this one? I still see it happening even with the nsenter workaround. My kubelet log is full off
and rkt list is unhappy:
|
currently broken due to kubernetes/kubernetes#24576 and this one rkt/rkt#3181
rkt has a gnarly bug (rkt/rkt#3181) that won't be fixed in a hurry (rkt/rkt#3486). It least to continuous task failures that eventually totally wreak worker nodes (kubernetes-retired#244). In the meantime we can use docker just as easily for this simple task. This work around was discussed in kubernetes-retired#199.
@s-urbaniak unfortunately the base issue here (shared bindmounts + unsharing before umount in gc) isn't fixed by v1.22.0. It's possible some of the related problems are fixed, but if it's from kubelet wrapper / rkt fly / shared bindmounts, it's less likely to be resolved. |
Still seeing this problem while trying to use the work-around listed above together with:
Any updates? |
rkt has a gnarly bug (rkt/rkt#3181) that won't be fixed in a hurry (rkt/rkt#3486). It least to continuous task failures that eventually totally wreak worker nodes (kubernetes-retired#244). In the meantime we can use docker just as easily for this simple task. This work around was discussed in kubernetes-retired#199.
This was uncovered while @pbx0 was working on
kubelet-wrapper
+ rktnetes.I'm reviving the discussion since I'd like to have a definitive answer as to whether creating shared bindmounts of the rkt data-directory should be supported or not in light of containerized-kubelet + rktlet.
I think that this is something we should try to support for the sort of "rkt in rkt, but with a shared data directory" cases (e.g. one pod which wants to get information about other pods).
This issue is related to #1922 and #1856
The issue can be easily reproduced by doing the following:
The reason this breaks as such is that
rkt rm
andrkt gc
explicitly do a remount unshare of each mount under a pod. I think this was added because of some issue with systmed-udevd holding mounts, but I forget the details, sorry!This working could be useful for containerized-kubelet / kubelet-wrapper and will impact some design choices.
@pbx0 and @yifan-gu have some context on this as well.
cc @lucab @s-urbaniak, this relates to rktlet since depending on whether this could or should be supported, we might have to break out of any container we're in for all rkt cli calls.
The text was updated successfully, but these errors were encountered: