Bind mounts in container not updated after volume is remounted again

[cofyc]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

@kubernetes/sig-storage-bugs

What happened:

For fuse-backed volumes, fuse process may exit on error, the volume on the host will disappear. Kubelet will remount the volume later because it will find volume mount point does not exist anymore. But bind mount in container will still be the old one, processes in container will fail with error:

Transport endpoint is not connected

What you expected to happen:

• Processes in container can access new volume mount point without restart if possible
• Or containers are restarted with new bind mounts, then processes in container can access new volume mount point.
• Or other way to recover in this scenario.

How to reproduce it (as minimally and precisely as possible):

• Create a cephfs volume and enable fuse support
• Create a pod to mount the volume
• Kill ceph-fuse process on the host

Anything else we need to know?:

Should we restart container after remounting volumes of it?

Related issues

• GlusterFS volumes don't remount into pods #46253

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[cofyc]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[gnufied]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[cofyc]

/lifecycle frozen

[knightXun]

that's really hit my s3fs-fuse. If we use s3fs mount a bucket into a host as a dir, then mount this dir into container. At this point if we restart s3fs process, the dir in the container can't be access. Always show me 'Transport endpoint is not connected'. Does the ceph-fuse has the same problem?

[jingxu97]

@cofyc you mentioned "Kubelet will remount the volume later because it will find volume mount point does not exist anymore. ". But from my understanding, kubelet does not check whether mount still exist or not and remount it.

For this type problem, are the following steps necessary to recover? Is there any other way?

1. kubelet remount the volume,
2. container restarts with the new mount.

[knightXun]

I think the only way to fix this is to change some logic in cri and container-runtime.

[cofyc]

@jingxu97

I might make a mistake. I guess the userspace filesystem process (I was using ceph-fuse) was restarted by systemd.

yes, the kubelet can remount the volume and restarts the container to update the stale bind mounts in the container.

However, it's too complicated I guess. As a workaround solution, users can run the userspace filesystem process in a process supervisor (systemd), then the mount point will be remounted again after the process restarts. Second, the process in the pod can exit and wait for the kubelet to restart it again when it cannot access the filesystem or the mount point is found stale.

[garenchan]

Restarting container is a feasible approach. Is there a way to do this without restarting container?