hostPath volumes used with subPath volume mounts don't support reconstruction

[msau42]

Is this a BUG REPORT or FEATURE REQUEST?:
@kubernetes/sig-storage-bugs

What happened:
Normally this isn't a problem, but if you use subpath with hostpath volumes, then that means the subpath mounts will not get cleaned up during the reconstruction window (a pod is force deleted while kubelet is down)

[msau42]

Noticed this issue while working on #61373

[liggitt]

How is this different than #61372?

[msau42]

#61372 affects reconstruction for PVCs

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[msau42]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[msau42]

/remove-lifecycle stale

We intend to address this by redesigning the volume reconstruction feature to use kubelet checkpointing.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[msau42]

/reopen
/lifecycle frozen

[k8s-ci-robot]

@msau42: Reopened this issue.

In response to this:

/reopen
/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neujie]

Better practice:

• kill or delete pod
• clear the pod's all subpath
  otherwise a lot of useless dir will be left

[dobsonj]

/assign

[dobsonj]

The in-tree hostPath driver makes no attempt to unmount:

kubernetes/pkg/volume/hostpath/host_path.go

Lines 266 to 274 in 6c96ac0

	// TearDown does nothing.
	func (c *hostPathUnmounter) TearDown() error {
	return nil
	}
	// TearDownAt does not make sense for host paths - probably programmer error.
	func (c *hostPathUnmounter) TearDownAt(dir string) error {
	return fmt.Errorf("TearDownAt() does not make sense for host paths")
	}

Once that is fixed, these skips need to be removed:

kubernetes/test/e2e/storage/testsuites/subpath.go

Lines 347 to 350 in 6c96ac0

	if strings.HasPrefix(driverName, "hostPath") {
	// TODO: This skip should be removed once #61446 is fixed
	e2eskipper.Skipf("Driver %s does not support reconstruction, skipping", driverName)
	}

kubernetes/test/e2e/storage/testsuites/subpath.go

Lines 359 to 362 in 6c96ac0

	if strings.HasPrefix(driverName, "hostPath") {
	// TODO: This skip should be removed once #61446 is fixed
	e2eskipper.Skipf("Driver %s does not support reconstruction, skipping", driverName)
	}

[dobsonj]

The in-tree hostPath driver makes no attempt to unmount

So it's not really the TearDown code I quoted above, the subpaths would normally be cleaned up here:

kubernetes/pkg/volume/util/operationexecutor/operation_generator.go

Lines 825 to 830 in 2e55595

	// Remove all bind-mounts for subPaths
	podDir := filepath.Join(podsDir, string(volumeToUnmount.PodUID))
	if err := subpather.CleanSubPaths(podDir, volumeToUnmount.InnerVolumeSpecName); err != nil {
	eventErr, detailedErr := volumeToUnmount.GenerateError("error cleaning subPath mounts", err)
	return volumetypes.NewOperationContext(eventErr, detailedErr, migrated)
	}

But we never make any attempt to call UnmountVolume in this scenario.

kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go

Lines 185 to 187 in 2e55595

	klog.V(5).InfoS(mountedVolume.GenerateMsgDetailed("Starting operationExecutor.UnmountVolume", ""))
	err := rc.operationExecutor.UnmountVolume(
	mountedVolume.MountedVolume, rc.actualStateOfWorld, rc.kubeletPodsDir)

unmountVolumes loops over rc.actualStateOfWorld.GetAllMountedVolumes(), but if we're just using a hostPath (type = Directory), there won't be a mount point, and it won't be listed as a mounted volume. But the subpaths still use bind mounts, and those never get unmounted.

For future reference it can be reproduced manually with:

kind: Pod
apiVersion: v1
metadata:
  name: my-intree-inline-app
spec:
  containers:
    - name: my-frontend
      image: busybox
      volumeMounts:
      - mountPath: "/data"
        name: my-inline-vol
      - mountPath: "/data/subpath1"
        name: my-inline-vol
        subPath: subpath1
      - mountPath: "/data/subpath2"
        name: my-inline-vol
        subPath: subpath2
      command: [ "sleep", "1000000" ]
  volumes:
    - name: my-inline-vol
      hostPath:
        path: /tmp/dir1
        type: Directory

1. create pod with spec above, check mountpoints
2. kill kubelet
3. force delete pod
4. start kubelet again
5. check mount points and kubelet.log