volumemanager could not process reconstructed volume‘s consistency while pvc volume delay adding to dsw while apiserver access unstabitily

[249043822]

What happened:

We have even encountered such case that pod continuously reports “Unable to attach or mount volumes for pod;...” when kubelet restarted. The reasons are listed in details as below:

1. In dswp.populatorLoop, as result of apiserver instantaneous unstablility, dswp get pvc‘s pv spec failed, then the pvc volume does not be added to dsw.

   kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go

   Line 542 in b0010c2

   dswp.getPVSpec(pvName, pvcSource.ReadOnly, pvcUID)

2. In rc.reconciliationLoopFunc() -> rc.sync(), syncStates() lists all the volume directories locally, and recontructs the volume from local volume, if the volume isn't in dsw, it will put the recontructed volume to asw, but now the volume.OuterVolumeSpecName isn't correct.
   

   kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go

   Line 583 in b0010c2

   outerVolumeSpecName: volume.volumeSpecName,

3. By chance, at this time, In dswp.populatorLoop, the fail to added pvc volume is added to dsw again normally
4. Now in reconcile() -> mountAttachVolumes(), it compares dsw and asw by volume's innerVolumename, it's all the same. but it could not sense the OuterVolumeSpecName dismatch. so it will not re-mount again to update asw cache
   

   kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go

   Line 205 in b0010c2

   volMounted, devicePath, err := rc.actualStateOfWorld.PodExistsInVolume(volumeToMount.PodName, volumeToMount.VolumeName)

5. In syncPod, it checks pod's volume attach or mount success by comparing volume.OuterVolumeSpecName and pod.spec.volumeName, but this pv'c OuterVolumeSpecName isn't correct in asw. so pod continuously reports “Unable to attach or mount volumes for pod;...”
   

   kubernetes/pkg/kubelet/volumemanager/volume_manager.go

   Line 459 in b0010c2

   mountedVolumes.Insert(mountedVolume.OuterVolumeSpecName)

What you expected to happen:

reconcile() could sense the reconstruct volume's dismatch to the dsw, then re-mount again to make volume cache consistency

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.19.4
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

[249043822]

/sig node

[249043822]

/triage accepted

[249043822]

/assign @gnufied @jingxu97

[249043822]

CC @sanwishe @yangjunmyfm192085

[yangjunmyfm192085]

I think this is a point that can be optimized. /cc@ehashman @dchen1107

[249043822]

I will give a fix for this

[jingxu97]

@249043822 Thanks for reporting the issue, this innerSpecName and outerSpecName are confusing and cause issues. Please ping me for review when you have a fix. Thanks!

[ehashman]

Please don't triage your own issues immediately, give someone else a chance to review them :(

/sig storage

[249043822]

Please don't triage your own issues immediately, give someone else a chance to review them :(

/sig storage

@ehashman okay, I know