Attacher/Detacher refactor for local storage #66884

NickrenREN · 2018-08-02T03:05:32Z

Proposal link: kubernetes/community#2438

What this PR does / why we need it:

Attacher/Detacher refactor for the plugins which just need to mount device, but do not need to attach, such as local storage plugin.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Attacher/Detacher refactor for local storage

/sig storage
/kind feature

NickrenREN · 2018-08-02T05:18:15Z

/retest

/assign @msau42 @jsafrane @saad-ali

msau42 · 2018-08-10T21:19:15Z

pkg/controller/volume/attachdetach/testing/testvolumespec.go

@@ -294,6 +294,10 @@ func (plugin *TestPlugin) NewAttacher() (volume.Attacher, error) {
 	return &attacher, nil
 }

+func (plugin *TestPlugin) NewDeviceMounter() (volume.DeviceMounter, error) {
+	return plugin.NewAttacher()


Hm, I wonder if this would cause any issues for attachable plugins if they had been keeping some state in the Attacher object.

The refactor move MountDevice() and GetDeviceMountPath() from Attacher to DeviceMounter.
The difference is:
Before this change: we will create an Attacher and let it call WaitForAttach, GetDeviceMountPath() and MountDevice()
After this change: we will create an Attacher and a DeviceMounter, let Attacher call WaitForAttach , let DeviceMounter call GetDeviceMountPath() and MountDevice().

Here, It seems that GetDeviceMountPath() and MountDevice() will not change the state kept in Attacher object(which is attachedVolumeMap), and also the ErrorEncountered will not affect the logic, since we will return directly if error occurs

BTW: for now, we will create a new Attacher in both GenerateMountVolumeFunc and GenerateAttachVolumeFunc which are called by different components. It seems that we should avoid keeping state in Attacher object.

please correct me if i am wrong.

I'm worried about situations where a plugin may have kept some state to serialize between WaitForAttach and MountDevice, like a lock or some cached data.

Got it, for now, i can not find the state kept between WaitForAttach and MountDevice through the plugins.
But should we cache the attacher info for the situations you are worried about ? e.g. if the attacher is created before, we do not need to create a new one in NewDeviceMounter...

/cc @jsafrane @saad-ali

I went through volume plugins, none of them really passes information from WaitForAttach to MountDevice via the attacher instance.

Thanks guys for searching! This sounds fine to me then.

jsafrane · 2018-08-13T16:08:55Z

I quickly went through the PR and I like it.
/approve
(this does not mean lgtm, I need more time or someone else may lgtm it)

gnufied · 2018-08-13T16:59:19Z

pkg/volume/volume.go


+// DeviceMounter can mount a block volume to a global path.


Can we drop block volume word from here? it seems iffy - there are block volumes which don't necessarily use DeviceMounter and there could be non-block volumes that use the interface?

there are block volumes which don't necessarily use DeviceMounter

which is true, for block mode.

there could be non-block volumes that use the interface

Actually, for now, local storage is the only use case that we only implement DeviceMounter( not implement Attacher interface).
We will only mount block device to a global path. If the local storage is a fs dir, we will do nothing, just set global path to its Path in pv spec.

gnufied · 2018-08-13T19:10:54Z

minor nit but otherwise lgtm!

msau42 · 2018-08-13T18:05:08Z

pkg/controller/volume/attachdetach/testing/testvolumespec.go

@@ -294,6 +294,10 @@ func (plugin *TestPlugin) NewAttacher() (volume.Attacher, error) {
 	return &attacher, nil
 }

+func (plugin *TestPlugin) NewDeviceMounter() (volume.DeviceMounter, error) {
+	return plugin.NewAttacher()


Thanks guys for searching! This sounds fine to me then.

msau42 · 2018-08-13T18:57:55Z

pkg/volume/util/operationexecutor/operation_generator.go

@@ -725,13 +734,31 @@ func (og *operationGenerator) GenerateUnmountDeviceFunc(
 		return volumetypes.GeneratedOperations{}, deviceToDetach.GenerateErrorDetailed("UnmountDevice.FindAttachablePluginBySpec failed", err)
 	}

-	volumeDetacher, err := attachableVolumePlugin.NewDetacher()
+	// Get DeviceMounter plugin


Do we still need to get attachable plugin above?

Also, how does the caller of this detect if the plugin is device mountable?

Do we still need to get attachable plugin above?

we don't need it now, thanks, i will remove it

Also, how does the caller of this detect if the plugin is device mountable?

The UnmountDevice func is only called when the volume mode is Filesystem and we find the attachedVolume is globally mounted , which is set true in MarkDeviceAsMounted. Which means the plugin is device mountable.

Ok, just want to make sure that this attachedVolume doesn't actually require AttachablePlugin?

I think so.
If the volume does not implement the attacher interface, it is assumed to be attached and the actual state of the world is updated accordingly.
see: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/operationexecutor/operation_executor.go#L133

msau42 · 2018-08-13T19:55:27Z

pkg/volume/util/operationexecutor/operation_generator.go

@@ -478,35 +478,45 @@ func (og *operationGenerator) GenerateMountVolumeFunc(
 		volumeAttacher, _ = attachableVolumePlugin.NewAttacher()
 	}

+	// get deviceMounter, if possible
+	deviceMountableVolumePlugin, _ := og.volumePluginMgr.FindDeviceMountablePluginBySpec(volumeToMount.VolumeSpec)


Can you add unit tests for these new plugin scenarios?

I am not sure i understand the scenarios. Just added some UTs for Concurrent Mount operations for devicemountable plugins in the third commit. PTAL, thanks

I mean adding test cases for DeviceMountable-only plugins for Mount and Unmount To ensure that we have correctly decoupled the attacher

We haven't implemented any DeviceMountable-only plugins for now, this PR is just interface refactor. We can add these UTs in my another PR: #63011
WDYT ?

NickrenREN · 2018-08-14T05:01:48Z

/retest

NickrenREN · 2018-08-14T06:20:55Z

Errors seem not related to this change

/retest

msau42 · 2018-08-14T22:49:55Z

/lgtm

saad-ali

/lgtm

k8s-ci-robot · 2018-08-14T23:15:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jsafrane, msau42, NickrenREN, saad-ali

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/controller/volume/attachdetach/OWNERS~~ [jsafrane,saad-ali]
~~pkg/kubelet/volumemanager/OWNERS~~ [saad-ali]
~~pkg/volume/OWNERS~~ [jsafrane,saad-ali]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

msau42 · 2018-08-14T23:18:35Z

/retest

k8s-github-robot · 2018-08-15T12:46:02Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2018-08-15T14:03:47Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-ci-robot requested review from anguslees and chrislovecnm August 2, 2018 03:06

k8s-ci-robot assigned jsafrane, msau42 and saad-ali Aug 2, 2018

msau42 reviewed Aug 10, 2018

View reviewed changes

k8s-ci-robot requested review from jsafrane and saad-ali August 11, 2018 04:01

gnufied reviewed Aug 13, 2018

View reviewed changes

msau42 reviewed Aug 13, 2018

View reviewed changes

NickrenREN force-pushed the attacher-detacher-refactor branch from 9e9374e to bc15887 Compare August 14, 2018 02:53

NickrenREN added 3 commits August 14, 2018 11:12

attacher/detacher refactor

c7e4466

update volume plugins accordingly

81e360d

add UTs for devicemountable conditions

55784f8

NickrenREN force-pushed the attacher-detacher-refactor branch from bc15887 to 55784f8 Compare August 14, 2018 03:15

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 14, 2018

saad-ali approved these changes Aug 14, 2018

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 14, 2018

k8s-github-robot merged commit c5e74d1 into kubernetes:master Aug 15, 2018

NickrenREN deleted the attacher-detacher-refactor branch August 15, 2018 15:06

NickrenREN mentioned this pull request Aug 16, 2018

Support both directory and block device for local volume plugin FileSystem VolumeMode #63011

Merged

msau42 mentioned this pull request Sep 4, 2018

Refactor volume.Attacher interface so that MountDevice is no longer coupled with Attach #64781

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Attacher/Detacher refactor for local storage #66884

Attacher/Detacher refactor for local storage #66884

NickrenREN commented Aug 2, 2018

NickrenREN commented Aug 2, 2018

msau42 Aug 10, 2018

NickrenREN Aug 11, 2018

msau42 Aug 11, 2018

NickrenREN Aug 11, 2018 •

edited

Loading

jsafrane Aug 13, 2018

msau42 Aug 13, 2018

jsafrane commented Aug 13, 2018

gnufied Aug 13, 2018

NickrenREN Aug 14, 2018 •

edited

Loading

gnufied commented Aug 13, 2018

msau42 Aug 13, 2018

msau42 Aug 13, 2018

msau42 Aug 13, 2018

NickrenREN Aug 14, 2018 •

edited

Loading

msau42 Aug 14, 2018

NickrenREN Aug 14, 2018 •

edited

Loading

msau42 Aug 13, 2018

NickrenREN Aug 14, 2018

msau42 Aug 14, 2018

NickrenREN Aug 14, 2018

NickrenREN commented Aug 14, 2018

NickrenREN commented Aug 14, 2018

msau42 commented Aug 14, 2018

saad-ali left a comment

k8s-ci-robot commented Aug 14, 2018

msau42 commented Aug 14, 2018

k8s-github-robot commented Aug 15, 2018

k8s-github-robot commented Aug 15, 2018

Attacher/Detacher refactor for local storage #66884

Attacher/Detacher refactor for local storage #66884

Conversation

NickrenREN commented Aug 2, 2018

NickrenREN commented Aug 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NickrenREN Aug 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsafrane commented Aug 13, 2018

Choose a reason for hiding this comment

NickrenREN Aug 14, 2018 • edited Loading

Choose a reason for hiding this comment

gnufied commented Aug 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NickrenREN Aug 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NickrenREN Aug 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NickrenREN commented Aug 14, 2018

NickrenREN commented Aug 14, 2018

msau42 commented Aug 14, 2018

saad-ali left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Aug 14, 2018

msau42 commented Aug 14, 2018

k8s-github-robot commented Aug 15, 2018

k8s-github-robot commented Aug 15, 2018

NickrenREN Aug 11, 2018 •

edited

Loading

NickrenREN Aug 14, 2018 •

edited

Loading

NickrenREN Aug 14, 2018 •

edited

Loading

NickrenREN Aug 14, 2018 •

edited

Loading