Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove controller node plugin driver dependency for non-attachable fl… #47503

Merged
merged 1 commit into from
Jun 22, 2017

Conversation

chakri-nelluri
Copy link
Contributor

@chakri-nelluri chakri-nelluri commented Jun 14, 2017

…ex volume drivers (Ex: NFS).

What this PR does / why we need it:
Removes requirement to install flex volume drivers on master node for non-attachable drivers likes NFS.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #47109

Fixes issue w/Flex volume, introduced in 1.6.0, where drivers without an attacher would fail (node indefinitely waiting for attach). Drivers that don't implement attach should return `attach: false` on `init`.

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 14, 2017
@k8s-ci-robot
Copy link
Contributor

Hi @chakri-nelluri. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with @k8s-bot ok to test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-github-robot k8s-github-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. release-note-label-needed labels Jun 14, 2017
@msau42
Copy link
Member

msau42 commented Jun 14, 2017

/assign @saad-ali

cc @verult

@msau42
Copy link
Member

msau42 commented Jun 14, 2017

@k8s-bot ok to test

@k8s-ci-robot k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 14, 2017
ok, err := attachablePlugin.isAttachable()
if err != nil {
glog.Errorf("Unable to probe plugin: %s", attachablePlugin.driverName)
continue
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is going to require all existing flex drivers to implement the new probe call. Can we have a default instead?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the capabilities call isn't implemented, isAttachable() will return true, and the old behavior stays.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. This change doesn't require you to modify the existing plugins.

@msau42
Copy link
Member

msau42 commented Jun 14, 2017

cc @kubernetes/sig-storage-pr-reviews

@k8s-ci-robot k8s-ci-robot added the sig/storage Categorizes an issue or PR as relevant to SIG Storage. label Jun 14, 2017
flexVolumePlugin
}

var _ volume.AttachableVolumePlugin = &flexVolumeAttachablePlugin{}
var _ volume.PersistentVolumePlugin = &flexVolumePlugin{}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The naming here might cause confusions (attachable vs persistent?) ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are not flex objects. These are infra objects and we are just declaring it.

}

// By default all plugins are attachable, unless they report otherwise.
cap, ok := res.CapabilityMap[attachCapability]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like cap is a keyword. Should be OK here, though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to rename it to capabilities, similar to annotations & labels. Will update the PR.


// By default all plugins are attachable, unless they report otherwise.
cap, ok := res.CapabilityMap[attachCapability]
if ok && !cap {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"&& !cap" isn't necessary here. The two return statements can also be collapsed into one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep.. remove !cap :) We still need two return statements.

} else {
// Plugin does not support attach/detach, so return flexVolumePlugin which supports
// SetupAt & TearDown functionality
plugin := &flexVolumePlugin{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could reuse the same instance in line 45.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack

// Check whether the plugin is attachable.
ok, err := attachablePlugin.isAttachable()
if err != nil {
glog.Errorf("Unable to probe plugin: %s", attachablePlugin.driverName)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend including the isAttachable() error in the message to give a reason why probe is not successful.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep. Copy & paste typo. will fix it.

@@ -198,6 +200,10 @@ type DriverStatus struct {
VolumeName string `json:"volumeName,omitempty"`
// Represents volume is attached on the node
Attached bool `json:"attached,omitempty"`
// Returns capabilites of the driver.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack

ok, err := attachablePlugin.isAttachable()
if err != nil {
glog.Errorf("Unable to probe plugin: %s", attachablePlugin.driverName)
continue
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the capabilities call isn't implemented, isAttachable() will return true, and the old behavior stays.


return (*attacherDefaults)(a).GetDeviceMountPath(spec, mountsDir)
volumeName, err := a.plugin.GetVolumeName(spec)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GetVolumeName() is called before every makeGlobalMountPath(). Maybe move the call inside makeGlobalMountPath().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack

unsupportedCommands: []string{},
})

attachablePlugin := &flexVolumeAttachablePlugin{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Factoring is a little strange here. Two suggestions:

  1. How about introducing a constructor method NewFlexVolumePlugin(...) that takes all the flexVolumePlugin fields as a parameter. It would internally test for attachability and return either a flexVolumeAttachablePlugin or a flexVolumePlugin--that'll simplify the code here.
  2. For testing attachability, instead of constructing a flexVolumePlugin object and calling isAttachable() on it, it would be cleaner to have IsAttachable(...) be a static method that takes all the flexVolumePlugin fields as a parameter, make the appropriate calls to the driver to determine attachability, and then return true/false.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK.

Copy link
Contributor Author

@chakri-nelluri chakri-nelluri Jun 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback. Refactored it. PTAL.

return false, err
}

// By default all plugins are attachable, unless they report otherwise.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about making the default false, aka non-attachable? And if a plugin implements attach, then they must explicitly also set capability to true?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is going to break existing plugins. That is the only reason. Doesn't benefit too much either-way.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify it would break existing post-1.6 drivers right? Pre-1.6 drivers would continue to work as is if the default was false, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. For Post 1.6 drivers. Users have to modify their drivers to work for 1.6+ anyways, because of change mount callout.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack. Ok fine to leave as is.

@jsafrane
Copy link
Member

I don't understand how this PR "Removes requirement to install flex volume drivers on master node" - if there is no flex volume driver on the master, then controller-manager does not create a volume plugin for it and the situation is exactly the same as before this PR. Calling init and parsing its output won't change a thing, because there is nothing to call on the master.

@chakri-nelluri
Copy link
Contributor Author

@jsafrane. This will fix it on Kubelet side. Now Kubelet won't wait for controller-manager to attach the volume for it.

@chakri-nelluri
Copy link
Contributor Author

Thanks @saad-ali @verult @msau42. Rebased it to one commit.

Copy link
Member

@saad-ali saad-ali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

runner: exec.New(),
unsupportedCommands: []string{},
})
plugin, err := NewFlexVolumePlugin(pluginDir, f.Name())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beautiful! Thanks for refactoring, so much cleaner!

@k8s-github-robot k8s-github-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-label-needed labels Jun 15, 2017
@saad-ali
Copy link
Member

/lgtm
/approve

I added a release note to original comment.

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 15, 2017
@saad-ali saad-ali added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 15, 2017
@k8s-github-robot k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 15, 2017
Copy link
Contributor Author

@chakri-nelluri chakri-nelluri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack

@@ -71,21 +123,13 @@ func (plugin *flexVolumePlugin) GetVolumeName(spec *volume.Spec) (string, error)
call := plugin.NewDriverCall(getVolumeNameCmd)
call.AppendSpec(spec, plugin.host, nil)

_, err := call.Run()
status, err := call.Run()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good Catch. Oops.. Mangled in rebase. Let me fix it.

@saad-ali
Copy link
Member

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 19, 2017
@saad-ali
Copy link
Member

@k8s-bot pull-kubernetes-e2e-gce-etcd3 test this

@verult
Copy link
Contributor

verult commented Jun 19, 2017

/approve

@k8s-github-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chakri-nelluri, saad-ali, verult
We suggest the following additional approver: zmerlynn

Assign the PR to them by writing /assign @zmerlynn in a comment when ready.

Associated issue: 47109

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@marun
Copy link
Contributor

marun commented Jun 20, 2017

/test pull-kubernetes-e2e-gce-etcd3

2 similar comments
@saad-ali
Copy link
Member

/test pull-kubernetes-e2e-gce-etcd3

@saad-ali
Copy link
Member

/test pull-kubernetes-e2e-gce-etcd3

@saad-ali
Copy link
Member

This test suite is super flaky! @marun you may want to consider manual merge.

@marun
Copy link
Contributor

marun commented Jun 21, 2017

/retest

@dchen1107 dchen1107 assigned msau42 and saad-ali and unassigned saad-ali, bprashanth and jingxu97 Jun 22, 2017
@dchen1107
Copy link
Member

/retest

@k8s-github-robot
Copy link

Automatic merge from submit-queue (batch tested with PRs 47878, 47503, 47857)

@feiskyer
Copy link
Member

feiskyer commented Aug 14, 2017

Will this fix be cherry-picked to 1.7 branch? The milestone is v1.7 but there is no cherry-pick labels.

@verult
Copy link
Contributor

verult commented Aug 16, 2017

k8s-github-robot pushed a commit to kubernetes/community that referenced this pull request Aug 19, 2017
Automatic merge from submit-queue

Updating FlexVolume doc with capabilities addition.

The following PR introduces a FlexVolume API change, so the documentation needs to be updated.
kubernetes/kubernetes#47503
@resouer
Copy link
Contributor

resouer commented Aug 22, 2017

FYI: you need to use 1.7.4+ to include this fix.

@kokhang
Copy link
Contributor

kokhang commented Sep 13, 2017

@verult, @resouer : just found out that non-attachable flexvolumes was broken on 1.6. is there any work around to make it work without having to set the enable-controller-attach-detach=false?

Also, if we set this flag, would that screw things up with other volume plugins?

@verult
Copy link
Contributor

verult commented Sep 13, 2017

@kokhang as long as the driver exists on master, everything should work as expected.

If the flag is set, other plugins may run into attach race conditions between nodes. This was the main problem AttachDetachController solves.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/storage Categorizes an issue or PR as relevant to SIG Storage. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

nfs flexvolume example not working