linstor-csi is not compatible with `--enable-controller-attach-detach=false` #4

kvaps · 2019-03-02T03:23:14Z

Hi, my pods are stuck on Init because attaching volumes is not working, here is kubelet log:

kubelet[5246]: I0302 04:17:45.983773    5246 reconciler.go:227] operationExecutor.AttachVolume started for volume "pvc-c4fd2215-3c18-11e9-bffa-001999d764e2" (UniqueName: "kubernetes.io/csi/io.drbd.linstor-csi^bf753dd6-48df-4157-a421-74d802a59e44") pod "hc1-wd43-0" (UID: "13418de2-3c97-11e9-bffa-001999d764e2")
kubelet[5246]: E0302 04:17:45.986787    5246 csi_attacher.go:103] kubernetes.io/csi: attacher.Attach failed: volumeattachments.storage.k8s.io is forbidden: User "system:node:m12c16" cannot create resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope: can only get individual resources of this type

It seems something wrong with RBAC:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:node
rules:
...
- apiGroups:
  - storage.k8s.io
  resources:
  - volumeattachments
  verbs:
  - get

kubectl version

Client Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.0-alpha.0.1172+29b3ec91089d0e-dirty", GitCommit:"29b3ec91089d0e213c15118e3c39087bcfbb6400", GitTreeState:"dirty", BuildDate:"2018-12-20T23:05:11Z", GoVersion:"go1.11.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

The text was updated successfully, but these errors were encountered:

kvaps · 2019-03-02T10:08:43Z

It seems that problems somewhere in node_authorizer.go
It have no any option for allow create of volumeattachments

kvaps · 2019-03-02T10:42:24Z

I found exactly the problem:
If my kubelets have --enable-controller-attach-detach=false option, then I have this issue.
But if my kubelets have --enable-controller-attach-detach=true option, then everything is working fine.

That's crazy, because --enable-controller-attach-detach=false option is required by old flexvolume driver. And I have a lots of flexvolumes created using old linstor driver, now I can't start using csi just because it requires --enable-controller-attach-detach=true which makes old volumes non-working 🙁

deleteriousEffect · 2019-03-04T15:22:03Z

It seems that problems somewhere in node_authorizer.go
It have no any option for allow create of volumeattachments

You're saying to can't change modify that RBAC at all? What have you tried? What where the error messages you got from those attempts?

kvaps · 2019-03-04T16:24:14Z

@haySwim hi,

You're saying to can't change modify that RBAC at all?

Еhis problem shouldn't and can't be solved by RBAC, because:

Kubelets are using Node authorization mode, not RBAC, actually this behavior is controlled by node_authorizer.go
You can't create common rolebinding for any system:node:* due inability using wildcards here.

What have you tried?

I have temporary solution with creating rolebinding rule for every my node.

What where the error messages you got from those attempts?

From kubelets logs directly

Look, here is two possible ways:

If you have --enable-controller-attach-detach option set to true
Then volumes are normally attaching by csi-attacher in linstor-csi-controller pod.

But if you have --enable-controller-attach-detach option set to false
Then kubelet tries to attach the volume by itself, it is trying to create special resource VolumeAttachment in storage.k8s.io/v1 but this action is restricted by node_authorizer.go

What's that, is it upstream bug? - say me if so, I'll prepare PR for fix that in node_authorizer.go

I have two questions to you right now:

Was your csi-driver was tested with --enable-controller-attach-detach=false, is it correct way to use csi-driver with this option?
You have deprecated old flexvolume driver, ok, but we can't switch to new csi-driver just because we have a lots of volumes created with flexvolume. What is correct way to upgrade them?
Since flexvolume requires --enable-controller-attach-detach=false, but csi requires --enable-controller-attach-detach=true we cant use both drivers in one time.

kvaps · 2019-03-05T12:50:27Z

I can answer by myself.

Was your csi-driver was tested with --enable-controller-attach-detach=false, is it correct way to use csi-driver with this option?

No, this is working very bad, eg. detaching is not working at all.

About using flexvolume and CSI together I was updated flexvolume driver to not use attach/detach operations at all, I was moved them into standard mount/unmount cycle. That's make possible to use current flexvolume driver with --enable-controller-attach-detach=true option which is required by new CSI driver.

Unfortunately I can't create PR, because current repo is archived. All the changes are listed here:
LINBIT/linstor-flexvolume@master...kvaps:enable-attach-detach

deleteriousEffect · 2019-03-05T17:12:35Z

@haySwim hi,

You're saying to can't change modify that RBAC at all?

Еhis problem shouldn't and can't be solved by RBAC, because:
1. Kubelets are using `Node` authorization mode, not `RBAC`, actually this behavior is controlled by [node_authorizer.go](https://github.com/kubernetes/kubernetes/blob/release-1.13/plugin/pkg/auth/authorizer/node/node_authorizer.go)

2. You can't create common rolebinding for any `system:node:*` due inability using wildcards here.
What have you tried?

I have temporary solution with creating rolebinding rule for every my node.

Easy interop between this plugin and the previous two is a non-goal of this project. If you have a workaround, great. If not, you need to migrate your data over to one driver or the other.

What where the error messages you got from those attempts?

From kubelets logs directly

Sorry, that should have been "What were..."

Look, here is two possible ways:

If you have --enable-controller-attach-detach option set to true
Then volumes are normally attaching by csi-attacher in linstor-csi-controller pod.

But if you have --enable-controller-attach-detach option set to false
Then kubelet tries to attach the volume by itself, it is trying to create special resource VolumeAttachment in storage.k8s.io/v1 but this action is restricted by node_authorizer.go

What's that, is it upstream bug? - say me if so, I'll prepare PR for fix that in node_authorizer.go

I'm not actually sure how that is supposed to interact with Kubernetes's CSI implementation i.e., not this plugin, or the CSI spec, but the actual Kubernetes CSI components. For what it's worth, our plugin is tested and intended to be used with --enable-controller-attach-detach=true

I have two questions to you right now:

1. Was your csi-driver was tested with `--enable-controller-attach-detach=false`, is it correct way to use csi-driver with this option?

No. The default, --enable-controller-attach-detach=true, is required.

2. You have deprecated old flexvolume driver, ok, but we can't switch to new csi-driver just because we have a lots of volumes created with flexvolume. What is correct way to upgrade them?

You'll have to create new volumes with CSI and migrate (or recreate) your data manually. The exact method that you use to do this is up to you, you know your data best. These drivers, although they both talk to LINSTOR, have different internal representations of volumes and different capacities and there's no obviously right way to "register" the old volumes with CSI.

I think this issue should be noted in our documentation. I'm sure there's a very sleek and clever way to do a migration in a semi-automatic way, but it would fundamentally be a data migration and this feature would not be perennially useful as more and more people move away from the old driver.

   Since flexvolume requires `--enable-controller-attach-detach=false`, but csi requires `--enable-controller-attach-detach=true` we cant use both drivers in one time.

Correct.

I can answer by myself.

Please practice a bit of patience, we are providing this software and our assistance to you for free, after all. You should not expect, and will not receive, super high priority response times unless you actually find a serious bug.

Also, I feel that the title change is a bit misleading. So I have edited it to reflect my own understanding of this issue, as is my right as a maintainer of this project.

Was your csi-driver was tested with --enable-controller-attach-detach=false, is it correct way to use csi-driver with this option?

No, this is working very bad, eg. detaching is not working at all.

Again, yes use --enable-controller-attach-detach=true. There was no mention of this because it's the default.

About using flexvolume and CSI together I was updated flexvolume driver to not use attach/detach operations at all, I was moved them into standard mount/unmount cycle. That's make possible to use current flexvolume driver with --enable-controller-attach-detach=true option which is required by new CSI driver.

This is a general Kubernetes issue as far as I can tell: different volume plugins might require conflicting global settings.

Unfortunately I can't create PR, because current repo is archived.

This was the intent behind archiving the repo.

All the changes are listed here:
LINBIT/linstor-flexvolume@master...kvaps:enable-attach-detach

You're welcome to maintain and develop your fork as much as you wish, but it's totally your responsibility. There are currently no plans on our side to put forth any more time and energy towards that project.

kvaps · 2019-03-05T22:26:51Z

Please practice a bit of patience, we are providing this software and our assistance to you for free, after all. You should not expect, and will not receive, super high priority response times unless you actually find a serious bug.

Please do not think that I reproach you for something, I understand everything, you guys are doing a nice project, thank you for that and I'm really glad to help you with that.

It seems my phrase was look a bit rude, sorry if it is so, but believe me I didn't want to hurt you at all.

I've just reported that I've already tested it and can say for sure that driver is not working with this option. Some information is better than lack of information, isn't it?

Also, I feel that the title change is a bit misleading. So I have edited it to reflect my own understanding of this issue, as is my right as a maintainer of this project.

Agree, it's up to you.

I'm not actually sure how that is supposed to interact with Kubernetes's CSI implementation i.e., not this plugin, or the CSI spec, but the actual Kubernetes CSI components. For what it's worth, our plugin is tested and intended to be used with --enable-controller-attach-detach=true

I'm sure that --enable-controller-attach-detach=false will not be used in new installations anymore. CSI provides better interface which is not requires to override global parameters like that.
There was also talks to deprecate it kubernetes/kubernetes#55517

deleteriousEffect · 2019-03-07T22:41:30Z

Yes, --enable-controller-attach-detach is not the prettiest thing. I do appreciate the information of course as it does highlight a gap in documentation. Once that is written up, I'll close this issue.

kvaps · 2019-03-27T21:00:16Z

BTW I wrote small script for convert old flexvolumes to CSI:
https://github.com/kvaps/linstor-csi-migrator

Maybe someone it will be useful

deleteriousEffect closed this as completed Mar 4, 2019

deleteriousEffect reopened this Mar 4, 2019

kvaps changed the title ~~User "system:node:<node>" cannot create resource "volumeattachments" in API group "storage.k8s.io"~~ No way to start using CSI-driver instead FlexVolume Mar 5, 2019

deleteriousEffect changed the title ~~No way to start using CSI-driver instead FlexVolume~~ linstor-csi is not compatible with --enable-controller-attach-detach=false Mar 5, 2019

deleteriousEffect closed this as completed Apr 26, 2019

erikwilson mentioned this issue Aug 28, 2019

CSI plugin functionality is broken: "attacher.Attach failed: volumeattachments.storage.k8s.io is forbidden" k3s-io/k3s#732

Closed

windayski mentioned this issue Dec 9, 2019

attacher.Attach failed: volumeattachments.storage.k8s.io is forbidden ceph/ceph-csi#734

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

linstor-csi is not compatible with `--enable-controller-attach-detach=false` #4

linstor-csi is not compatible with `--enable-controller-attach-detach=false` #4

kvaps commented Mar 2, 2019 •

edited

Loading

kvaps commented Mar 2, 2019

kvaps commented Mar 2, 2019 •

edited

Loading

deleteriousEffect commented Mar 4, 2019

kvaps commented Mar 4, 2019 •

edited

Loading

kvaps commented Mar 5, 2019 •

edited

Loading

deleteriousEffect commented Mar 5, 2019

kvaps commented Mar 5, 2019 •

edited

Loading

deleteriousEffect commented Mar 7, 2019

kvaps commented Mar 27, 2019

linstor-csi is not compatible with --enable-controller-attach-detach=false #4

linstor-csi is not compatible with --enable-controller-attach-detach=false #4

Comments

kvaps commented Mar 2, 2019 • edited Loading

kvaps commented Mar 2, 2019

kvaps commented Mar 2, 2019 • edited Loading

deleteriousEffect commented Mar 4, 2019

kvaps commented Mar 4, 2019 • edited Loading

kvaps commented Mar 5, 2019 • edited Loading

deleteriousEffect commented Mar 5, 2019

kvaps commented Mar 5, 2019 • edited Loading

deleteriousEffect commented Mar 7, 2019

kvaps commented Mar 27, 2019

linstor-csi is not compatible with `--enable-controller-attach-detach=false` #4

linstor-csi is not compatible with `--enable-controller-attach-detach=false` #4

kvaps commented Mar 2, 2019 •

edited

Loading

kvaps commented Mar 2, 2019 •

edited

Loading

kvaps commented Mar 4, 2019 •

edited

Loading

kvaps commented Mar 5, 2019 •

edited

Loading

kvaps commented Mar 5, 2019 •

edited

Loading