Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Volumes with csi- prefix stoped working (volume not present in storage backend) #7

Closed
kvaps opened this issue Mar 27, 2019 · 7 comments

Comments

@kvaps
Copy link
Member

kvaps commented Mar 27, 2019

Hi, I've just updated csi-provisioner, which is probably implements this fix #3.

Now when I can create new PVCs, volumes are creating for them with the name like: pvc-bbd4682a-5024-11e9-82fd-c2b1dc5386a1 - this is working fine.

But old volumes with the name like csi-0964269a-b154-43a1-876c-d15989446479 are stopped working at all:

csi-attacher log:

I0327 00:31:03.233747       1 controller.go:173] Started VA processing "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.233778       1 csi_handler.go:93] CSIHandler: processing VA "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.233787       1 csi_handler.go:120] Attaching "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.233793       1 csi_handler.go:259] Starting attach operation for "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.233846       1 csi_handler.go:214] PV finalizer is already set on "pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f"
I0327 00:31:03.236093       1 csi_handler.go:524] Can't get CSINodeInfo pve1: csinodeinfos.csi.storage.k8s.io "pve1" not found
I0327 00:31:03.236158       1 csi_handler.go:173] VA finalizer is already set on "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.236166       1 csi_handler.go:187] NodeID annotation is already set on "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.236180       1 connection.go:242] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0327 00:31:03.236187       1 connection.go:243] GRPC request: {"node_id":"pve1","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"storage.kubernetes.io/csiProvisionerIdentity":"1551278453900-8081-io.drbd.linstor-csi"},"volume_id":"deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23"}
I0327 00:31:03.330688       1 connection.go:245] GRPC response: {}
I0327 00:31:03.331017       1 connection.go:246] GRPC error: rpc error: code = NotFound desc = ControllerPublishVolume failed for deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23: volume not present in storage backend
I0327 00:31:03.331030       1 csi_handler.go:388] Saving attach error to "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.333783       1 controller.go:139] Ignoring VolumeAttachment "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec" change
I0327 00:31:03.333827       1 csi_handler.go:398] Saved attach error to "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.333844       1 csi_handler.go:103] Error processing "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec": failed to attach: rpc error: code = NotFound desc = ControllerPublishVolume failed for deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23: volume not present in storage backend

Example PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: io.drbd.linstor-csi
  creationTimestamp: "2019-02-27T16:57:07Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/io-drbd-linstor-csi
  name: pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f
  resourceVersion: "40803092"
  selfLink: /api/v1/persistentvolumes/pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f
  uid: b7316819-3ab0-11e9-8fc7-f6cd6192cc92
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Mi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: eig
    namespace: default
    resourceVersion: "29068182"
    uid: 86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f
  csi:
    driver: io.drbd.linstor-csi
    fsType: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1551278453900-8081-io.drbd.linstor-csi
    volumeHandle: deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23
  persistentVolumeReclaimPolicy: Delete
  storageClassName: linstor-basic-storage
  volumeMode: Filesystem
status:
  phase: Bound

Linstor does not provides any opportunity for rename already created resources, and Kubernetes does not provides to change spec.csi.volumeHandle for the already created PV.

Does it mean, that this situation have no other solution than manually edit linstor database for remove csi- prefix?

@kvaps
Copy link
Member Author

kvaps commented Mar 27, 2019

I've more investigated in this, I have the next behavior:

If spec.csi.volumeHandle equals deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 (this is PV crerated by old linstor-csi-provisioner), I have an error volume not present in storage backend:

time="2019-03-27T13:27:52Z" level=debug msg="looking up resource by CSI volume ID" func="github.com/LINBIT/linstor-csi/pkg/client.(*Linstor).GetByID" file="/home/hayley/projects/linstor-csi/pkg/client/linstor.go:257" annotationsKey=csi-volume-annotations controllers="localhost:3376" csiVolumeID=deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 linstorCSIComponent=client
time="2019-03-27T13:27:52Z" level=info msg="running external command" func=github.com/LINBIT/golinstor.ResourceDeployment.traceCombinedOutput file="/home/hayley/go/pkg/mod/github.com/!l!i!n!b!i!t/golinstor@v0.12.1/linstor.go:337" command="linstor -m --controllers localhost:3376 resource-definition list" package=golinstor resourceName=CSIGetByID
time="2019-03-27T13:27:52Z" level=error msg="method failed" func=github.com/LINBIT/linstor-csi/pkg/driver.Driver.Run.func1 file="/home/hayley/projects/linstor-csi/pkg/driver/driver.go:714" error="rpc error: code = NotFound desc = ControllerPublishVolume failed for deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23: volume not present in storage backend" linstorCSIComponent=driver method=/csi.v1.Controller/ControllerPublishVolume nodeID=pve2 provisioner=io.drbd.linstor-csi req="volume_id:\"deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23\" node_id:\"pve1\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1551278453900-8081-io.drbd.linstor-csi\" > " resp="<nil>" version=v0.3.2

Otherwise if spec.csi.volumeHandle equals csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23, I have an another error:

time="2019-03-27T13:29:01Z" level=debug msg="looking up resource by CSI volume ID" func="github.com/LINBIT/linstor-csi/pkg/client.(*Linstor).GetByID" file="/home/hayley/projects/linstor-csi/pkg/client/linstor.go:257" annotationsKey=csi-volume-annotations controllers="localhost:3376" csiVolumeID=csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 linstorCSIComponent=client
time="2019-03-27T13:29:01Z" level=info msg="running external command" func=github.com/LINBIT/golinstor.ResourceDeployment.traceCombinedOutput file="/home/hayley/go/pkg/mod/github.com/!l!i!n!b!i!t/golinstor@v0.12.1/linstor.go:337" command="linstor -m --controllers localhost:3376 resource-definition list" package=golinstor resourceName=CSIGetByID
time="2019-03-27T13:29:01Z" level=debug msg="converted resource definition to volume" func="github.com/LINBIT/linstor-csi/pkg/client.(*Linstor).resDefToVolume" file="/home/hayley/projects/linstor-csi/pkg/client/linstor.go:154" annotationsKey=csi-volume-annotations controllers="localhost:3376" linstorCSIComponent=client resourceDefinition="{VlmDfns:[{VlmDfnUUID:067622a9-4af7-408a-9869-2065cb138046 VlmMinor:1007 VlmNr:0 VlmSize:51201}] RscDfnSecret:c7qk4vl6mfuCbKFCE4EW RscDfnUUID:6f530b5f-a63c-441c-b606-d9496764db62 RscName:csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 RscDfnPort:7007 RscDfnProps:[{Value:{\"name\":\"pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f\",\"id\":\"deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23\",\"createdBy\":\"io.drbd.linstor-csi\",\"creationTime\":\"2019-02-27T16:57:06.125301711Z\",\"sizeBytes\":52428800,\"readonly\":false,\"parameters\":{\"autoPlace\":\"2\",\"storagePool\":\"pve\"}} Key:Aux/csi-volume-annotations} {Value:PVE2 Key:DrbdPrimarySetOn}]}" volume="&{Name:pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f ID:deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 CreatedBy:io.drbd.linstor-csi CreationTime:2019-02-27 16:57:06.125301711 +0000 UTC SizeBytes:52428800 Readonly:false Parameters:map[autoPlace:2 storagePool:pve]}"
time="2019-03-27T13:29:01Z" level=info msg="found existing volume" func=github.com/LINBIT/linstor-csi/pkg/driver.Driver.ControllerPublishVolume file="/home/hayley/projects/linstor-csi/pkg/driver/driver.go:514" existingVolume="&{Name:pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f ID:deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 CreatedBy:io.drbd.linstor-csi CreationTime:2019-02-27 16:57:06.125301711 +0000 UTC SizeBytes:52428800 Readonly:false Parameters:map[autoPlace:2 storagePool:pve]}" linstorCSIComponent=driver nodeID=pve2 provisioner=io.drbd.linstor-csi version=v0.3.2
time="2019-03-27T13:29:01Z" level=info msg="attaching volume" func="github.com/LINBIT/linstor-csi/pkg/client.(*Linstor).Attach" file="/home/hayley/projects/linstor-csi/pkg/client/linstor.go:310" annotationsKey=csi-volume-annotations controllers="localhost:3376" linstorCSIComponent=client targetNode=pve1 volume="&{Name:pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f ID:deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 CreatedBy:io.drbd.linstor-csi CreationTime:2019-02-27 16:57:06.125301711 +0000 UTC SizeBytes:52428800 Readonly:false Parameters:map[autoPlace:2 storagePool:pve]}"
time="2019-03-27T13:29:01Z" level=info msg="running external command" func=github.com/LINBIT/golinstor.ResourceDeployment.traceCombinedOutput file="/home/hayley/go/pkg/mod/github.com/!l!i!n!b!i!t/golinstor@v0.12.1/linstor.go:337" command="linstor -m --controllers localhost:3376 resource list" package=golinstor resourceName=deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23
time="2019-03-27T13:29:01Z" level=info msg="running external command" func=github.com/LINBIT/golinstor.ResourceDeployment.traceCombinedOutput file="/home/hayley/go/pkg/mod/github.com/!l!i!n!b!i!t/golinstor@v0.12.1/linstor.go:337" command="linstor -m --controllers localhost:3376 resource create pve1 deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 -s DfltDisklessStorPool --diskless" package=golinstor resourceName=deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23
time="2019-03-27T13:29:01Z" level=error msg="method failed" func=github.com/LINBIT/linstor-csi/pkg/driver.Driver.Run.func1 file="/home/hayley/projects/linstor-csi/pkg/driver/driver.go:714" error="rpc error: code = Internal desc = ControllerPublishVolume failed for csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23: failed to run command \"[\\n  {\\n    \\\"ret_code\\\": 13835058055302349101, \\n    \\\"correction\\\": \\\"Create a resource definition with the name 'deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23' first.\\\", \\n    \\\"error_report_ids\\\": [\\n      \\\"5C9ABDCA-00000-000312\\\"\\n    ], \\n    \\\"obj_refs\\\": [\\n      {\\n        \\\"value\\\": \\\"deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23\\\", \\n        \\\"key\\\": \\\"RscDfn\\\"\\n      }\\n    ], \\n    \\\"details\\\": \\\"Node(s): 'pve1', Resource: 'deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23'\\\", \\n    \\\"message\\\": \\\"Resource definition 'deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23' not found.\\\", \\n    \\\"cause\\\": \\\"The specified resource definition 'deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23' could not be found in the database\\\"\\n  }\\n]\\n\": error status from one or more linstor operations: [{\"details_format\":\"\",\"message_format\":\"\",\"obj_refs\":[{\"value\":\"deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23\",\"key\":\"RscDfn\"}],\"variables\":null,\"ret_code\":13835058055302349101}]" linstorCSIComponent=driver method=/csi.v1.Controller/ControllerPublishVolume nodeID=pve2 provisioner=io.drbd.linstor-csi req="volume_id:\"csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23\" node_id:\"pve1\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1551278453900-8081-io.drbd.linstor-csi\" > " resp="<nil>" version=v0.3.2

It tries to call:

linstor -m --controllers localhost:3376 resource create pve1 deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 -s DfltDisklessStorPool --diskless

that's weird because linstor know this resource as csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23:

linstor r l -r csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23
+----------------------------------------------------------------------------+
| ResourceName                             | Node | Port | Usage  |    State |
|----------------------------------------------------------------------------|
| csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 | pve2 | 7007 | Unused | UpToDate |
| csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 | pve3 | 7007 | Unused | UpToDate |
+----------------------------------------------------------------------------+

@deleteriousEffect
Copy link
Contributor

can I get the output of linstor -m resource-definition list

@kvaps
Copy link
Member Author

kvaps commented Mar 27, 2019

@haySwim sure:

linstor-rd-list.json.txt

@deleteriousEffect
Copy link
Contributor

Alright, I understand the issue. Give me a day or so to think about how to solve it in an intelligent way.

@kvaps
Copy link
Member Author

kvaps commented Mar 29, 2019

Hi @haySwim,

I solved this problem by modifying metadata of my old volumes, what have been done:

  • Every resource in linstor was updated property Aux/csi-volume-annotations like:
{
   "name": "pvc-a4303601-3ab0-11e9-baf1-22b1a44f6c4f",
-  "id": "9b527570-bedb-438b-9745-3bada7bbc83e",
+  "id": "csi-9b527570-bedb-438b-9745-3bada7bbc83e",
   "createdBy": "io.drbd.linstor-csi",
   "creationTime": "2019-02-27T16:57:52.909364881Z",
   "sizeBytes": 52428800,
   "readonly": false,
   "parameters": {
     "ReplicasOnDifferen": "side",
     "autoPlace": "2",
     "storagePool": "pve"
   }
 }
  • Every kubernetes PV was updated like:
 apiVersion: v1
 kind: PersistentVolume
 metadata:
   annotations:
     pv.kubernetes.io/provisioned-by: io.drbd.linstor-csi
   creationTimestamp: "2019-03-29T00:00:49Z"
   finalizers:
   - kubernetes.io/pv-protection
   name: pvc-a4303601-3ab0-11e9-baf1-22b1a44f6c4f
   resourceVersion: "41688434"
   selfLink: /api/v1/persistentvolumes/pvc-a4303601-3ab0-11e9-baf1-22b1a44f6c4f
   uid: b61f757e-51b5-11e9-b606-52b5c95dfe5c
 spec:
   accessModes:
   - ReadWriteOnce
   capacity:
     storage: 50Mi
   claimRef:
     apiVersion: v1
     kind: PersistentVolumeClaim
     name: foo
     namespace: default
     resourceVersion: "29068421"
     uid: a4303601-3ab0-11e9-baf1-22b1a44f6c4f
   csi:
     driver: io.drbd.linstor-csi
     fsType: ext4
     volumeAttributes:
       storage.kubernetes.io/csiProvisionerIdentity: 1551278453900-8081-io.drbd.linstor-csi
-    volumeHandle: 9b527570-bedb-438b-9745-3bada7bbc83e
+    volumeHandle: csi-9b527570-bedb-438b-9745-3bada7bbc83e
   persistentVolumeReclaimPolicy: Delete
   storageClassName: linstor-basic-storage
   volumeMode: Filesystem
 status:
   phase: Bound

After these changes old csi volumes attachment started working properly. For automate process I've prepared small script for generate right commands: https://github.com/kvaps/linstor-csi-migrator

I want to know your opinion: What do you think, is it correct way for solve this problem?
Thanks!

@deleteriousEffect
Copy link
Contributor

I think this is a good solution and I'm happy that you found a workaround.

Right now we're at major version 0 and development is pretty active, so we're not trying especially hard to maintain backwards compatibility. I think ultimately the way that this metadata is stored is going to get changed in breaking way soon-ish, but should stabilize afterwards.

@kvaps
Copy link
Member Author

kvaps commented Mar 29, 2019

OK, great! Hopefully next changes will be more portable to existing volumes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants