Error handling with volume import feature

[YvosOnTheHub]

Describe the bug
If one tries to import the same volume twice (1 successful attempt & one failed one), tridenctl feedback is not explicit, the 2nd PVC is created & remains in "Pending" state forever

Environment
Provide accurate information about the environment to help us reproduce the issue.
[Lab on Demand v4.0]

• Trident version: 20.10
• Trident installation flags used: operator style
• Container runtime: Docker 18.09.1
• Kubernetes version: 1.18.6
• OS: Red Hat Enterprise Linux Server release 7.5
• NetApp backend types: ONTAP 9.7P2
• Other: life is good, sun is shining, love trident

To Reproduce
1. Create 2 namespaces (volimport1 & volimport2)
2. Create 3 PVC files

[root@rhel3 ~]# more pvc1.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: volimport1
namespace: volimport1
spec:
accessModes:
- ReadWriteMany
storageClassName: storage-class-nas

[root@rhel3 ~]# more pvc2.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: volimport2
namespace: volimport2
spec:
accessModes:
- ReadWriteMany
storageClassName: storage-class-nas

3. Import the volume "vol_to_import" with tridentctl

[root@rhel3 ~]# tridentctl -n trident import volume NAS_Vol-default vol_to_import -f pvc1.yaml
+------------------------------------------+---------+-------------------+----------+--------------------------------------+--------+---------+
| NAME | SIZE | STORAGE CLASS | PROTOCOL | BACKEND UUID | STATE | MANAGED |
+------------------------------------------+---------+-------------------+----------+--------------------------------------+--------+---------+
| pvc-d760648d-00f1-4873-8870-007ff611e1b9 | 1.0 GiB | storage-class-nas | file | 8fab01ed-70b2-4e81-8e63-5ea05f5d312e | online | true |
+------------------------------------------+---------+-------------------+----------+--------------------------------------+--------+---------+
[root@rhel3 ~]# kg pvc,pv -n volimport1
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/volimport1 Bound pvc-d760648d-00f1-4873-8870-007ff611e1b9 1Gi RWX storage-class-nas 7m51s

NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-d760648d-00f1-4873-8870-007ff611e1b9 1Gi RWX Delete Bound volimport1/volimport1 storage-class-nas 7m50s

=> in ONTAP, "vol_to_import" volume rename into "nas1_pvc_d760648d_00f1_4873_8870_007ff611e1b9"

4. Import the same volume into the second namespace

[root@rhel3 ~]# tridentctl -n trident import volume NAS_Vol-default nas1_pvc_d760648d_00f1_4873_8870_007ff611e1b9 -f pvc2.yaml
Error: Post "http://127.0.0.1:8000/trident/v1/volume/import": EOF
command terminated with exit code 1

=> not very clear error ...

5. Look in the logs:

[root@rhel3 ~]# tridentctl -n trident logs
time="2020-11-28T17:11:58Z" level=error msg="GRPC error: rpc error: code = Unknown desc = PV nas1_pvc_d760648d_00f1_4873_8870_007ff611e1b9 already exists for volume pvc-d760648d-00f1-4873-8870-007ff611e1b9" requestID=64fdff60-2a9b-4d63-b41b-6f9205f93e47 requestSource=CSI
time="2020-11-28T17:13:02Z" level=info msg="Found PVC for requested volume pvc-36bc8f11-f2f0-43b8-a158-2b92631766a9." UID=36bc8f11-f2f0-43b8-a158-2b92631766a9 name=volimport2 namespace=volimport2 requestID=e813e843-12e2-478d-85d9-9530f604b36a requestSource=CSI size="{{1073741824 0} {} 1073741824 DecimalSI}" storageClass=storage-class-nas
time="2020-11-28T17:13:02Z" level=info msg="Found storage class for requested volume pvc-36bc8f11-f2f0-43b8-a158-2b92631766a9." name=storage-class-nas requestID=e813e843-12e2-478d-85d9-9530f604b36a requestSource=CSI
time="2020-11-28T17:13:54Z" level=error msg="error waiting for PV pvc-36bc8f11-f2f0-43b8-a158-2b92631766a9; PV pvc-36bc8f11-f2f0-43b8-a158-2b92631766a9 was not in cache after 180.00 seconds" handler=ImportVolume requestID=0cdd3760-b291-4855-b11d-c25d51ebc74b requestSource=REST

=> much better description

6. Clone the ONTAP volume into "vol2_to_import" & try to import it

[root@rhel3 ~]# trident import volume NAS_Vol-default vol2_to_import -f pvc2.yaml
Error: could not import volume: error occurred during PVC creation: persistentvolumeclaims "volimport2" already exists (400 Bad Request)
command terminated with exit code 1

[root@rhel3 ~]# kg pvc --all-namespaces
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
volimport1 volimport1 Bound pvc-d760648d-00f1-4873-8870-007ff611e1b9 1Gi RWX storage-class-nas 15m
volimport2 volimport2 Pending storage-class-nas 13m

[root@rhel3 ~]# kg pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-d760648d-00f1-4873-8870-007ff611e1b9 1Gi RWX Delete Bound volimport1/volimport1 storage-class-nas 15m

=> import fails, logic, since the PVC volimport2 is still there, waiting for Godot

7. Logs

[root@rhel3 ~]# tridentctl -n trident logs
time="2020-11-28T17:21:25Z" level=warning msg="ImportVolume: error occurred during PVC creation." claim=volimport2 error="error occurred during PVC creation: persistentvolumeclaims "volimport2" already exists" requestID=33b75c90-8e53-442a-b704-cf7c34f4e7af requestSource=REST
time="2020-11-28T17:21:25Z" level=error msg="error occurred during PVC creation: persistentvolumeclaims "volimport2" already exists" handler=ImportVolume requestID=33b75c90-8e53-442a-b704-cf7c34f4e7af requestSource=REST

Expected behavior
Instead of the "Error: Post "http://127.0.0.1:8000/trident/v1/volume/import": EOF" message, I would expect Trident to explicitly say that the volume has already been imported, & also, not to created a 2nd PVC that will remain in the "Pending" state forever or clean up this pending PVC

[scaleoutsean]

For what it's worth, using a self-built Trident v21.01.2:

$ tridentctl version -n trident
+------------------------+------------------------+
|     SERVER VERSION     |     CLIENT VERSION     |
+------------------------+------------------------+
| 21.01.2-custom+unknown | 21.01.2-custom+unknown |
+------------------------+------------------------+

Attempt to import existing volume fails as well:

$ tridentctl import volume -n trident ontap-nas-vlan1-gold trident_pvc_58de0b77_6b46_4056_9e52_45b316aa51d8 -f pvc-gold-nfs-clone.yaml

Error: Post "http://127.0.0.1:8000/trident/v1/volume/import": EOF
command terminated with exit code 1

From the log it's clear why it failed (but error above could be more helpful):

time="2021-04-13T07:52:09Z" level=info msg="Found storage class for requested volume pvc-c07d459d-e3db-4a8c-8986-25b2d24d5aef." name=gold-nfs requestID=f013ff6f-d571-470b-a4d3-4a730c15903d requestSource=CSI
time="2021-04-13T07:52:09Z" level=error msg="GRPC error: rpc error: code = Unknown desc = PV trident_pvc_58de0b77_6b46_4056_9e52_45b316aa51d8 already exists for volume pvc-58de0b77-6b46-4056-9e52-45b316aa51d8" requestID=f013ff6f-d571-470b-a4d3-4a730c15903d requestSource=CSI
time="2021-04-13T07:52:28Z" level=error msg="error waiting for PV pvc-c07d459d-e3db-4a8c-8986-25b2d24d5aef; PV pvc-c07d459d-e3db-4a8c-8986-25b2d24d5aef was not in cache after 180.00 seconds" handler=ImportVolume requestID=e2d90d33-2277-475e-96e1-6c5c373e2397 requestSource=REST
time="2021-04-13T07:52:39Z" level=info msg="Found PVC for requested volume pvc-c07d459d-e3db-4a8c-8986-25b2d24d5aef." UID=c07d459d-e3db-4a8c-8986-25b2d24d5aef name=resized namespace=default requestID=43173506-96a7-4c81-8b2c-d653f7b82e24 requestSource=CSI size="{{1073741824 0} {<nil>} 1073741824 DecimalSI}" storageClass=gold-nfs
time="2021-04-13T07:52:39Z" level=info msg="Found storage class for requested volume pvc-c07d459d-e3db-4a8c-8986-25b2d24d5aef." name=gold-nfs requestID=43173506-96a7-4c81-8b2c-d653f7b82e24 requestSource=CSI
time="2021-04-13T07:52:39Z" level=error msg="GRPC error: rpc error: code = Unknown desc = PV trident_pvc_58de0b77_6b46_4056_9e52_45b316aa51d8 already exists for volume pvc-58de0b77-6b46-4056-9e52-45b316aa51d8" requestID=43173506-96a7-4c81-8b2c-d653f7b82e24 requestSource=CSI

Since the import failed, the PVC shouldn't have been assigned but it was.

$ kubectl get pvc
NAME       STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
original   Bound     pvc-58de0b77-6b46-4056-9e52-45b316aa51d8   1Gi        RWO            gold-nfs       9m33s
resized    Pending                                                                        gold-nfs       3m33s

[gnarl]

This issue is fixed in commit c4722ed and will be included in the Trident 22.01 release.