importing an ontapnas volume panics `trident-main`

[clementnuss]

Describe the bug
Importing an ontapnas volume after a simulated restore of the underlying QTree makes trident-main panic.

Environment

• Trident version: 22.04.0
• Container runtime: containerd github.com/containerd/containerd v1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
• Kubernetes version: 1.24.4
• Kubernetes orchestrator: kubeadm
• Kubernetes enabled feature gates: n/a
• OS: Debian 11
• NetApp backend types: ONTAP0 NAS
• Other: Volumes are mounted through NFS

To Reproduce
simulate a disaster recovery: delete a PVC with reclaimPolicy delete. after a few minutes the PV and the data is gone from the ONTAP NAS.

Now restore the QTree with your backup software (in this case Veritas NetBackup), and try importing the PVC again.

Expected behavior
the PVC/PV pair should be recreated and accessible
Additional context

the command used to import the PVC is the following:

/usr/local/bin/tridentctl -n kube-trident import volume ontapnas <volumeAttributes.internalName> -f pvc-data01.yaml

Logs

trident-main time="2022-09-06T12:31:25Z" level=error msg="error getting quota rule" error="tree quota for /vol/trident_qtree_pool_#####_NFGLQBHYKQ/####_pvc_b429450a_43e2_4f15_b2be_aa990f635a98 not found" requestID=2c657968-fabd-46f4-ba19-b7ad2b787758 requestSource=REST
trident-main time="2022-09-06T12:31:25Z" level=info msg="Imported an existing volume." handler=ImportVolume requestID=2c657968-fabd-46f4-ba19-b7ad2b787758 requestSource=REST
trident-main 2022/09/06 12:31:25 http: panic serving 127.0.0.1:51240: invalid WriteHeader code 0
trident-main goroutine 2351402 [running]:
trident-main net/http.(*conn).serve.func1()
trident-main     /usr/local/go/src/net/http/server.go:1802 +0xb9
trident-main panic({0x2c9ab20, 0xc001c8fe30})
trident-main     /usr/local/go/src/runtime/panic.go:1047 +0x266
trident-main net/http.checkWriteHeaderCode(...)
trident-main     /usr/local/go/src/net/http/server.go:1089
trident-main net/http.(*response).WriteHeader(0xc0009e27e0, 0x0)
trident-main     /usr/local/go/src/net/http/server.go:1123 +0x5c8
trident-main github.com/netapp/trident/frontend/rest.(*loggingResponseWriter).WriteHeader(0x2f562a0, 0xc00175bcf8)
trident-main     /go/src/github.com/netapp/trident/frontend/rest/logger.go:28 +0x25
trident-main github.com/netapp/trident/frontend/rest.writeHTTPResponse({0x38453f8, 0xc0016cf830}, {0x3828710, 0xc00175bce0}, {0x2f562a0, 0xc00175bcf8}, 0xc0bded5e192f6faa)
trident-main     /go/src/github.com/netapp/trident/frontend/rest/controller_handlers.go:81 +0x252
trident-main github.com/netapp/trident/frontend/rest.AddGeneric.func1()
trident-main     /go/src/github.com/netapp/trident/frontend/rest/controller_handlers.go:192 +0x136
trident-main panic({0x2e4a140, 0x57d69e0})
trident-main     /usr/local/go/src/runtime/panic.go:1038 +0x215
trident-main github.com/netapp/trident/storage_drivers/ontap/api/azgo.(*QuotaEntryType).Volume(...)
trident-main     /go/src/github.com/netapp/trident/storage_drivers/ontap/api/azgo/type-quota-entry.go:168
trident-main github.com/netapp/trident/storage_drivers/ontap/api.OntapAPIZAPI.convertQuota({{0xf, 0x0}}, {0x38453f8, 0xc0016cf830}, {{{0x0, 0x0}, {0x0, 0x0}}, 0x0, 0x0, ...})
trident-main     /go/src/github.com/netapp/trident/storage_drivers/ontap/api/abstraction_zapi.go:1641 +0x2a7
trident-main github.com/netapp/trident/storage_drivers/ontap/api.OntapAPIZAPI.QuotaGetEntry({{0x38e21d0, 0xc000a60fd0}}, {0x38453f8, 0xc0016cf830}, {0xc001304270, 0xc0016cf830}, {0xc002056b80, 0x2c}, {0x33c1ee9, 0x4})
trident-main     /go/src/github.com/netapp/trident/storage_drivers/ontap/api/abstraction_zapi.go:1622 +0x378
trident-main github.com/netapp/trident/storage_drivers/ontap.(*NASQtreeStorageDriver).GetVolumeExternal(0xc00361a800, {0x38453f8, 0xc0016cf830}, {0xc0038e0740, 0x37})
trident-main     /go/src/github.com/netapp/trident/storage_drivers/ontap/ontap_nas_qtree.go:1410 +0xfb
trident-main github.com/netapp/trident/storage.(*StorageBackend).GetVolumeExternal(0xc0000ed030, {0x38453f8, 0xc0016cf830}, {0xc0038e0740, 0x37})
trident-main     /go/src/github.com/netapp/trident/storage/backend.go:557 +0x157
trident-main github.com/netapp/trident/core.(*TridentOrchestrator).GetVolumeExternal(0xc0007e8000, {0x38453f8, 0xc0016cf830}, {0xc0038e0740, 0x37}, {0xc000a304a0, 0x8})
trident-main     /go/src/github.com/netapp/trident/core/orchestrator_core.go:2169 +0x41d
trident-main github.com/netapp/trident/frontend/csi/helpers/kubernetes.(*Plugin).ImportVolume(0xc000500800, {0x38453f8, 0xc0016cf830}, 0xc002dd2140)
trident-main     /go/src/github.com/netapp/trident/frontend/csi/helpers/kubernetes/import.go:86 +0xa2a
trident-main github.com/netapp/trident/frontend/rest.ImportVolume.func1({0xc001a39800, 0x1a5, 0x200})
trident-main     /go/src/github.com/netapp/trident/frontend/rest/controller_handlers.go:681 +0x252
trident-main github.com/netapp/trident/frontend/rest.AddGeneric({0x3828710, 0xc00175bce0}, 0xc00373da00, {0x3845938, 0xc00175bcf8}, 0xc001077820)
trident-main     /go/src/github.com/netapp/trident/frontend/rest/controller_handlers.go:206 +0x36a
trident-main github.com/netapp/trident/frontend/rest.ImportVolume({0x3828710, 0xc00175bce0}, 0xc00373da00)
trident-main     /go/src/github.com/netapp/trident/frontend/rest/controller_handlers.go:658 +0x85
trident-main net/http.HandlerFunc.ServeHTTP(0x0, {0x3828710, 0xc00175bce0}, 0x17)
trident-main     /usr/local/go/src/net/http/server.go:2047 +0x2f
trident-main github.com/netapp/trident/frontend/rest.Logger.func1({0x3833f30, 0xc0009e27e0}, 0xc00373d900)
trident-main     /go/src/github.com/netapp/trident/frontend/rest/logger.go:44 +0x3c3
trident-main net/http.HandlerFunc.ServeHTTP(0xc00373d800, {0x3833f30, 0xc0009e27e0}, 0x580d880)
trident-main     /usr/local/go/src/net/http/server.go:2047 +0x2f
trident-main github.com/gorilla/mux.(*Router).ServeHTTP(0xc0002f3680, {0x3833f30, 0xc0009e27e0}, 0xc00373d700)
trident-main     /go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1cf
trident-main net/http.serverHandler.ServeHTTP({0x3825098}, {0x3833f30, 0xc0009e27e0}, 0xc00373d700)
trident-main     /usr/local/go/src/net/http/server.go:2879 +0x43b
trident-main net/http.(*conn).serve(0xc0007e8500, {0x38453f8, 0xc0004d22a0})
trident-main     /usr/local/go/src/net/http/server.go:1930 +0xb08
trident-main created by net/http.(*Server).Serve
trident-main     /usr/local/go/src/net/http/server.go:3034 +0x4e8

[gnarl]

Hi @clementnuss.

From the error message it appears that the qtree was created using the ontap-nas-economy driver and that you are trying to use to use the ontap-nas economy driver to import a copy of a qtree that was originally created by ONTAP in another FlexVol. The first error in the provided log appears to be trying to find the "tree quota" for an ontap-nas-economy volume/qtree that no longer exists.

Trident does support importing volumes created by the ontap-nas driver. The ontap-nas driver is not designed to work with qtrees though. Support for qtrees only exists in the ontap-nas-economy driver.

Let us know what you are attempting to do and if I've missed anything in the above comment.

[clementnuss]

hi @gnarl

From the error message it appears that the qtree was created using the ontap-nas-economy driver and that you are trying to use to use the ontap-nas economy driver to import a copy of a qtree that was originally created by ONTAP in another FlexVol.

partially correct: it's still on the same FlexVol. the following happened:

• creation of a PVC/PV
• trident (with backend ontap-nas-economy) created the tridentvolume and the underlying storage QTree
• backup of the QTree
• deleted the PVC, which deleted the PV, the tridentvolume and the underlying QTree and Quota on ONTAP
• restored the backup (made with Veritas NetBackup) on the ONTAP NAS, in the exact location that was previously deleted (on the same FlexVol, with the same name)

The first error in the provided log appears to be trying to find the "tree quota" for an ontap-nas-economy volume/qtree that no longer exists.

as qtree was deleted by trident, the corresponding quota was also deleted. I guess when we restore the quota isn't properly restored ? but is the quota actually mandatory ?

Trident does support importing volumes created by the ontap-nas driver. The ontap-nas driver is not designed to work with qtrees though. Support for qtrees only exists in the ontap-nas-economy driver.

we are working with the ontap-nas-economy backend all along the way.

does that make things clearer ?

[clementnuss]

hi @gnarl

isn't my issue related to #343 ?
import a QTree isn't supported ?

[scaleoutsean]

import a QTree isn't supported ?

ontap-nas-economy isn't listed as supported for volume import.

https://docs.netapp.com/us-en/trident-2301/trident-use/vol-import.html