Upgrade to 22.07 TridentBackendConfig nas and nas-eco failed state

[zwiy]

Describe the bug
After the upgrade from v21.10.1 (and 22.04.0) to v22.07.0 only the ontap-nas and ontap-nas-economy drivers are stuck in a failed state:

message: Failed to apply the backend update; updating the data plane IP address isn't currently supported

But the IP address wasn't changed.

ONTAP SAN drivers (ontap-san, ontap-san-economy) don't have these problem.

Trident-main Logs:

time="2022-08-16T07:10:33Z" level=error msg="error syncing backend configuration 'trident/backend-tbc-ontap-nas-eco', requeuing; updating the data plane IP address isn't currently supported" logSource=trident-crd-controller requestID=f7cc7b4e-6b78-4cf2-831a-64b489b5278e requestSource=CRD
--
Tue, 16 Aug 2022 09:10:33 | time="2022-08-16T07:10:33Z" level=warning msg="Could not determine controller serial numbers. API status: failed, Reason: Unable to find API: system-node-get-iter, Code: 13005" crdControllerEvent=update requestID=5599b2c3-0edb-4481-88e0-4ae561d97765 requestSource=CRD
Tue, 16 Aug 2022 09:10:33 | time="2022-08-16T07:10:33Z" level=info msg="Storage driver initialized." crdControllerEvent=update driver=ontap-nas-economy requestID=5599b2c3-0edb-4481-88e0-4ae561d97765 requestSource=CRD
Tue, 16 Aug 2022 09:10:33 | time="2022-08-16T07:10:33Z" level=info msg="Created new storage backend." backend="&{0xc00050dc00 backend-tbc-ontap-nas-eco true online map[sr_109430_a_SAS_1:0xc000f43da0 sr_109430_b_SAS_1:0xc000f43e00] map[] false}" crdControllerEvent=update requestID=5599b2c3-0edb-4481-88e0-4ae561d97765 requestSource=CRD
Tue, 16 Aug 2022 09:10:33 | time="2022-08-16T07:10:33Z" level=error msg="Backend update failed." crdControllerEvent=update error="updating the data plane IP address isn't currently supported" requestID=5599b2c3-0edb-4481-88e0-4ae561d97765 requestSource=CRD
Tue, 16 Aug 2022 09:10:33 | time="2022-08-16T07:10:33Z" level=warning msg="error deleting export policy: error deleting export policy: API status: failed, Reason: entry doesn't exist, Code: 15661" crdControllerEvent=update requestID=5599b2c3-0edb-4481-88e0-4ae561d97765 requestSource=CRD
Tue, 16 Aug 2022 09:10:43 | time="2022-08-16T07:10:43Z" level=info msg="New status is same as the old phase, no status update needed." TridentBackendConfigCR=backend-tbc-ontap-nas-eco
Tue, 16 Aug 2022 09:10:43 | time="2022-08-16T07:10:43Z" level=error msg="Invalid backend state." expectedState=online/deleting requestID=29600a04-e450-4b05-bdea-847d918156ea requestSource=Periodic state=failed
Tue, 16 Aug 2022 09:10:43 | time="2022-08-16T07:10:43Z" level=error msg="error syncing backend configuration 'trident/backend-tbc-ontap-nas-eco', requeuing; updating the data plane IP address isn't currently supported" crdControllerEvent=update logSource=trident-crd-controller requestID=5599b2c3-0edb-4481-88e0-4ae561d97765 requestSource=CRD
Tue, 16 Aug 2022 09:10:43 | time="2022-08-16T07:10:43Z" level=info msg=-------------------------------------------------
Tue, 16 Aug 2022 09:10:43 | time="2022-08-16T07:10:43Z" level=info msg=-------------------------------------------------
Tue, 16 Aug 2022 09:10:43 | time="2022-08-16T07:10:43Z" level=error msg="error syncing backend configuration 'trident/backend-tbc-ontap-nas-eco', requeuing; updating the data plane IP address isn't currently supported" logSource=trident-crd-controller requestID=7dc4c32b-74cc-494d-8087-8c6ca702eb6d requestSource=CRD
Tue, 16 Aug 2022 09:11:08 | time="2022-08-16T07:11:08Z" level=error msg="Invalid backend state." expectedState=online/deleting requestID=29600a04-e450-4b05-bdea-847d918156ea requestSource=Periodic state=failed
Tue, 16 Aug 2022 09:11:38 | time="2022-08-16T07:11:38Z" level=error msg="Invalid backend state." expectedState=online/deleting requestID=29600a04-e450-4b05-bdea-847d918156ea requestSource=Periodic state=failed

Current Configuration:

apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: backend-tbc-ontap-nas-eco
spec:
  version: 1
  storageDriverName: ontap-nas-economy
  managementLIF: XXX.XXX.XXX.19
  autoExportPolicy: true
  autoExportCIDRs:
    - XXX.XXX.XXX.0/24
  storagePrefix: int_nas_eco_
  credentials: 
    name: backend-tbc-ontap-nas-eco-secret

I'tried to set the dataLIF IP address but it didn't change anything in the status.

The only way to get it work is deleting the TridentBackendConfig and reapplying the configuration (made for backend-tbc-ontap-nas) - but this doesn't work if there are any volumes associated with the backend:

NAME                        BACKEND NAME                BACKEND UUID                           PHASE   STATUS    STORAGE DRIVER      DELETION POLICY
backend-tbc-ontap-nas       backend-tbc-ontap-nas       4b90f7bb-d688-4017-a2e6-ae494b0ab96c   Bound   Success   ontap-nas           delete
backend-tbc-ontap-nas-eco   backend-tbc-ontap-nas-eco   da0e31f6-4d54-410a-ad7b-de801cd163fe   Bound   Failed    ontap-nas-economy   delete
backend-tbc-ontap-san       backend-tbc-ontap-san       5e95b9fd-4e44-46fa-a7c1-958568abf64f   Bound   Success   ontap-san           delete
backend-tbc-ontap-san-eco   backend-tbc-ontap-san-eco   be0bf334-a1e3-462b-ae66-8237ec8aa30a   Bound   Success   ontap-san-economy   delete

Environment

• Trident version: 22.07
• Kubernetes version: v1.22.10 / v1.21.7+rke2r2
• Kubernetes orchestrator: Rancher v2.6.6 / RKE / RKE2
• OS: Ubuntu 20.04
• NetApp backend types: ONTAP FAS NAS drivers: ontap-nas, ontap-nas-economy

The problem occures on several clusters after the upgrade: RKE/RKE2, v1.22.10 and v1.21.7+rke2r2

Is there another workaround to get this work?

[rohit-arora-dev]

@zwiy

1. I am trying to understand the issue here, the error message you reported "message: Failed to apply the backend update; updating the data plane IP address isn't currently supported", is it a result of adding or modifying or deleting dataLIF? Or did you run into it just by upgrading to 22.07 without any dataLIF modifications?
2. Can you please confirm if the secret backend-tbc-ontap-nas-eco-secret does exist.
3. Can you please provide debug logs, these logs are missing debug information. If using Trident Operator, please use debug under spec or if using tridentctl please use -d flag when re-installing Trident.

[zwiy]

@ntap-arorar

1. I ran into that by upgrading to 22.07. No modifications were made before or afterwards - only helm upgrade .... The dataLIF is not set in the CRD. My first atempt to fix the problem was adding it but it didn't solve the problem.
2. Yes, it does. Credentials are correct (otherwise another warning is logged).
3. I'm using the Trident Operator installed by Helm:

There are a lot of logs: I try to give you the interesting parts (these parts are repeated several times):

time="2022-08-17T05:11:11Z" level=warning msg="Username is specified in both config and secret; overriding from secret."
time="2022-08-17T05:11:11Z" level=warning msg="Password is specified in both config and secret; overriding from secret."
time="2022-08-17T05:11:11Z" level=error msg="Could not initialize storage driver." error="error initializing ontap-nas-economy driver: more than one authentication method (username/password and clientPrivateKey) present in backend config; please ensure only one authentication method is provided" requestID=c224519f-59fc-456b-88c0-5fa35fbcae65 requestSource=Internal
time="2022-08-17T05:11:11Z" level=debug msg="Failed storage backend." backendName=backend-tbc-ontap-nas-eco backendUUID= driver=ontap-nas-economy requestID=c224519f-59fc-456b-88c0-5fa35fbcae65 requestSource=Internal
time="2022-08-17T05:11:11Z" level=debug msg="NewStorageBackendForConfig failed." backend="&{0xc00084a400 backend-tbc-ontap-nas-eco da0e31f6-4d54-410a-ad7b-de801cd163fe false failed map[] map[] 21245ec1-ba18-4c45-9a9b-2c94bd805471 false}" backendUUID=da0e31f6-4d54-410a-ad7b-de801cd163fe configRef=21245ec1-ba18-4c45-9a9b-2c94bd805471 err="problem initializing storage driver 'ontap-nas-economy': error initializing ontap-nas-economy driver: more than one authentication method (username/password and clientPrivateKey) present in backend config; please ensure only one authentication method is provided" requestID=c224519f-59fc-456b-88c0-5fa35fbcae65 requestSource=Internal
time="2022-08-17T05:11:11Z" level=warning msg="Cannot terminate an uninitialized backend." backend=backend-tbc-ontap-nas-eco backendUUID=da0e31f6-4d54-410a-ad7b-de801cd163fe driver=ontap-nas-economy requestID=c224519f-59fc-456b-88c0-5fa35fbcae65 requestSource=Internal state=failed
time="2022-08-17T05:11:11Z" level=warning msg="Problem adding backend." backendErr="problem initializing storage driver 'ontap-nas-economy': error initializing ontap-nas-economy driver: more than one authentication method (username/password and clientPrivateKey) present in backend config; please ensure only one authentication method is provided" handler=Bootstrap newBackendExternal="&{backend-tbc-ontap-nas-eco da0e31f6-4d54-410a-ad7b-de801cd163fe file CommonStorageDriverConfig:&storagedrivers.CommonStorageDriverConfig{Version:1, StorageDriverName:\"ontap-nas-economy\", BackendName:\"backend-tbc-ontap-nas-eco\", Debug:false, DebugTraceFlags:map[string]bool(nil), DisableDelete:false, StoragePrefixRaw:json.RawMessage{0x22, 0x69, 0x6e, 0x74, 0x5f, 0x6e, 0x61, 0x73, 0x5f, 0x65, 0x63, 0x6f, 0x5f, 0x22}, StoragePrefix:(*string)(0xc0005b1500), SerialNumbers:[]string(nil), DriverContext:\"csi\", LimitVolumeSize:\"\", Credentials:map[string]string{\"name\":\"<REDACTED>\", \"type\":\"<REDACTED>\"}} ManagementLIF:\"\" DataLIF:\"\" IgroupName:\"\" SVM:\"\" Username:<REDACTED> Password:<REDACTED> Aggregate:\"\" UsageHeartbeat:\"\" QtreePruneFlexvolsPeriod:\"\" QtreeQuotaResizePeriod:\"\" QtreesPerFlexvol:\"\" LUNsPerFlexvol:\"\" EmptyFlexvolDeferredDeletePeriod:\"\" NfsMountOptions:\"\" LimitAggregateUsage:\"\" AutoExportPolicy:false AutoExportCIDRs:[]string(nil) OntapStorageDriverPool:storagedrivers.OntapStorageDriverPool{Labels:map[string]string(nil), Region:\"\", Zone:\"\", SupportedTopologies:[]map[string]string(nil), OntapStorageDriverConfigDefaults:storagedrivers.OntapStorageDriverConfigDefaults{SpaceAllocation:\"\", SpaceReserve:\"\", SnapshotPolicy:\"\", SnapshotReserve:\"\", SnapshotDir:\"\", UnixPermissions:\"\", ExportPolicy:\"\", SecurityStyle:\"\", SplitOnClone:\"\", FileSystemType:\"\", Encryption:\"\", Mirroring:\"\", TieringPolicy:\"\", QosPolicy:\"\", AdaptiveQosPolicy:\"\", CommonStorageDriverConfigDefaults:storagedrivers.CommonStorageDriverConfigDefaults{Size:\"\"}}} Storage:[]storagedrivers.OntapStorageDriverPool(nil) UseCHAP:false UseREST:false ChapUsername:<REDACTED> ChapInitiatorSecret:<REDACTED> ChapTargetUsername:<REDACTED> ChapTargetInitiatorSecret:<REDACTED> ClientPrivateKey:<REDACTED> ClientCertificate:\"\" TrustedCACertificate:\"\" ReplicationPolicy:\"\" ReplicationSchedule:\"\"  map[] failed false [] 21245ec1-ba18-4c45-9a9b-2c94bd805471}" requestID=c224519f-59fc-456b-88c0-5fa35fbcae65 requestSource=Internal

time="2022-08-17T05:11:38Z" level=error msg="error syncing backend configuration 'trident/backend-tbc-ontap-nas-eco', requeuing; updating the data plane IP address isn't currently supported" logSource=trident-crd-controller requestID=95d700ed-59c2-4cb2-9640-a1c2f3fa0bc4 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="TridentCrdController#processNextWorkItem" logSource=trident-crd-controller requestID=5768a25f-3a7e-4c42-a542-9bbc08039a70 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="TridentCrdController#reconcileBackendConfig" crdControllerEvent=add logSource=trident-crd-controller requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="TridentCrdController#handleTridentBackendConfig" Key=trident/backend-tbc-ontap-nas-eco crdControllerEvent=add eventType=add logSource=trident-crd-controller objectType=trident-backend-config requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="TridentBackendConfig Spec is valid." Spec="backendConfig: map[autoExportCIDRs:[XXX.XXX.52.0/24] autoExportPolicy:true backendName:backend-tbc-ontap-nas-eco credentials:<REDACTED> dataLIF:XXX.XXX.1.16 deletionPolicy:delete managementLIF:XXX.XXX.1.19 storageDriverName:ontap-nas-economy storagePrefix:int_nas_eco_ version:1]" crdControllerEvent=add logSource=trident-crd-controller requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="TridentCrdController#updateBackendConfig" backendConfig.Name=backend-tbc-ontap-nas-eco backendConfig.UID=21245ec1-ba18-4c45-9a9b-2c94bd805471 backendName=backend-tbc-ontap-nas-eco backendUUID=da0e31f6-4d54-410a-ad7b-de801cd163fe crdControllerEvent=add logSource=trident-crd-controller requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Logged EMS message." crdControllerEvent=add driver=ontap-nas-economy requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Shut down EMS logs for the driver." crdControllerEvent=add driver=ontap-nas-economy requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Updating backend in core." backendConfig.Name=backend-tbc-ontap-nas-eco backendConfig.UID=21245ec1-ba18-4c45-9a9b-2c94bd805471 backendName=backend-tbc-ontap-nas-eco backendUUID=da0e31f6-4d54-410a-ad7b-de801cd163fe crdControllerEvent=add logSource=trident-crd-controller requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="found original backend" GetExternalConfig="CommonStorageDriverConfig:&storagedrivers.CommonStorageDriverConfig{Version:1, StorageDriverName:\"ontap-nas-economy\", BackendName:\"backend-tbc-ontap-nas-eco\", Debug:false, DebugTraceFlags:map[string]bool(nil), DisableDelete:false, StoragePrefixRaw:json.RawMessage{0x22, 0x69, 0x6e, 0x74, 0x5f, 0x6e, 0x61, 0x73, 0x5f, 0x65, 0x63, 0x6f, 0x5f, 0x22}, StoragePrefix:(*string)(0xc000b8bb70), SerialNumbers:[]string(nil), DriverContext:\"csi\", LimitVolumeSize:\"\", Credentials:map[string]string{\"name\":\"<REDACTED>\", \"type\":\"<REDACTED>\"}} ManagementLIF:\"\" DataLIF:\"\" IgroupName:\"\" SVM:\"\" Username:<REDACTED> Password:<REDACTED> Aggregate:\"\" UsageHeartbeat:\"\" QtreePruneFlexvolsPeriod:\"\" QtreeQuotaResizePeriod:\"\" QtreesPerFlexvol:\"\" LUNsPerFlexvol:\"\" EmptyFlexvolDeferredDeletePeriod:\"\" NfsMountOptions:\"\" LimitAggregateUsage:\"\" AutoExportPolicy:false AutoExportCIDRs:[]string(nil) OntapStorageDriverPool:storagedrivers.OntapStorageDriverPool{Labels:map[string]string(nil), Region:\"\", Zone:\"\", SupportedTopologies:[]map[string]string(nil), OntapStorageDriverConfigDefaults:storagedrivers.OntapStorageDriverConfigDefaults{SpaceAllocation:\"\", SpaceReserve:\"\", SnapshotPolicy:\"\", SnapshotReserve:\"\", SnapshotDir:\"\", UnixPermissions:\"\", ExportPolicy:\"\", SecurityStyle:\"\", SplitOnClone:\"\", FileSystemType:\"\", Encryption:\"\", Mirroring:\"\", TieringPolicy:\"\", QosPolicy:\"\", AdaptiveQosPolicy:\"\", CommonStorageDriverConfigDefaults:storagedrivers.CommonStorageDriverConfigDefaults{Size:\"\"}}} Storage:[]storagedrivers.OntapStorageDriverPool(nil) UseCHAP:false UseREST:false ChapUsername:<REDACTED> ChapInitiatorSecret:<REDACTED> ChapTargetUsername:<REDACTED> ChapTargetInitiatorSecret:<REDACTED> ClientPrivateKey:<REDACTED> ClientCertificate:\"\" TrustedCACertificate:\"\" ReplicationPolicy:\"\" ReplicationSchedule:\"\" " crdControllerEvent=add originalBackend.BackendUUID=da0e31f6-4d54-410a-ad7b-de801cd163fe originalBackend.ConfigRef=21245ec1-ba18-4c45-9a9b-2c94bd805471 originalBackend.Name=backend-tbc-ontap-nas-eco requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg=">>>>>> updateBackendByBackendUUID" backendName=backend-tbc-ontap-nas-eco backendUUID=da0e31f6-4d54-410a-ad7b-de801cd163fe configJSON="<suppressed>" crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Parsed storage prefix." crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD storagePrefix=int_nas_eco_
time="2022-08-17T05:11:38Z" level=debug msg="Credentials field not empty." crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Parsed commonConfig: Version:1 StorageDriverName:\"ontap-nas-economy\" BackendName:\"backend-tbc-ontap-nas-eco\" Debug:false DebugTraceFlags:map[string]bool(nil) DisableDelete:false StoragePrefixRaw:json.RawMessage{0x22, 0x69, 0x6e, 0x74, 0x5f, 0x6e, 0x61, 0x73, 0x5f, 0x65, 0x63, 0x6f, 0x5f, 0x22} StoragePrefix:(*string)(0xc000c367e0) SerialNumbers:[]string(nil) DriverContext:\"\" LimitVolumeSize:\"\" Credentials:<REDACTED> " crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Retrieved backend secret." crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Initializing storage driver." crdControllerEvent=add driver=ontap-nas-economy requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Configuration defaults" AutoExportCIDRs="[XXX.XXX.52.0/24]" AutoExportPolicy=true Encryption=false ExportPolicy="<automatic>" FileSystemType=ext4 LimitAggregateUsage= LimitVolumeSize= Mirroring=false NfsMountOptions= SecurityStyle=unix Size=1G SnapshotDir=false SnapshotPolicy=none SnapshotReserve= SpaceAllocation=true SpaceReserve=none SplitOnClone=false StoragePrefix=int_nas_eco_ TieringPolicy= UnixPermissions=---rwxrwxrwx crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Addresses found from ManagementLIF lookup." addresses="[XXX.XXX.1.19]" crdControllerEvent=add hostname=XXX.XXX.1.19 requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Using derived SVM." MCC=false SVM=K8s-Int UUID=b03e6d97-e0b8-11eb-836c-d039ea2911fa crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Using SVM." SVM=K8s-Int crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="ONTAP API version." Ontapi=1.201 crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg=NodeListSerialNumbers crdControllerEvent=add desiredAttributes="desired-attributes: { }\nnode-details-info: node-details-info: { }\ncpu-busytime: nil\ncpu-firmware-release: nil\nenv-failed-fan-count: nil\nenv-failed-fan-message: nil\nenv-failed-power-supply-count: nil\nenv-failed-power-supply-message: nil\nenv-over-temperature: nil\nis-all-flash-optimized: nil\nis-all-flash-select-optimized: nil\nis-capacity-optimized: nil\nis-cloud-optimized: nil\nis-diff-svcs: nil\nis-epsilon-node: nil\nis-node-cluster-eligible: nil\nis-node-healthy: nil\nis-perf-optimized: nil\nmaximum-aggregate-size: nil\nmaximum-number-of-volumes: nil\nmaximum-volume-size: nil\nnode: nil\nnode-asset-tag: nil\nnode-location: nil\nnode-model: nil\nnode-nvram-id: nil\nnode-owner: nil\nnode-serial-number: \nnode-storage-configuration: nil\nnode-system-id: nil\nnode-uptime: nil\nnode-uuid: nil\nnode-vendor: nil\nnvram-battery-status: nil\nproduct-version: nil\nsas2-sas3-mixed-stack-support: nil\nvm-system-disks: nil\nvmhost-info: nil\n\n" err="<nil>" info="node-details-info: { }\ncpu-busytime: nil\ncpu-firmware-release: nil\nenv-failed-fan-count: nil\nenv-failed-fan-message: nil\nenv-failed-power-supply-count: nil\nenv-failed-power-supply-message: nil\nenv-over-temperature: nil\nis-all-flash-optimized: nil\nis-all-flash-select-optimized: nil\nis-capacity-optimized: nil\nis-cloud-optimized: nil\nis-diff-svcs: nil\nis-epsilon-node: nil\nis-node-cluster-eligible: nil\nis-node-healthy: nil\nis-perf-optimized: nil\nmaximum-aggregate-size: nil\nmaximum-number-of-volumes: nil\nmaximum-volume-size: nil\nnode: nil\nnode-asset-tag: nil\nnode-location: nil\nnode-model: nil\nnode-nvram-id: nil\nnode-owner: nil\nnode-serial-number: \nnode-storage-configuration: nil\nnode-system-id: nil\nnode-uptime: nil\nnode-uuid: nil\nnode-vendor: nil\nnvram-battery-status: nil\nproduct-version: nil\nsas2-sas3-mixed-stack-support: nil\nvm-system-disks: nil\nvmhost-info: nil\n" requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD response="netapp: { }\nversion,attr: \nxmlns,attr: \nresults: results: { }\nstatus,attr: failed\nreason,attr: Unable to find API: system-node-get-iter\nerrno,attr: 13005\nattributes-list: attributes-list: { }\nnode-details-info: []\n\nnext-tag: nil\nnum-records: 0\n\n"
time="2022-08-17T05:11:38Z" level=warning msg="Could not determine controller serial numbers. API status: failed, Reason: Unable to find API: system-node-get-iter, Code: 13005" crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Qtree driver settings." FlexvolExportPolicy="<automatic>" FlexvolNamePrefix=trident_qtree_pool_int_nas_eco_ QtreesPerFlexvol=200 SharedLockID=b03e6d97-e0b8-11eb-836c-d039ea2911fa-int_nas_eco_ crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Read storage pools assigned to SVM." crdControllerEvent=add pools="[sr_109430_a_SAS_1 sr_109430_b_SAS_1]" requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD svm=K8s-Int
time="2022-08-17T05:11:38Z" level=debug msg="Read aggregate attributes." aggregate=sr_109430_a_SAS_1 crdControllerEvent=add mediaType=hdd requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Read aggregate attributes." aggregate=sr_109430_b_SAS_1 crdControllerEvent=add mediaType=hdd requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg=NewLabelOffer offers="map[]"
time="2022-08-17T05:11:38Z" level=debug msg=NewLabelOffer offers="map[]"
time="2022-08-17T05:11:38Z" level=debug msg="Data LIFs" crdControllerEvent=add dataLIFs="[XXX.XXX.1.16 XXX.XXX.1.26]" requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Found NAS LIFs." crdControllerEvent=add dataLIFs="[XXX.XXX.1.16 XXX.XXX.1.26]" requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Addresses found from hostname lookup." addresses="[XXX.XXX.1.16]" crdControllerEvent=add hostname=XXX.XXX.1.16 requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Found matching Data LIF." crdControllerEvent=add hostNameAddress=XXX.XXX.1.16 requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Configured quota resize period." IntervalSeconds=60 crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Configured EMS heartbeat." crdControllerEvent=add intervalHours=24 requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=info msg="Storage driver initialized." crdControllerEvent=add driver=ontap-nas-economy requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=info msg="Created new storage backend." backend="&{0xc000095800 backend-tbc-ontap-nas-eco  true online map[sr_109430_a_SAS_1:0xc0011984e0 sr_109430_b_SAS_1:0xc001198720] map[]  false}" crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Updating an existing backend." backend=backend-tbc-ontap-nas-eco backend.BackendUUID=da0e31f6-4d54-410a-ad7b-de801cd163fe backendUUID=da0e31f6-4d54-410a-ad7b-de801cd163fe crdControllerEvent=add originalBackend.BackendUUID=da0e31f6-4d54-410a-ad7b-de801cd163fe originalBackend.Name=backend-tbc-ontap-nas-eco requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=error msg="Backend update failed." crdControllerEvent=add error="updating the data plane IP address isn't currently supported" requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="<<<<<< updateBackendByBackendUUID" backendName=backend-tbc-ontap-nas-eco backendUUID=da0e31f6-4d54-410a-ad7b-de801cd163fe configJSON="<suppressed>" crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Terminating backend." backend=backend-tbc-ontap-nas-eco backendUUID=da0e31f6-4d54-410a-ad7b-de801cd163fe crdControllerEvent=add driver=ontap-nas-economy requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD state=online
time="2022-08-17T05:11:38Z" level=debug msg="Attempting to acquire shared lock (resize)." crdControllerEvent=add lock=b03e6d97-e0b8-11eb-836c-d039ea2911fa-int_nas_eco_ requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Acquired shared lock (resize)." crdControllerEvent=add lock=b03e6d97-e0b8-11eb-836c-d039ea2911fa-int_nas_eco_ requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Housekeeping, resizing quotas." crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Released shared lock (resize)." crdControllerEvent=add lock=b03e6d97-e0b8-11eb-836c-d039ea2911fa-int_nas_eco_ requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=warning msg="error deleting export policy: error deleting export policy: API status: failed, Reason: entry doesn't exist, Code: 15661" crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:38Z" level=debug msg="Waiting for housekeeping tasks to exit." crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:39Z" level=debug msg="REST API call received." duration="10.207µs" method=GET requestID=250217ab-8d07-4c56-b832-a67e70a207ed requestSource=REST route=GetVersion uri=/trident/v1/version
time="2022-08-17T05:11:39Z" level=debug msg="REST API call complete." duration="122.613µs" method=GET requestID=250217ab-8d07-4c56-b832-a67e70a207ed requestSource=REST route=GetVersion status_code=200 uri=/trident/v1/version
time="2022-08-17T05:11:45Z" level=debug msg="Node updated in cache." name=int-infra-01 requestID=640d0e37-1f6d-4950-80fa-7f46a124e365 requestSource=Kubernetes
time="2022-08-17T05:11:48Z" level=debug msg="REST API call received." duration="10.272µs" method=GET requestID=b3d04e9b-87c7-4a5b-9711-a20ba496e4c9 requestSource=REST route=GetVersion uri=/trident/v1/version
time="2022-08-17T05:11:48Z" level=debug msg="REST API call complete." duration="96.607µs" method=GET requestID=b3d04e9b-87c7-4a5b-9711-a20ba496e4c9 requestSource=REST route=GetVersion status_code=200 uri=/trident/v1/version
time="2022-08-17T05:11:48Z" level=debug msg="Performing housekeeping task 0." crdControllerEvent=add driver=ontap-nas-economy requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD task=resize tick="2022-08-17 05:11:48.940927336 +0000 UTC m=+38.463023621"
time="2022-08-17T05:11:48Z" level=debug msg="Attempting to acquire shared lock (resize)." crdControllerEvent=add lock=b03e6d97-e0b8-11eb-836c-d039ea2911fa-int_nas_eco_ requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:48Z" level=debug msg="Acquired shared lock (resize)." crdControllerEvent=add lock=b03e6d97-e0b8-11eb-836c-d039ea2911fa-int_nas_eco_ requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:48Z" level=debug msg="Housekeeping, resizing quotas." crdControllerEvent=add requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:48Z" level=debug msg="Released shared lock (resize)." crdControllerEvent=add lock=b03e6d97-e0b8-11eb-836c-d039ea2911fa-int_nas_eco_ requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:48Z" level=debug msg="Shut down housekeeping tasks for the driver." crdControllerEvent=add driver=ontap-nas-economy requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD task=resize
time="2022-08-17T05:11:48Z" level=debug msg="Failed to apply the backend update." TridentBackendConfigCR=backend-tbc-ontap-nas-eco crdControllerEvent=add logSource=trident-crd-controller requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:48Z" level=info msg="New status is same as the old phase, no status update needed." TridentBackendConfigCR=backend-tbc-ontap-nas-eco
time="2022-08-17T05:11:48Z" level=error msg="error syncing backend configuration 'trident/backend-tbc-ontap-nas-eco', requeuing; updating the data plane IP address isn't currently supported" crdControllerEvent=add logSource=trident-crd-controller requestID=65e7f78c-4eef-44d2-bd01-9babf80f07e5 requestSource=CRD
time="2022-08-17T05:11:48Z" level=info msg=-------------------------------------------------
time="2022-08-17T05:11:48Z" level=info msg=-------------------------------------------------

[rohit-arora-dev]

@zwiy
The error error initializing ontap-nas-economy driver: more than one authentication method (username/password and clientPrivateKey) present in backend config; please ensure only one authentication method is provided suggests that your secret backend-tbc-ontap-nas-eco-secret contains two types of storage controller authentication methods i.e. clientPrivateKey as well as username/password.

There was a change in 22.07 that prohibits the use of two authentication methods for a backend, you can only use one. Can you please remove one of the storage controller authentication methods from the secret, please keep the one that is valid. After you modify the secret (backend-tbc-ontap-nas-eco-secret) it will trigger a backend update, which I believe should resolve the above error.

[zwiy]

@ntap-arorar
The secret contains only a username and password like the other one (san, san-eco). Or did I miss something?

~$ k -n trident describe secrets backend-tbc-ontap-nas-eco-secret
Name:         backend-tbc-ontap-nas-eco-secret
Namespace:    trident
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
username:  X bytes
password:  XX bytes

It's exactly the same structure like the san secret.

~$ k -n trident describe secrets backend-tbc-ontap-san-secret
Name:         backend-tbc-ontap-san-secret
Namespace:    trident
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
password:  XX bytes
username:  X bytes

If I delete the failing TridentBackendConfiguration and recreate it with the same YAML afterwards. Everything is working.
The problem is, I can't delete it if there are any volumes using it. So maybe the only workaround would be a "disaster recovery"?

Thanks for your help!

[zwiy]

Maybe this message refers to this issue? #755

[rohit-arora-dev]

@zwiy
You are correct I do not see two authentication methods as the error suggests so this requires more investigation. Can you please open a support case with NetApp Support?
While doing so can you please also include output of kubectl get tbe -o yaml, kubectl get tbc -o yaml and complete controller logs as they will help during the triage.

Thanks!

[rohit-arora-dev]

Hello @zwiy

I investigated this issue and was able to reproduce it. This issue only affects pre-22.07 backends that have been created using the credentials field (i.e. backends using user-defined secrets) and it manifests itself during 22.07 upgrades.

We will work on fixing this issue, in the meantime, there is a workaround to get out of this situation. You will need to do the following:

1. Add ClientPrivateKey: "" to the backend-tbc-ontap-nas-eco-secret secret.
2. Then, restart Trident deployment using the kubectl rollout restart -n trident deployment trident-csi command.

Once the Trident deployment pod are online, the state of your backend should also be online.

[zwiy]

Hello @ntap-arorar

Thank you for your help! Your solution works and all backends are now online.

Have a nice day!

[oranenj]

I hit this same issue, but fixed it with a heavier hammer:

oc patch --as system:admin tridentbackend tbe-zxx2k --type=merge -p '{"config":{"ontap_config":{"clientPrivateKey": ""}}}'

followed by a restart.
This doesn't seem to have broken anything.

[liranse]

Could you please add this issue to release notes?
Spent a lot of time trying to figure it out.

[gnarl]

This issue is fixed with commit 9b78a23 and will be included in the Trident 22.10 release.

[Ujkugri]

We had the same Issue in our ONTAP SAN drivers. The Workaround fixed the problem.