ARO (NFS): subnets under resource group '123' was not found

[TiloGit]

What happened:
SMB Works,
NFS Fails:

Error:
The Resource 'Microsoft.Network/virtualNetworks/subnets' under resource group 'CL-EXT-OCP-NPRD-RG' was not found

What you expected to happen: Get NFS PVC

How to reproduce it: Yes request new PVC

Anything else we need to know?: ARO has existing Network.
cloud-config

{
"tenantId": "633f3069-zzzz-zzzz-zzzz-2ab4251c88ee",
"subscriptionId": e6db90b4-zzzz-zzzz-zzzz-4e81412fef0b,
"resourceGroup": "aro-ctbe8z5v",
"useManagedIdentityExtension": false,
"aadClientId": "25__zzz_removed__",
"aadClientSecret": "s__zzz_removed__"
"vnetResourceGroup": "CL-EXT-OCP-NPRD-RG"
"vnetName": "CL-EXT-OCP-NET-NPRD-VNT"
}

Logs:
csi-azurefile-controller-7db68c9c5c-z8hj7-azurefile_red.log

Environment:

• CSI Driver version: Driver Version: v1.1.0
• Kubernetes version (use kubectl version): OC Server Version: 4.5.31 Kubernetes Version: v1.18.3+e574db2
• OS (e.g. from /etc/os-release): Red Hat Enterprise Linux CoreOS 45.82.202101291733-0 (Ootpa)
• Kernel (e.g. uname -a): 4.18.0-193.41.1.el8_2.x86_64 cri-o://1.18.4-6.rhaos4.5.gitbe796d1.el8
• Install tools: kubectl
• Others: Azure OpenShift (ARO) with exiting Network

[TiloGit]

my guess is that the service account has no read access to the network/subnet
Also if I feed the subnet name via cloud-config "subnetName" do I use Worker or Master subnet of ARO?

[TiloGit]

fixed missing , in cloud-config and added subnetName but similar issue.

[TiloGit]

The SPN (aadClientId) has Contributor access rights to the entire subscription.
The SMB got created by by that SPN,
For NFS getting error: Resource 'Microsoft.Network/virtualNetworks/CL-EXT-OCP-NET-NPRD-VNT' under resource group 'CL-EXT-OCP-NPRD-RG' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}

Log:
csi-azurefile-controller-7db68c9c5c-dsbnb-azurefile - redact.log

[andyzhangx]

would you use bring your own storage account feature first, try example here: https://github.com/kubernetes-sigs/azurefile-csi-driver/tree/v0.9.0/deploy/example/nfs

Create a Premium_LRS Azure storage account with following configurations to support NFS share
account kind: FileStorage
secure transfer required(enable HTTPS traffic only): false
select virtual network of agent nodes in Firewalls and virtual networks

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azurefile-csi
provisioner: file.csi.azure.com
parameters:
  resourceGroup: EXISTING_RESOURCE_GROUP_NAME  # optional, only set this when storage account is not in the same resource group as agent node
  storageAccount: EXISTING_STORAGE_ACCOUNT_NAME

[andyzhangx]

For your original error, could you make sure the following resource id exists? it's not permission issue.

I0301 20:14:33.818812       1 controllerserver.go:166] set vnetResourceID(/subscriptions/e6db90b4-zzzz-zzzz-zzzz-4e81412fef0b/resourceGroups/CL-EXT-OCP-NPRD-RG/providers/Microsoft.Network/virtualNetworks/CL-EXT-OCP-NET-NPRD-VNT/subnets/CL-EXT-OCP-CLS01-NPRD-WORKER-SNT) for NFS protocol
I0301 20:14:33.843071       1 azure_subnetclient.go:131] Received error in subnet.get.request: resourceID: /subscriptions/e6db90b4-zzzz-zzzz-zzzz-4e81412fef0b/resourceGroups/CL-EXT-OCP-NPRD-RG/providers/Microsoft.Network/virtualNetworks/CL-EXT-OCP-NET-NPRD-VNT/subnets/CL-EXT-OCP-CLS01-NPRD-WORKER-SNT, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Network/virtualNetworks/CL-EXT-OCP-NET-NPRD-VNT' under resource group 'CL-EXT-OCP-NPRD-RG' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}

[TiloGit]

Hi Andy,

that is what I currently try.

here my class

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azure-file-nfs
provisioner: file.csi.azure.com
parameters:
  protocol: nfs
  location: canadacentral
  resourceGroup: CL-EXT-OCP-NPRD-RG
  storageAccount: ocpnfsstg
  secretNamespace: kube-system
reclaimPolicy: Retain
volumeBindingMode: Immediate

[TiloGit]

got it working. RG for VNET was different and needed to be fixed in cloud config.