create private endpoint failure in Azure Government Cloud

[nidomoko]

What happened:
When applying the following YAML in Azure Gov Cloud storage provisioning fails because the subscription can't be found. Error and YAML below:

Warning ProvisioningFailed 54s (x2 over 86s) file.csi.azure.com_aks-agentpool-38900945-vmss000001_59df1bf5-68fc-415c-94f4-2f6586312c66 failed to provision volume with StorageClass "reposc": rpc error: code = Internal desc = failed to ensure storage account: Failed to create private DNS zone(privatelink.file.core.windows.net) in resourceGroup(MC_repo_group_repo_usgovarizona), error: privatedns.PrivateZonesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="SubscriptionNotFound" Message="The subscription 'bf714633-d899-4258-ba6a-afa2fd9ab5e9' could not be found."

Below is the azure.json file found for a AKS cluster in gov:
{
"cloud": "AzureUSGovernmentCloud",
"tenantId": "removed",
"subscriptionId": "bf714633-d899-4258-ba6a-afa2fd9ab5e9",
"aadClientId": "msi",
"aadClientSecret": "msi",
"resourceGroup": "MC_repo_group_repo_usgovarizona",
"location": "usgovarizona",
"vmType": "vmss",
"subnetName": "aks-subnet",
"securityGroupName": "aks-agentpool-38900945-nsg",
"vnetName": "aks-vnet-38900945",
"vnetResourceGroup": "",
"routeTableName": "aks-agentpool-38900945-routetable",
"primaryAvailabilitySetName": "",
"primaryScaleSetName": "aks-agentpool-38900945-vmss",
"cloudProviderBackoffMode": "v2",
"cloudProviderBackoff": true,
"cloudProviderBackoffRetries": 6,
"cloudProviderBackoffDuration": 5,
"cloudProviderRateLimit": true,
"cloudProviderRateLimitQPS": 10,
"cloudProviderRateLimitBucket": 100,
"cloudProviderRateLimitQPSWrite": 10,
"cloudProviderRateLimitBucketWrite": 100,
"useManagedIdentityExtension": true,
"userAssignedIdentityID": "removed",
"useInstanceMetadata": true,
"loadBalancerSku": "Standard",
"disableOutboundSNAT": false,
"excludeMasterFromStandardLB": true,
"providerVaultName": "",
"maximumLoadBalancerRuleCount": 250,
"providerKeyName": "k8s",
"providerKeyVersion": ""
}

What you expected to happen:
Expectation is that this driver performs in GOV cloud as it would in Public cloud

How to reproduce it:
Apply below YAML in Azure Gov Cloud:

Describe PVC:

Anything else we need to know?:
The above YAML works in Public cloud as expected. Issues only occur in Gov cloud
I reprod this in two different clusters in Gov cloud
Test 1:
AKS cluster was 1.20.9 and I installed the latest driver version via helm
Test 2:
AKS cluster was 1.21.2

Environment:

• CSI Driver version: latest version
• Kubernetes version (use kubectl version): 1.20.9 and 1.21.2
• OS (e.g. from /etc/os-release): VERSION="18.04.6 LTS (Bionic Beaver)"
• Kernel (e.g. uname -a): Linux aks-agentpool-38900945-vmss000000 5.4.0-1059-azure integrate e2e test with kubernetes/test-infra #62~18.04.1-Ubuntu SMP Tue Sep 14 17:53:18 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: For cluster on 1.20.9, Helm. For cluster on 1.21.2, nothing since it comes installed.
• Others:

[andyzhangx]

could you try remove networkEndpointType:xxx parameter and try again? it looks like there is a bug in private link creation when it's in non public cloud.

[nidomoko]

Hey @andyzhangx I was able to get this to work without the networkEndpointType.

What I did was manually create a storage account, configure the private endpoint, and DNS zone. Then I created the SC and PVC using the below YAML:

Essentially, I manually created the networking components instead of letting the driver do it, and then specified in the parameters the FQDN, the account name, and account resource group.

This seems to be working fine this way:

[andyzhangx]

this issue is already fixed in master branch, and would be in v1.8.0 release