secrets-store-csi-driver results in forbidden errors when deployed via Azure ARC onto a K3S cluster #948
Comments
|
@EliiseS Thanks for opening issue. As you mentioned the problem is with missing Roles and Role Bindings. Sync secrets and monitoring needs them to work correctly. Could you try installing them manually? I am looking into possibility that this maybe RBAC issue specific to k3s. |
|
@nilekhc I did some research on the issue, and I hope this helps to further pinpoint the issue. I suspect that the helm chart does not install the rolebindings based on the conditions in the charts/csi-secrets-store-provider-azure/templates/rolebinding.yaml given that for What I did to come to that conclusion:
Result:
For the prom-mdm-convert the issue seems to be the missing |
|
@LeonardHd You are certainly debugging in right direction. Let me give you some context. When we release new Arc extension version, we replace arc-values.yaml into values.yaml to get arc specific configuration and then pack the helm chart. Having said that, Also, I see that you are pinning extension version to |
|
Here is sample output of helm get values after installing extension. ~ $ helm get values -n kube-system sscsi
USER-SUPPLIED VALUES:
Azure:
Cluster:
Cloud: AZUREPUBLICCLOUD
Distribution: kind
Infrastructure: generic
Region: eastus2euap
ResourceId: /subscriptions/<REDACTED>/resourceGroups/akv-arc/providers/Microsoft.Kubernetes/ConnectedClusters/ext-test
Extension:
Name: sscsi
ResourceId: /subscriptions/<REDACTED>/resourceGroups/akv-arc/providers/Microsoft.Kubernetes/ConnectedClusters/ext-test/providers/Microsoft.KubernetesConfiguration/extensions/sscsi
Identity:
MSIAdapterYaml: |
<REDACTED>
Type: SystemAssigned
isEnabled: true
proxySettings:
isCustomCert: false
isProxyEnabled: false
IdentityPrincipalId: <REDACTED>
IdentityType: SystemAssigned
scope: cluster
secrets-store-csi-driver:
enableSecretRotation: true
rotationPollInterval: 30s
syncSecret:
enabled: true |
|
@nilekhc I have installed the extension as follows (on a new Arc-enabled k3s cluster just to be sure 🤓). I kept the version and namespace pinned for now - thanks a lot for the recommendation to not pin it / omit the release train.
I used the following bicep to deploy the extension: |
|
@LeonardHd Could you provide me an output of |
|
Sure, no problem: |
|
Thanks for providing an access to your test env @LeonardHd. I tried it out and it works as expected. Here are some of the outputs. ~$ az connectedk8s connect -n k3s-test -g arc-ext
This operation might take a while...
Downloading helm client for first time. This can take few minutes...
{
"agentPublicKeyCertificate": "<REDACTED>",
"agentVersion": null,
"connectivityStatus": "Connecting",
"distribution": "k3s",
"id": "/subscriptions/<REDACTED>/resourceGroups/arc-ext/providers/Microsoft.Kubernetes/connectedClusters/k3s-test",
"identity": {
"principalId": "<REDACTED>",
"tenantId": "<REDACTED>",
"type": "SystemAssigned"
},
"infrastructure": "generic",
"kubernetesVersion": null,
"lastConnectivityTime": null,
"location": "eastus2euap",
"managedIdentityCertificateExpirationTime": null,
"name": "k3s-test",
"offering": null,
"provisioningState": "Succeeded",
"resourceGroup": "arc-ext",
"systemData": {
"createdAt": "2022-08-01T22:17:13.260943+00:00",
"createdBy": "<REDACTED>",
"createdByType": "User",
"lastModifiedAt": "2022-08-01T22:17:13.260943+00:00",
"lastModifiedBy": "<REDACTED>",
"lastModifiedByType": "User"
},
"tags": {},
"totalCoreCount": null,
"totalNodeCount": null,
"type": "microsoft.kubernetes/connectedclusters"
}~$ az k8s-extension create \
--name k3stest \
--extension-type Microsoft.AzureKeyVaultSecretsProvider \
--scope cluster \
--cluster-name k3s-test \
--resource-group arc-ext \
--cluster-type connectedClusters \
--release-namespace kube-system \
--configuration-settings 'secrets-store-csi-driver.enableSecretRotation=true' \
'secrets-store-csi-driver.rotationPollInterval=30s' \
'secrets-store-csi-driver.syncSecret.enabled=true'
{
"aksAssignedIdentity": null,
"autoUpgradeMinorVersion": true,
"configurationProtectedSettings": {},
"configurationSettings": {
"secrets-store-csi-driver.enableSecretRotation": "true",
"secrets-store-csi-driver.rotationPollInterval": "30s",
"secrets-store-csi-driver.syncSecret.enabled": "true"
},
"customLocationSettings": null,
"errorInfo": null,
"extensionType": "microsoft.azurekeyvaultsecretsprovider",
"id": "/subscriptions/<REDACTED>/resourceGroups/arc-ext/providers/Microsoft.Kubernetes/connectedClusters/k3s-test/providers/Microsoft.KubernetesConfiguration/ex
tensions/k3stest",
"identity": {
"principalId": "<REDACTED>",
"tenantId": null,
"type": "SystemAssigned"
},
"name": "k3stest",
"packageUri": null,
"provisioningState": "Succeeded",
"releaseTrain": "Stable",
"resourceGroup": "arc-ext",
"scope": {
"cluster": {
"releaseNamespace": "kube-system"
},
"namespace": null
},
"statuses": [],
"systemData": {
"createdAt": "2022-08-01T22:25:02.943164+00:00",
"createdBy": null,
"createdByType": null,
"lastModifiedAt": "2022-08-01T22:25:02.943164+00:00",
"lastModifiedBy": null,
"lastModifiedByType": null
},
"type": "Microsoft.KubernetesConfiguration/extensions",
"version": "1.2.1"
}Output of helm get values k3stest -n kube-system does show that Identity.isEnabled is set to Azure:
Cluster:
Cloud: AZUREPUBLICCLOUD
Distribution: k3s
Infrastructure: generic
Region: eastus2euap
ResourceId: /subscriptions/<REDACTED>/resourceGroups/arc-ext/providers/Microsoft.Kubernetes/ConnectedClusters/k3s-test
Extension:
Name: k3stest
ResourceId: /subscriptions/<REDACTED>/resourceGroups/arc-ext/providers/Microsoft.Kubernetes/ConnectedClusters/k3s-test/providers/Microsoft.KubernetesConfigura
tion/extensions/k3stest
Identity:
Type: SystemAssigned
isEnabled: true
proxySettings:
isCustomCert: false
isProxyEnabled: false
IdentityPrincipalId: <REDACTED>
IdentityType: SystemAssigned
scope: cluster
secrets-store-csi-driver:
enableSecretRotation: true
rotationPollInterval: 30s
syncSecret:
enabled: trueAfter creating workload pod it was able to mount secret on the volume. ~$ kubectl exec -it busybox-secrets-store-inline -- cat /mnt/secrets-store/secret1
test |
|
@nilekhc I have managed to dig further, and I believe I have found the issue.
Details In fact, when deploying via the Arc Extension Tab the The difference I noticed between CLI and Portal/ARM is that the This made me add a ARM/Bicep deployment with SystemAssigned identity helm get values for ARM/Bicep deployment with SystemAssigned identity Azure Portal ARM Deployment Azure Portal Deployment Helm Values: |
|
Thanks @LeonardHd for debugging and a PR. I'll review the same |
|
@LeonardHd I have also released these changes in portal. |
Have you
What steps did you take and what happened:
What did you expect to happen:
Secret Storage to work as expected on ARC connected K3S cluster. If there is a better place to post this issue, please let us know :)
Anything else you would like to add:
The bicep template used for deployment, where you can see that the syncSecret is enabled. I've also verified its enabled through the Azure Arc UI in Azure portal:
Here are all of the logs in no particular order: https://gist.github.com/EliiseS/dab2821dd860e9e0402e7843b71dae82
One key takeaway for me has been that the cluster roles are missing, which is most likely the reason the service account is forbidden from doing anything.
Which access mode did you use to access the Azure Key Vault instance:
[e.g. Service Principal, Pod Identity, User Assigned Managed Identity, System Assigned Managed Identity]
Environment:
kubectl versionandkubectl get nodes -o wide):https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/tutorial-akv-secrets-provider
Co-author: @LeonardHd
The text was updated successfully, but these errors were encountered: