"Does not have secrets get permission on key vault" after azd in GHA

[savannahostrowski]

After setting up azd in CI, I am seeing a 403 error when trying to use azd locally --> "does not have secrets get permission on key vault"; can no longer reprovision from local enviroment after trying to set up

@weikanglim per our discussion this afternoon - can you add more details about what might be happening here?

[weikanglim]

If provision runs in GHA targeting the same environment/resource group as the one provisioned locally, this happens. The key workaround is probably to have dev vs. prod separation. Use a different environment name for GHA so that the resources are different. Alternatively, update keyvault.bicep to always include AZURE_PRINCIPAL_ID as well as the object ID of the user that should always have access to KeyVault.

Full explanation: When we provision the KeyVault, we need to assign a principal access to the KeyVault. Since bicep is fully declarative, and the KeyVault API does not support incremental update, the accessPolicies is modified to grant access to the AZURE_PRINCIPAL_ID provided. This means that if the user runs provision using a service principal in pipeline that targets the same environment, the existing KeyVault is updated to allow the principal and not the user.

When the user then runs azd provision locally, the function secretOrRandomPassword fails to access KeyVault due to the permission change. The same behavior currently occurs with https://github.com/azure-samples/todo-csharp-sql

[savannahostrowski]

cc: @gkulin for docs updates

[ellismg]

Full explanation: When we provision the KeyVault, we need to assign a principal access to the KeyVault. Since bicep is fully declarative, and the KeyVault API does not support Azure/bicep#9981, the accessPolicies is modified to grant access to the AZURE_PRINCIPAL_ID provided. This means that if the user runs provision using a service principal in pipeline that targets the same environment, the existing KeyVault is updated to allow the principal and not the user.

The linked issue discusses about how Azure RBAC could be used instead of an incremental update of the accessPolicies property of the resource. Should we be exploring that in our templates instead? That would allow us to create separate role assignment objects per principal to grant access to, which should ameliorate this problem, correct, since the previously role assignments would not be touched as part of the bicep update when a new one is created?

[weikanglim]

Do you mean for KeyVault RBAC? What we've heard and seen in the past is that KeyVault RBAC has eventual consistency semantics that makes it harder to work with. It takes a few minutes to propagate the effectual changes which isn't quite acceptable inside an automated provisioning workflow. I haven't looked at whether this has improved recently.

[rajeshkamal5050]

@gkulin Let us capture it as part of https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/troubleshoot

Sharing environment/resource group when running/provisioning local vs GHA errors does not have secrets get permission on key vault caused by permissions change i.e., user accessing keyvault vs principal accessing keyvault workaround is to use different environment/resource group for local vs GHA cc @weikanglim @ellismg