ForbiddenByRbac when using azure key vault backend with version 3.8+

[andrey-gava]

Hi!
We successfully using sops 3.7.3 with azure key vault as backend.
But when we try to use same flow with 3.8+ version it fails with ForbiddenByRbac error.
I tried both login type - az login and service principle credentials. Both fails.
I have next roles permission to resource: [Key Vault Crypto Officer, Key Vault Crypto User]

Something changed in how sops authenticate with azure resources?

./sops-v3.8.1.linux.amd64 ~/git/environments/aks-saas/secrets.yaml

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  https://*******.vault.azure.net/keys/sops-aks-saas-key/*********************: FAILED
    - | failed to decrypt sops data key with Azure Key Vault key
      | 'https://*******.vault.azure.net/keys/sops-aks-saas-key/*********************':
      | POST
      | https://*******.vault.azure.net/keys/sops-aks-saas-key/*********************/decrypt
      | --------------------------------------------------------------------------------
      | RESPONSE 403: 403 Forbidden
      | ERROR CODE: Forbidden
      | --------------------------------------------------------------------------------
      | {
      |   "error": {
      |     "code": "Forbidden",
      |     "message": "Caller is not authorized to perform action
      | on resource.\r\nIf role assignments, deny assignments or
      | role definitions were changed recently, please observe
      | propagation time.\r\nCaller:
      | appid=********************;oid=*********************;iss=https://sts.windows.net/**************/\r\nAction:
      | 'Microsoft.KeyVault/vaults/keys/decrypt/action'\r\nResource:
      | '/subscriptions/************************/resourcegroups/****/providers/microsoft.keyvault/vaults/*******/keys/sops-aks-saas-key'\r\nAssignment:
      | (not found)\r\nDenyAssignmentId: null\r\nDecisionReason:
      | null \r\nVault: *******;location=********\r\n",
      |     "innererror": {
      |       "code": "ForbiddenByRbac"
      |     }
      |   }
      | }
      | --------------------------------------------------------------------------------

[maonat]

I know this is an old post but... Where is this command being launched? I had the same issue on our self-hosted gh agents when updated to sops latest version and had to rollback to ensure this was working.
In my case, the OS was authenticating to the keyvault using the managed identity on the host and was ignoring completely the Azure/Login action... This probably is related. Try giving the proper key-vault access to the managed identity assigned to the host where you're issuing the command from.

[andrey-gava]

I know this is an old post but... Where is this command being launched? I had the same issue on our self-hosted gh agents when updated to sops latest version and had to rollback to ensure this was working. In my case, the OS was authenticating to the keyvault using the managed identity on the host and was ignoring completely the Azure/Login action... This probably is related. Try giving the proper key-vault access to the managed identity assigned to the host where you're issuing the command from.

I launch it from my laptop. I use simple user credentials that being generated trough az login command. With this credentials I can access all secrets and keys in vault. Also as I said I even tried to use service principle with more broader permission, and it didnt work either.

[andrey-gava]

This error mentioned by other users: #1415

[maonat]

I know this is an old post but... Where is this command being launched? I had the same issue on our self-hosted gh agents when updated to sops latest version and had to rollback to ensure this was working. In my case, the OS was authenticating to the keyvault using the managed identity on the host and was ignoring completely the Azure/Login action... This probably is related. Try giving the proper key-vault access to the managed identity assigned to the host where you're issuing the command from.

I launch it from my laptop. I use simple user credentials that being generated trough az login command. With this credentials I can access all secrets and keys in vault. Also as I said I even tried to use service principle with more broader permission, and it didnt work either.

And what are the permissions? Are you assigning access policies ok the keyvault or using RBAC? (Or because it's either one or the other. If you have already an access policy in place in that keyvault, you won't be able to use RBAC.)

[andrey-gava]

And what are the permissions? Are you assigning access policies ok the keyvault or using RBAC? (Or because it's either one or the other. If you have already an access policy in place in that keyvault, you won't be able to use RBAC.)

We use RBAC access. My user role assignments on KeyVault and subscription level:
[Key Vault Crypto Officer]
[Key Vault Crypto User]
[Contributor]
[Reader]

Flow for user:
Execute az login.
Go trough multi-factor authentication.
Try to decrypt file with 3.7.3 (OK).
Try to decrypt file with 3.9.1 (ERROR)

| RESPONSE 403: 403 Forbidden
      | ERROR CODE: Forbidden
      | --------------------------------------------------------------------------------
      | {
      |   "error": {
      |     "code": "Forbidden",
      |     "message": "Caller is not authorized to perform action
      | on resource.\r\nIf role assignments, deny assignments or
      | role definitions were changed recently, please observe
      | propagation time.

Export mentioned here environments:
https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#hdr-User_with_username_and_password-EnvironmentCredential

export AZURE_TENANT_ID="******"
export AZURE_CLIENT_ID="******"
export AZURE_USERNAME="******"
export AZURE_PASSWORD="******"

Try to decrypt file with 3.7.3 (ERROR).
Try to decrypt file with 3.9.1 (ERROR)

| {"error":"interaction_required","error_description":"AADSTS50076:
      | Due to a configuration change made by your administrator, or
      | because you moved to a new location, you must use
      | multi-factor authentication to access *****

Flow for service principle
export AZURE_CONFIG_DIR="PATH_TO_NONE_DEFAULT_DIR"
Execute az login --service-principal -u *** -p *** --tenant ***
Try to decrypt file with 3.7.3 (OK).
Try to decrypt file with 3.9.1 (ERROR)

| RESPONSE 403: 403 Forbidden
      | ERROR CODE: Forbidden
      | --------------------------------------------------------------------------------
      | {
      |   "error": {
      |     "code": "Forbidden",
      |     "message": "Caller is not authorized to perform action
      | on resource.\r\nIf role assignments, deny assignments or
      | role definitions were changed recently, please observe
      | propagation time.

Export mentioned here environments:
https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#hdr-Service_principal_with_client_secret-EnvironmentCredential

AZURE_TENANT_ID="******"
AZURE_CLIENT_ID="******"
AZURE_CLIENT_SECRET="******"

Try to decrypt file with 3.7.3 (OK).
Try to decrypt file with 3.9.1 (OK)

So in my opinion looks like newer sops version cant get user token with Azure CLI credentials

az --version
azure-cli                         2.59.0 *

core                              2.59.0 *
telemetry                          1.1.0

Extensions:
account                            0.2.5
aks-preview                      0.5.173
desktopvirtualization               1.0.0
storage-preview                    0.8.4

Dependencies:
msal                              1.27.0
azure-mgmt-resource             23.1.0b2

Python location '/opt/az/bin/python3'
Extensions directory '/home/user/.azure/cliextensions'

Python (Linux) 3.11.8 (main, Mar 27 2024, 04:03:04) [GCC 11.4.0]