[Azure] Unity Catalog: Perform permission check during Storage Credential creation instead of role ID lookup

[mixam24]

Describe the issue

I'm trying to create a new storage credential in Unity Catalog using my personal user that has Unity Metastore Admin permissions. Due to security reasons, our organization does not allow personal users to have the built-in "Contributor" role. Instead, we assign a custom one, which is almost identical to the built-in role. Hence, my personal identity does not have built-in "Contributor" role.

The custom role definition is as follows:

{
  "assignableScopes": [
    "/subscriptions/6e85b02d-ec23-493e-bf2a-4be65492e6b5",
    "/subscriptions/96875dfe-c52a-4851-83b0-254863be206d",
    "/providers/Microsoft.Management/managementgroups/2d6bfa44-b523-4579-a70d-a145eaf272d5"
  ],
  "createdBy": "70b08e41-e31b-43db-8c57-b2ed5681d82c",
  "createdOn": "2019-07-24T12:59:19.599048+00:00",
  "description": "[Version 1.2] Grants full access to manage all resources, but does not allow you to assign roles in Azure RBAC, manage assignments in Azure Blueprints, or share image galleries.",
  "id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleDefinitions/ee72b902-9895-4cbb-9a54-10ee567a1a52",
  "name": "ee72b902-9895-4cbb-9a54-10ee567a1a52",
  "permissions": [
    {
      "actions": ["*"],
      "condition": null,
      "conditionVersion": null,
      "dataActions": [],
      "notActions": [
        "Microsoft.Authorization/*/Delete",
        "Microsoft.Authorization/*/Write",
        "Microsoft.Authorization/elevateAccess/Action",
        "Microsoft.Blueprint/blueprintAssignments/write",
        "Microsoft.Blueprint/blueprintAssignments/delete",
        "Microsoft.Compute/galleries/share/action",
        "Microsoft.Purview/consents/write",
        "Microsoft.Purview/consents/delete",
        "Microsoft.Subscription/rename/action",
        "Microsoft.Resources/deploymentStacks/manageDenySetting/action"
      ],
      "notDataActions": []
    }
  ],
  "roleName": "Custom Contributor",
  "roleType": "CustomRole",
  "type": "Microsoft.Authorization/roleDefinitions",
  "updatedBy": "366d6245-093c-4bc0-b8a0-dce37cdf383e",
  "updatedOn": "2024-05-16T08:53:05.530192+00:00"
}

The attempt to create a storage credential fails with the following error:
Creation of a storage credential required the contributor role over the corresponding access connector with ID '...'. Please contact your account admin.

I believe that the service performs role assignment lookups only by ID and does not check that my user has the necessary permissions to perform the action.

Steps to reproduce the behavior

1. List the identity role assignments over the databricks access connector

   ~/ ❯ az role assignment list --assignee c0409af9-7ccb-4e18-adf2-93dd50a39adc \
        --include-groups --include-inherited  \
        --scope "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/my-resource-group/providers/Microsoft.Databricks/accessConnectors/my-connector-unity-catalog"
   
   [
    {
      "condition": null,
      "conditionVersion": null,
      "createdBy": "51d2f4f3-b46e-4c80-a394-0fcd87449619",
      "createdOn": "2023-09-23T07:49:44.104274+00:00",
      "delegatedManagedIdentityResourceId": null,
      "description": null,
      "id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleAssignments/441d2705-cbb4-40af-ad30-31d0f3422436",
      "name": "441d2705-cbb4-40af-ad30-31d0f3422436",
      "principalId": "8bc9963a-1e72-4314-8703-3cd680368384",
      "principalName": "My Subscription Group",
      "principalType": "Group",
      "roleDefinitionId": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleDefinitions/acdd72a7-3385-48ef-bd42-f606fba81ae7",
      "roleDefinitionName": "Reader",
      "scope": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f",
      "type": "Microsoft.Authorization/roleAssignments",
      "updatedBy": "51d2f4f3-b46e-4c80-a394-0fcd87449619",
      "updatedOn": "2023-09-23T07:49:44.104274+00:00"
    },
    {
      "condition": null,
      "conditionVersion": null,
      "createdBy": "dde92205-34c0-4028-ba44-b2cd3d096701",
      "createdOn": "2024-03-20T12:55:28.371679+00:00",
      "delegatedManagedIdentityResourceId": null,
      "description": null,
      "id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleAssignments/82daa968-312f-42e7-8934-7c596545607b",
      "name": "82daa968-312f-42e7-8934-7c596545607b",
      "principalId": "8bc9963a-1e72-4314-8703-3cd680368384",
      "principalName": "My Subscription Group",
      "principalType": "Group",
      "roleDefinitionId": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleDefinitions/ee72b902-9895-4cbb-9a54-10ee567a1a52",
      "roleDefinitionName": "Custom Contributor",
      "scope": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f",
      "type": "Microsoft.Authorization/roleAssignments",
      "updatedBy": "dde92205-34c0-4028-ba44-b2cd3d096701",
      "updatedOn": "2024-03-13T11:55:33.378901+00:00"
    }
   ]
   

2. Describe the Custom Contributor role definition

   ~/ ❯ az role definition list --name "Custom Contributor"
   
   [
    {
      "assignableScopes": [
        "/subscriptions/6e85b02d-ec23-493e-bf2a-4be65492e6b5",
        "/subscriptions/96875dfe-c52a-4851-83b0-254863be206d",
        "/providers/Microsoft.Management/managementgroups/2d6bfa44-b523-4579-a70d-a145eaf272d5"
      ],
      "createdBy": "70b08e41-e31b-43db-8c57-b2ed5681d82c",
      "createdOn": "2019-07-24T12:59:19.599048+00:00",
      "description": "[Version 1.2] Grants full access to manage all resources, but does not allow you to assign roles in Azure RBAC, manage assignments in Azure Blueprints, or share image galleries.",
      "id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleDefinitions/ee72b902-9895-4cbb-9a54-10ee567a1a52",
      "name": "ee72b902-9895-4cbb-9a54-10ee567a1a52",
      "permissions": [
        {
          "actions": [
            "*"
          ],
          "condition": null,
          "conditionVersion": null,
          "dataActions": [],
          "notActions": [
            "Microsoft.Authorization/*/Delete",
            "Microsoft.Authorization/*/Write",
            "Microsoft.Authorization/elevateAccess/Action",
            "Microsoft.Blueprint/blueprintAssignments/write",
            "Microsoft.Blueprint/blueprintAssignments/delete",
            "Microsoft.Compute/galleries/share/action",
            "Microsoft.Purview/consents/write",
            "Microsoft.Purview/consents/delete",
            "Microsoft.Subscription/rename/action",
            "Microsoft.Resources/deploymentStacks/manageDenySetting/action"
          ],
          "notDataActions": []
        }
      ],
      "roleName": "Custom Contributor",
      "roleType": "CustomRole",
      "type": "Microsoft.Authorization/roleDefinitions",
      "updatedBy": "366d6245-093c-4bc0-b8a0-dce37cdf383e",
      "updatedOn": "2024-05-16T08:53:05.530192+00:00"
    }
   ]
   

3. Obtain AAD access token and set databricks host

   ~/ ❯ export DATABRICKS_HOST="https://adb-1234567880123456.7.azuredatabricks.net"
   ~/ ❯ export DATABRICKS_TOKEN=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d | jq .accessToken -r)
   

4. Demonstrate that the identity used has Metastore Admin permissions

   ~/ ❯ databricks unity-catalog storage-credentials list
   
   {
    "storage_credentials": [
      {
        "name": "ef6d050e-4555-4d8c-bab6-587ee1fea7f2-storage-credential-1234567890123",
        "azure_managed_identity": {
          "access_connector_id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/databricks-unity-metastore/providers/Microsoft.Databricks/accessConnectors/unity-metastore-westeu",
          "managed_identity_id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/databricks-unity-metastore/providers/Microsoft.ManagedIdentity/userAssignedIdentities/unity-metastore-westeu",
          "credential_id": "c8aab434-60bc-44c0-a00a-84cf95b5c23c"
        },
        "read_only": false,
        "owner": "admin@example.com",
        "id": "21ce434a-78c1-4a44-9850-c6ff2cceea6f",
        "metastore_id": "ef6d050e-4555-4d8c-bab6-587ee1fea7f2",
        "created_at": 1696498690290,
        "created_by": "admin@example.com",
        "updated_at": 1696498690290,
        "updated_by": "admin@example.com",
        "used_for_managed_storage": true,
        "full_name": "ef6d050e-4555-4d8c-bab6-587ee1fea7f2-storage-credential-1234567890123",
        "securable_type": "STORAGE_CREDENTIAL",
        "securable_kind": "STORAGE_CREDENTIAL_AZURE_MI",
        "isolation_mode": "ISOLATION_MODE_OPEN"
      },
      ...
    ]
   }
   

5. Create storage credential using access connector ID and managed identity ID

   ~/ ❯ databricks unity-catalog storage-credentials create --name test \
        --az-mi-access-connector-id "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/my-resource-group/providers/Microsoft.Databricks/accessConnectors/my-connector-unity-catalog" \
        --az-mi-id "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/my-resource-group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my-connector-unity-catalog-mi"
   
        Traceback (most recent call last):
        File "/usr/local/lib/python3.10/site-packages/databricks_cli/sdk/api_client.py", line 166, in perform_query
          resp.raise_for_status()
        File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
          raise HTTPError(http_error_msg, response=self)
      requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://adb-1234567880123456.7.azuredatabricks.net/api/2.1/unity-catalog/storage-credentials
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/usr/local/bin/databricks", line 8, in <module>
          sys.exit(cli())
        File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
          return self.main(*args, **kwargs)
        File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
          rv = self.invoke(ctx)
        File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
          return _process_result(sub_ctx.command.invoke(sub_ctx))
        File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
          return _process_result(sub_ctx.command.invoke(sub_ctx))
        File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
          return _process_result(sub_ctx.command.invoke(sub_ctx))
        File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
          return ctx.invoke(self.callback, **ctx.params)
        File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
          return __callback(*args, **kwargs)
        File "/usr/local/lib/python3.10/site-packages/databricks_cli/unity_catalog/cred_cli.py", line 114, in wrapper
          f(*args, **kwargs)
        File "/usr/local/lib/python3.10/site-packages/databricks_cli/configure/config.py", line 67, in decorator
          return function(*args, **kwargs)
        File "/usr/local/lib/python3.10/site-packages/databricks_cli/unity_catalog/cred_cli.py", line 166, in create_credential_cli
          cred_json = UnityCatalogApi(api_client).create_storage_credential(data, skip_val)
        File "/usr/local/lib/python3.10/site-packages/databricks_cli/unity_catalog/api.py", line 103, in create_storage_credential
          return self.client.create_storage_credential(cred_spec, skip_validation)
        File "/usr/local/lib/python3.10/site-packages/databricks_cli/unity_catalog/uc_service.py", line 175, in create_storage_credential
          return self.client.perform_query('POST', url, data=cred_spec, headers=headers)
        File "/usr/local/lib/python3.10/site-packages/databricks_cli/sdk/api_client.py", line 174, in perform_query
          raise requests.exceptions.HTTPError(message, response=e.response)
      requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://adb-1234567880123456.7.azuredatabricks.net/api/2.1/unity-catalog/storage-credentials
       Response from server:
       { 'details': [ { '@type': 'type.googleapis.com/google.rpc.ErrorInfo',
                       'domain': 'unity-catalog.databricks.com',
                       'metadata': { 'accessConnectorId': '/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/my-resource-group/providers/Microsoft.Databricks/accessConnectors/my-connector-unity-catalog'},
                       'reason': 'UC_STORAGE_CREDENTIAL_INVALID_CLOUD_PERMISSIONS'},
                     { '@type': 'type.googleapis.com/google.rpc.RequestInfo',
                       'request_id': 'eb684711-61ca-441b-ad34-8acc1e632b48',
                       'serving_data': ''}],
        'error_code': 'PERMISSION_DENIED',
        'message': 'Creating a storage credential requires the contributor role over '
                   'the corresponding access connector with ID '
                   '/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/my-resource-group/providers/Microsoft.Databricks/accessConnectors/my-connector-unity-catalog. '
                   'Please contact your account admin.'}
   

Expected Behavior

Creation of the storage credential succeeded since the identity used has enough access permissions.

Actual Behavior

The attempt of a storage credential creation is failing with the following error:
Creation of a storage credential required the contributor role over the corresponding access connector with ID '...'. Please contact your account admin.

OS and CLI version

~/ ❯ uname -s
Darwin

~/ ❯ echo $OSTYPE
darwin23.0

~/ ❯ databricks --version

Version 0.17.4

Is this a regression?

n.a.

Debug Logs

n.a.

[mixam24]

Any comments on this one?

[pietern]

@mixam24 This sounds like a product feature/issue, not a CLI issue. I forwarded the question internally.

I doubt that it matters, but could you try doing the same with the new CLI? You're using the legacy Python one.

Installation instructions can be found at https://docs.databricks.com/en/dev-tools/cli/install.html.

[mixam24]

@pietern thanks for your reply.

This sounds like a product feature/issue, not a CLI issue.

I agree, the same problem arises in UI and in terraform (= anything that interacts with REST API).

I forwarded the question internally.

Could you kindly recommend an alternative communication channel for the discussion of this issue? I have already contacted Microsoft support, but I have serious doubts regarding their proficiency in the topic and overall effectiveness of the communication.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.