Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ISSUE] Issue with databricks_external_location resource #3547

Closed
samuellee-zy opened this issue May 4, 2024 · 2 comments
Closed

[ISSUE] Issue with databricks_external_location resource #3547

samuellee-zy opened this issue May 4, 2024 · 2 comments

Comments

@samuellee-zy
Copy link

Configuration

resource "azurerm_storage_account" "unity_catalog" {
  name                     = "nhvrdemostorageaccuc0405"
  resource_group_name      = data.tfe_outputs.db_workspace.values.databricks_resource_group_name
  location                 = data.tfe_outputs.db_workspace.values.databricks_location
  tags                     =  {
    environment = "demo"
  }
  account_tier             = "Standard"
  account_replication_type = "LRS"
  is_hns_enabled           = true
}

resource "azurerm_databricks_access_connector" "example" {
  name                = "example-resource"
  resource_group_name = data.tfe_outputs.db_workspace.values.databricks_resource_group_name
  location            = data.tfe_outputs.db_workspace.values.databricks_location

  identity {
    type = "SystemAssigned"
  }

  tags = {
    environment = "demo"
  }
}

resource "azurerm_role_assignment" "mi_data_contributor" {
  scope                = azurerm_storage_account.unity_catalog.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_databricks_access_connector.example.identity[0].principal_id
}

// Create a container in storage account to be used by unity catalog metastore as root storage
resource "azurerm_storage_container" "ext_storage" {
  name                  = "nhvr-demo-container"
  storage_account_name  = azurerm_storage_account.unity_catalog.name
  container_access_type = "private"
}

resource "databricks_storage_credential" "external_mi" {
  name = "external_location_mi_credential_2"
  azure_managed_identity {
    access_connector_id = azurerm_databricks_access_connector.example.id
  }
  owner = "4e90b47c-5683-4553-9b01-753bd1248ba1"
  comment    = "Storage credential for all external locations"
  depends_on = [azurerm_databricks_access_connector.example]
}

// Create external location to be used as root storage by dev catalog
resource "databricks_external_location" "ext_storage" {
  name = "dev-catalog-external-location-2"
  url = format("abfss://%s@%s.dfs.core.windows.net",
    azurerm_storage_container.ext_storage.name,
  azurerm_storage_account.unity_catalog.name)
  credential_name = databricks_storage_credential.external_mi.id
  # owner           = "admins"
  comment         = "External location used by dev catalog as root storage"
  depends_on = [ databricks_storage_credential.external_mi, azurerm_storage_container.ext_storage ]
}

resource "databricks_grants" "some" {
  external_location = databricks_external_location.ext_storage.id
  grant {
    principal  = "4e90b47c-5683-4553-9b01-753bd1248ba1"
    privileges = ["ALL_PRIVILEGES"]
  }
}

Expected Behavior

External location should be created within the databricks workspace

Actual Behavior

Terraform applies without any issues, however within the databricks UI, no new external location was created.

image
image
image

Steps to Reproduce

  1. Ensure that you have an operational databricks workspace running (Premium SKU)
  2. terraform apply

Terraform and provider versions

Terraform v1.8.2
on darwin_arm64

  • provider registry.terraform.io/databricks/databricks v1.42.0
  • provider registry.terraform.io/hashicorp/azurerm v3.102.0
  • provider registry.terraform.io/hashicorp/tfe v0.54.0

Is it a regression?

Haven't attempted previous version of provider yet. Using the latest (databricks provider v1.42.0)

@samuellee-zy
Copy link
Author

Additionally, when I try to manually create the external location in the UI, I get this error message:

image

Which I believe validates the fact that the external location is actually created, however it's not showing up in Databricks?

@samuellee-zy
Copy link
Author

Apologies - Closed as I realised I haven't provided the right grants to the user

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant