azurerm_databricks_workspace workspace_url and workspace_id are empty.

[jwshive]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Affected Resource(s)

Terraform v0.15.0
provider registry.terraform.io/hashicorp/azurerm v2.56.0

Terraform Configuration Files

<-- in databricks.tf -->

resource "azurerm_databricks_workspace" "databricks" {
  name                = "${var.agency-name}-${var.application-name}-${var.environment}-adb"
  resource_group_name = "zus1-${var.agency-name}-${var.application-name}-${var.environment}-v1-rg"
  location            = var.resource-location
  sku                 = var.databricks-sku
  tags                = merge(local.common_tags, tomap({ "type" = "azure-databricks-workspace" }))

  custom_parameters {
    no_public_ip        = true
    virtual_network_id  = data.terraform_remote_state.networking.outputs.db-virtual-network
    public_subnet_name  = data.terraform_remote_state.networking.outputs.db-public-subnet
    private_subnet_name = data.terraform_remote_state.networking.outputs.db-private-subnet
  }
}

<-- In outputs.tf -->

output "databricks_workspace_id" {
  value = azurerm_databricks_workspace.databricks.workspace_id
}

output "databricks_id" {
  value = azurerm_databricks_workspace.databricks.id
}

output "databricks_domain_url" {
  value = "https://${azurerm_databricks_workspace.databricks.workspace_url}/"
}

Debug Output

  # azurerm_databricks_workspace.databricks must be replaced
-/+ resource "azurerm_databricks_workspace" "databricks" {
      ~ id                          = "/subscriptions/subscription/resourceGroups/resource-group/providers/Microsoft.Databricks/workspaces/agency-dts-dev-adb" -> (known after apply)
      ~ managed_resource_group_id   = "/subscriptions/subscription/resourceGroups/databricks-rg-resource-group" -> (known after apply)
      ~ managed_resource_group_name = "databricks-rg-resource-group" -> (known after apply)
        name                        = "agency-dts-dev-adb"

      + workspace_id                = (known after apply)
      + workspace_url               = (known after apply)
        # (3 unchanged attributes hidden)

      + custom_parameters {
          + no_public_ip        = true # forces replacement
          + private_subnet_name = "agency-dts-dev-private-subnet-1" # forces replacement
          + public_subnet_name  = "agency-dts-dev-public-subnet-1" # forces replacement
          + virtual_network_id  = "/subscriptions/subscription/resourceGroups/respirce=grpi[providers/Microsoft.Network/virtualNetworks/agemcy-dts-dev-vnet-1" # forces replacement
        }
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Changes to Outputs:
  + databricks_domain_url   = (known after apply)
  + databricks_host         = (known after apply)
  ~ databricks_id           = "/subscriptions/subscription/resourceGroups/resource-groupproviders/Microsoft.Databricks/workspaces/agency-dts-dev-adb" -> (known after apply)
  + databricks_workspace_id = (known after apply)

Panic Output

Expected Behaviour

I expect to be shown a workspace_id and a workspace_url for a newly created databricks workspace.

Actual Behaviour

The outputs are empty

🏕 => terraform output
...
databricks_domain_url = "https:///"
databricks_id = "/subscriptions/subscription/resourceGroups/resource-group/providers/Microsoft.Databricks/workspaces/agency-dts-dev-adb"
databricks_workspace_id = ""

Steps to Reproduce

1. terraform apply

Important Factoids

References

• Databricks support for Workspace ID and URL in attributes #6732

[favoretti]

Hi there and thank you for reporting this. From your plan output I see that your workspace will have to be re-created, which can happen if initial deployment has failed. I've ran some local tests, I do not see the behavior you're describing. Are you sure that the workspace that you got deployed is not in a Failed state in the portal? This would cause ID and URL to be empty, albeit showing you an existing Azure resource and each subsequent plan will try to re-create it.

[jwshive]

So you're correct but I was incorrect in assuming successful terraform output meant all resources were created.

This is the output from my apply

🦃 => date ; terraform apply -var-file=../variables.tfvars -auto-approve
Thu Apr 22 07:09:39 EDT 2021
azurerm_data_factory.primary-data_factory: Creating...
azurerm_logic_app_workflow.logic-app-1: Creating...
azurerm_databricks_workspace.databricks: Creating...
azurerm_logic_app_workflow.logic-app-1: Creation complete after 2s [id=/subscriptions/subscription/resourceGroups/resource-group/providers/Microsoft.Logic/workflows/agency-dts-dev-logic-app-1]
azurerm_data_factory.primary-data_factory: Creation complete after 3s [id=/subscriptions/subscription/resourceGroups/resource-group/providers/Microsoft.DataFactory/factories/agency-dts-dev-df]
azurerm_databricks_workspace.databricks: Still creating... [10s elapsed]
azurerm_databricks_workspace.databricks: Still creating... [20s elapsed]
azurerm_databricks_workspace.databricks: Still creating... [30s elapsed]
azurerm_databricks_workspace.databricks: Still creating... [40s elapsed]
azurerm_databricks_workspace.databricks: Creation complete after 48s [id=/subscriptions/subscription/resourceGroups/resource-group/providers/Microsoft.Databricks/workspaces/agency-dts-dev-adb]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

data_factory_name = "idoc-dts-dev-df"
databricks_domain_url = "https:///"
databricks_id = "/subscriptions/subscription/resourceGroups/resource-group/providers/Microsoft.Databricks/workspaces/agency-dts-dev-adb"
databricks_workspace_id = ""

However, looking at the resource with the azure cli:

🏕 => az databricks workspace list --query [].[id,name,provisioningState] -o json
[
  [
    "/subscriptions/subscription/resourceGroups/resource-group/providers/Microsoft.Databricks/workspaces/agency-dts-dev-adb",
    "agency-dts-dev-adb",
    "Failed"
  ]
]

So I guess my question now becomes, why does the Azure SDK or terraform believe this was created successfully?

I ran the apply in debug and found

2021-04-22T07:25:11.539-0400 [DEBUG] provider.terraform-provider-azurerm_v2.56.0_x5: {"status":"Failed","error":{"code":"SubnetMissingNSG","message":"The subnet 'agency-dts-dev-private-subnet-1' doesn't have a network security group attached to it. Please create and attach an empty network security group"}}

So it seems that the workspace never gave any indication that it was successful, but the output suggests it to be so.

[lbergeron01]

I've been facing the same issue for the past hour. Thanks @jwshive, running the debug like you did help me to find this:

2021-05-07T14:57:45.689-0400 [DEBUG] provider.terraform-provider-azurerm_v2.57.0_x5.exe: {"status":"Failed","error":{"code":"SubnetInUseError","message":"The subnet '/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP_NAME>/providers/Microsoft.Network/virtualNetworks/<SUBNET_ID>/subnets/<SUBNET_NAME>' is already in use by workspace '/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP_NAME>/providers/Microsoft.Databricks/workspaces/<WORKSPACE_NAME>'"}}

I think the azurerm_databricks_workspace resource is not handling error received from Azure API.

[stawii]

RTFM :) o.. there is none (at least I cannot find it in Azure docs).. Gladly there is side note about this kind of issue: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/databricks_workspace#managed_resource_group_name - it worked as a charm for me.

TL;DR: Define UNIQUE managed resource group name if you have (or plan) other databricks workspace in that resource group.