[ISSUE] Issue with databricks_cluster immediately after workspace creation

[jiropardo]

Configuration

# Copy-paste your Terraform configuration here

module "workspace" {
source = "./modules/workspace"
databricks_account_id = var.databricks_account_id
region = var.region
}

module "catalogs_binding" {
source = "./modules/catalog_bin"
workspace_id = module.workspace.workspace_id

depends_on = [ module.workspace ]

}

module "proxy" {

source = "./modules/proxy_cluster"
depends_on = [ module.workspace ]

}

and the cluster resource within proxy_cluster is

resource "databricks_cluster" "git_proxy" {
autotermination_minutes = 0
aws_attributes {
ebs_volume_count = 1
ebs_volume_size = 32
first_on_demand = 1
}
cluster_name = var.git_proxy_name
custom_tags = {
"ResourceClass" = "SingleNode"
}
provider = databricks.workspace
spark_version = data.databricks_spark_version.latest_lts.id
node_type_id = data.databricks_node_type.smallest.id
num_workers = 0
spark_conf = {
"spark.databricks.cluster.profile" : "singleNode",
"spark.master" : "local[*]",
}
spark_env_vars = {
"GIT_PROXY_ENABLE_SSL_VERIFICATION" : "False"
"GIT_PROXY_HTTP_PROXY" : "git_URL"
}
timeouts {
create = "30m"
update = "30m"
delete = "30m"
}
}

Expected Behavior

Cluster should be created or a more verbose error could be displayed to explain that even though the API returns workspace is RUNNING, it is not yet fully operational

Actual Behavior

Workspace was running

2024-04-10T19:35:44.640-0600 [DEBUG] provider.terraform-provider-databricks_v1.39.0: GET /api/2.0/accounts/XXX/workspaces/XXX
< HTTP/2.0 200 OK
< {
...
< "workspace_status": "RUNNING",

but worker environment is not recognized so cluster creation fails

2024-04-10T19:36:17.232-0600 [ERROR] provider.terraform-provider-databricks_v1.39.0: Response contains error diagnostic: @module=sdk.proto diagnostic_detail="" diagnostic_severity=ERROR tf_proto_version=5.4 tf_req_id=XXX tf_rpc=ApplyResourceChange @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_summary="cannot create cluster: XXX is not able to transition from TERMINATED to RUNNING: worker env WorkerEnvId(workerenv-XXXXX) not found

*Note that I replaced the actual values in above logs with XXXXX

Steps to Reproduce

1.39

Is it a regression?

Debug Output

Important Factoids

Would you like to implement a fix?

Below dependency flow addresses this behavior. It seems this little time is enough for the workspace information to fully propagate

module "proxy" {

source = "./modules/proxy_cluster"

depends_on = [ module.workspace, module.catalogs_binding ] # fixes issue

}