[ISSUE] Issue with `databricks_metastore_assignment` resource #2095

sivadotblog · 2023-03-09T14:35:53Z

We are managing the Unity Catalog terraform in its dedicated state file. Every new workspace we provision is also in its own dedicated state file. So to do a metastore assignment as a part of workspace creation, we are using an alias databricks provider (SPN authenticated) that is an account admin access. But metastore assignment fails.

Configuration

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.13.0"
    }
    databricks = {
      source  = "databricks/databricks"
      version = "1.10.0"
    }
  }
}

provider "databricks" {
  alias               = "account"
  host                = var.dbk-account-host
  account_id          = var.dbk-account-id
  azure_client_id     = var.dbk-acct-admin-client-id
  azure_client_secret = var.dbk-acct-admin-client-secret

}

resource "databricks_metastore_assignment" "metastore-assignment" {
  count        = var.skip-metastore-assignment ? 0 : 1
  provider     = databricks.account
  workspace_id = azurerm_databricks_workspace.adb.workspace_id
  metastore_id = lookup(var.metastore-id, var.location, "eastus2")
  depends_on = [
    azurerm_databricks_workspace.adb
  ]
}

Expected Behavior

Workspace should be assigned to the metastore

Actual Behavior

We get the following error

│ Error: cannot read metastore assignment: invalid character '<' looking for beginning of value
│
│ with module.adb-ws.databricks_metastore_assignment.metastore-assignment[0],
│ on databricks-workspace/module/metastore-assignment.tf line 7, in resource "databricks_metastore_assignment" "metastore-assignment":
│ 7: resource "databricks_metastore_assignment" "metastore-assignment" {
│

Steps to Reproduce

Use a different SPN other than the one used to create the Metastore.
Use an alias provider and spn auth for authenticating to databricks account.
use the resource "databricks_metastore_assignment" to assign the metastore.
Run terraform Plan/Apply

Terraform and provider versions

Terraform version: 1.1.6
I have tried it with different versions of Databricks provider including 1.12.0

Debug Output

2023-03-09T14:06:25.568Z [WARN] Provider "registry.terraform.io/databricks/databricks" produced an invalid plan for module.adb-ws.databricks_metastore_assignment.metastore-assignment[0], but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .default_catalog_name: planned value cty.StringVal("hive_metastore") for a non-computed attribute
2023-03-09T14:06:26.081Z [TRACE] provider.terraform-provider-databricks_v1.10.0: Called downstream: @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema/resource.go:838 @module=sdk.helper_schema tf_rpc=ApplyResourceChange tf_resource_type=databricks_metastore_assignment tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=6f51eed3-c2eb-01bd-3a53-ca7e3c19b513 timestamp=2023-03-09T14:06:26.080Z
2023-03-09T14:06:26.081Z [TRACE] provider.terraform-provider-databricks_v1.10.0: Received downstream response: diagnostic_error_count=1 tf_req_id=6f51eed3-c2eb-01bd-3a53-ca7e3c19b513 @module=sdk.proto diagnostic_warning_count=0 tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_duration_ms=448 tf_resource_type=databricks_metastore_assignment @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tf5serverlogging/downstream_request.go:37 tf_rpc=ApplyResourceChange timestamp=2023-03-09T14:06:26.081Z
2023-03-09T14:06:26.081Z [ERROR] provider.terraform-provider-databricks_v1.10.0: Response contains error diagnostic: diagnostic_severity=ERROR tf_proto_version=5.3 tf_resource_type=databricks_metastore_assignment tf_rpc=ApplyResourceChange @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/diag/diagnostics.go:55 @module=sdk.proto diagnostic_detail= diagnostic_summary="cannot read metastore assignment: invalid character '<' looking for beginning of value" tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=6f51eed3-c2eb-01bd-3a53-ca7e3c19b513 timestamp=2023-03-09T14:06:26.081Z
2023-03-09T14:06:26.081Z [TRACE] provider.terraform-provider-databricks_v1.10.0: Served request: @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server/server.go:831 @module=sdk.proto tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=6f51eed3-c2eb-01bd-3a53-ca7e3c19b513 tf_resource_type=databricks_metastore_assignment tf_rpc=ApplyResourceChange tf_proto_version=5.3 timestamp=2023-03-09T14:06:26.081Z
2023-03-09T14:06:26.081Z [TRACE] maybeTainted: module.adb-ws.databricks_metastore_assignment.metastore-assignment[0] encountered an error during creation, so it is now marked as tainted
2023-03-09T14:06:26.081Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for module.adb-ws.databricks_metastore_assignment.metastore-assignment[0]
2023-03-09T14:06:26.081Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: writing state object for module.adb-ws.databricks_metastore_assignment.metastore-assignment[0]
2023-03-09T14:06:26.081Z [TRACE] evalApplyProvisioners: module.adb-ws.databricks_metastore_assignment.metastore-assignment[0] is tainted, so skipping provisioning
2023-03-09T14:06:26.081Z [TRACE] maybeTainted: module.adb-ws.databricks_metastore_assignment.metastore-assignment[0] was already tainted, so nothing to do
2023-03-09T14:06:26.081Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for module.adb-ws.databricks_metastore_assignment.metastore-assignment[0]
2023-03-09T14:06:26.081Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: writing state object for module.adb-ws.databricks_metastore_assignment.metastore-assignment[0]
2023-03-09T14:06:26.081Z [ERROR] vertex "module.adb-ws.databricks_metastore_assignment.metastore-assignment[0]" error: cannot read metastore assignment: invalid character '<' looking for beginning of value
2023-03-09T14:06:26.081Z [TRACE] vertex "module.adb-ws.databricks_metastore_assignment.metastore-assignment[0]": visit complete, with errors

Important Factoids

I confirmed that the authentication was successful by creating other account level resources such as databicks_groups using the alias provider and it works as expected. this issue is narrowed meatsore assignment.
Few thing:
The Service Principle is an account admin
The SPN is also part of a group that is the metastore owner.

nkvuong · 2023-03-09T18:13:51Z

Unity Catalog APIs are accessible via workspace-level APIs, and this also applies to databricks_metastore_assignment

sivadotblog · 2023-03-09T18:22:49Z

Unity Catalog APIs are accessible via workspace-level APIs, and this also applies to databricks_metastore_assignment

We do not want every workspace creator to be an admin in the account. Workspace Admins create workspaces, UC /Account admins assign the metastore. hence the separation of SPNs

nkvuong · 2023-03-09T18:47:40Z

you can still authenticate as the UC/Account admins via a workspace to achieve this

it is not the best design, so this will be fixed in the next month or two

sivadotblog · 2023-03-09T21:37:54Z

you can still authenticate as the UC/Account admins via a workspace to achieve this

it is not the best design, so this will be fixed in the next month or two

Thank you. This worked. I did have to add the SPN to the workspace as an admin before i can assign.

#OLD
provider "databricks" {
  alias               = "account"
  host                = var.dbk-account-host
  account_id          = var.dbk-account-id
  azure_client_id     = var.dbk-acct-admin-client-id
  azure_client_secret = var.dbk-acct-admin-client-secret

}
#NEW
provider "databricks" {
  alias                       = "ws-assignment"
  host                        = module.adb-ws.adbWorkspaceURL
  azure_workspace_resource_id = module.adb-ws.adbResourceId
  azure_client_id             = var.dbk-acct-admin-client-id
  azure_client_secret         = var.dbk-acct-admin-client-secret

}

kangshung · 2023-05-08T15:53:02Z

I had a very similar issue and solved it in the same way.

I wish the error message was other than looking for < somewhere in the configuration.

jordandakota · 2023-05-09T17:32:50Z

@nkvuong Just checking in, it's been a month or two. Is there a planned fix in the works? This seems to also affect databricks_catalog and databricks_grant resource, likely all of the other unity resources.

nkvuong · 2023-08-30T17:11:23Z

#2182 fixes this for databricks_metastore_assignment

sivadotblog · 2023-09-04T03:31:22Z

@nkvuong

I still get the same error after i upgrade to 1.24 and changed the provider to use account level ADMIN SPN. it stll works with the workspace API

See debug logs:-

module.adb-nat-centralus-ws.databricks_metastore_assignment.metastore-assignment[0]: Refreshing state... [id=874595328895737|e3a40ab2-c820-4761-bfb2-278fc1800e7a]
2023-09-04T03:21:17.163Z [INFO] provider.terraform-provider-databricks_v1.24.0: Generating AAD token for Service Principal (55958b75-d1ee-4736-8b6b-ed4a3dce6f29): timestamp=2023-09-04T03:21:17.163Z
2023-09-04T03:21:17.424Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2023-09-04T03:21:17.426Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/databricks/databricks/1.24.0/linux_amd64/terraform-provider-databricks_v1.24.0 pid=101705
2023-09-04T03:21:17.426Z [DEBUG] provider: plugin exited
2023-09-04T03:21:17.766Z [ERROR] provider.terraform-provider-databricks_v1.24.0: Response contains error diagnostic: @module=sdk.proto diagnostic_severity=ERROR diagnostic_summary="cannot read metastore assignment: invalid character '<' looking for beginning of value" tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/databricks/databricks @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/diag/diagnostics.go:58 tf_req_id=8a096566-84cc-43d9-71fc-02f0fb2d1dc1 tf_rpc=ReadResource tf_resource_type=databricks_metastore_assignment diagnostic_detail= timestamp=2023-09-04T03:21:17.765Z
2023-09-04T03:21:17.766Z [ERROR] vertex "module.adb-nat-centralus-ws.databricks_metastore_assignment.metastore-assignment[0]" error: cannot read metastore assignment: invalid character '<' looking for beginning of value

nkvuong · 2023-09-04T07:22:22Z

@sivadotblog this looks very similar to #2615 - suggesting that it is an authentication issue with the account-level API

sivadotblog · 2023-09-04T11:37:33Z

@sivadotblog this looks very similar to #2615 - suggesting that it is an authentication issue with the account-level API

I dont believe its an auth issue.. using the account level provider I am able to create other objects such as users and groups. the same provider causes this error with metastore assignment . this issue is isolated to metastore assignment.

the Azure SPN is an admin for the workspace, and the account.

nkvuong closed this as completed Aug 30, 2023

sivadotblog mentioned this issue Sep 6, 2023

[ISSUE] Issue with databricks_metastore_assignment resource #2657

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ISSUE] Issue with `databricks_metastore_assignment` resource #2095

[ISSUE] Issue with `databricks_metastore_assignment` resource #2095

sivadotblog commented Mar 9, 2023 •

edited

nkvuong commented Mar 9, 2023

sivadotblog commented Mar 9, 2023 •

edited

nkvuong commented Mar 9, 2023

sivadotblog commented Mar 9, 2023 •

edited

kangshung commented May 8, 2023

jordandakota commented May 9, 2023 •

edited

nkvuong commented Aug 30, 2023

sivadotblog commented Sep 4, 2023

nkvuong commented Sep 4, 2023

sivadotblog commented Sep 4, 2023 •

edited

[ISSUE] Issue with databricks_metastore_assignment resource #2095

[ISSUE] Issue with databricks_metastore_assignment resource #2095

Comments

sivadotblog commented Mar 9, 2023 • edited

Configuration

Expected Behavior

Actual Behavior

Steps to Reproduce

Terraform and provider versions

Debug Output

Important Factoids

nkvuong commented Mar 9, 2023

sivadotblog commented Mar 9, 2023 • edited

nkvuong commented Mar 9, 2023

sivadotblog commented Mar 9, 2023 • edited

kangshung commented May 8, 2023

jordandakota commented May 9, 2023 • edited

nkvuong commented Aug 30, 2023

sivadotblog commented Sep 4, 2023

nkvuong commented Sep 4, 2023

sivadotblog commented Sep 4, 2023 • edited

[ISSUE] Issue with `databricks_metastore_assignment` resource #2095

[ISSUE] Issue with `databricks_metastore_assignment` resource #2095

sivadotblog commented Mar 9, 2023 •

edited

sivadotblog commented Mar 9, 2023 •

edited

sivadotblog commented Mar 9, 2023 •

edited

jordandakota commented May 9, 2023 •

edited

sivadotblog commented Sep 4, 2023 •

edited