[ISSUE] Changing Pinned status causes cluster restart

[alexott]

The change in the pinned status for existing cluster causes the cluster restart because /clusters/edit REST API is not idempotent. We need to implement more sophisticated state comparison, and call /clusters/edit only if actual cluster definition has changed. The cluster restart may also happen if we just add library, etc.

Terraform Version

Terraform v0.13.4, Databricks provider is built from Git today

Affected Resource(s)

• databricks_cluster

Environment variable names

Terraform Configuration Files

Initial:

resource "databricks_cluster" "test_libs" {
  cluster_name            = "Test Maven"
  spark_version           = "7.3.x-scala2.12"
  node_type_id            = "Standard_D3_v2"
  autotermination_minutes = 20

  autoscale {
    min_workers = 1
    max_workers = 2
  }
}

modified:

resource "databricks_cluster" "test_libs" {
  cluster_name            = "Test Maven"
  spark_version           = "7.3.x-scala2.12"
  node_type_id            = "Standard_D3_v2"
  autotermination_minutes = 20
  is_pinned               = true

  autoscale {
    min_workers = 1
    max_workers = 2
  }
}

Debug Output

Debug output: https://gist.github.com/alexott/77d1e43786d6b9981299b6682d0dcfd3

Expected Behavior

Cluster shouldn't be restarted, only pinned status change happens

Actual Behavior

Cluster is restarted, and pinned status changes

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. change the pinned status of the existing cluster
2. execute terraform apply

[alexott]

I has a fix for that detects when the changes are made in the is_pinned or in library*, and don't call /clusters/edit... But it will have code conflicts with #384, so I'll wait until it's merged