Do not report cluster fields supplied by a cluster policy as drift

.nextchanges/bundles/cluster-policy-no-drift.md

@@ -0,0 +1 @@
+* `bundle plan` no longer reports a permanent update on cluster fields supplied by a cluster policy; a field the config never declares is no longer drift when the cluster spec has a `policy_id`. ([#6531](https://github.com/databricks/cli/pull/6531))

acceptance/bundle/resources/cluster_policies/fixed_value_conflict/databricks.yml.tmpl

@@ -0,0 +1,36 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+workspace:
+  root_path: ~/.bundle/$UNIQUE_NAME
+
+resources:
+  cluster_policies:
+    pol:
+      name: test-policy-$UNIQUE_NAME
+      definition: |-
+        {
+          "custom_tags.CostCenter": {
+            "type": "fixed",
+            "value": "from-fixed"
+          }
+        }
+
+  jobs:
+    # The config sets the fixed attribute to a value the policy forbids.
+    j:
+      name: test-job-$UNIQUE_NAME
+      job_clusters:
+        - job_cluster_key: small
+          new_cluster:
+            policy_id: ${resources.cluster_policies.pol.id}
+            spark_version: $DEFAULT_SPARK_VERSION
+            node_type_id: $NODE_TYPE_ID
+            num_workers: 1
+            custom_tags:
+              CostCenter: not-what-the-policy-says
+      tasks:
+        - task_key: t
+          job_cluster_key: small
+          spark_python_task:
+            python_file: ./hello_world.py

acceptance/bundle/resources/cluster_policies/fixed_value_conflict/hello_world.py

@@ -0,0 +1 @@
+print("hello")

acceptance/bundle/resources/cluster_policies/fixed_value_conflict/output.txt

@@ -0,0 +1,23 @@
+
+=== Deploy a cluster whose config contradicts a fixed policy value
+
+>>> [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files...
+Error: cannot create resources.jobs.j: Cluster validation error: Validation failed for custom_tags, CostCenter must be from-fixed (is "not-what-the-policy-says") (400 INVALID_PARAMETER_VALUE)
+
+Endpoint: POST [DATABRICKS_URL]/api/2.2/jobs/create
+HTTP Status: 400 Bad Request
+API error_code: INVALID_PARAMETER_VALUE
+API message: Cluster validation error: Validation failed for custom_tags, CostCenter must be from-fixed (is "not-what-the-policy-says")
+
+Files: 6 uploaded, 0 deleted
+
+Exit code: 1
+
+>>> [CLI] bundle destroy --auto-approve
+The following resources will be deleted:
+  delete resources.cluster_policies.pol
+
+All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]
+
+Destroy: 1 deleted

acceptance/bundle/resources/cluster_policies/fixed_value_conflict/script

@@ -0,0 +1,9 @@
+envsubst < databricks.yml.tmpl > databricks.yml
+
+cleanup() {
+    trace $CLI bundle destroy --auto-approve
+}
+trap cleanup EXIT
+
+title "Deploy a cluster whose config contradicts a fixed policy value\n"
+errcode trace $CLI bundle deploy

acceptance/bundle/resources/cluster_policies/fixed_value_conflict/test.toml

@@ -0,0 +1,5 @@
+RecordRequests = false
+
+Ignore = [
+    "databricks.yml",
+]

acceptance/bundle/resources/cluster_policies/fixed_values_applied/databricks.yml.tmpl

@@ -0,0 +1,38 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+workspace:
+  root_path: ~/.bundle/$UNIQUE_NAME
+
+resources:
+  cluster_policies:
+    pol:
+      name: test-policy-$UNIQUE_NAME
+      definition: |-
+        {
+          "spark_version": {
+            "type": "fixed",
+            "value": "$DEFAULT_SPARK_VERSION"
+          },
+          "custom_tags.CostCenter": {
+            "type": "fixed",
+            "value": "policy-supplied"
+          }
+        }
+
+  jobs:
+    # new_cluster deliberately omits spark_version and custom_tags: the policy is
+    # expected to supply both. apply_policy_default_values is NOT set.
+    j:
+      name: test-job-$UNIQUE_NAME
+      job_clusters:
+        - job_cluster_key: small
+          new_cluster:
+            policy_id: ${resources.cluster_policies.pol.id}
+            node_type_id: $NODE_TYPE_ID
+            num_workers: 1
+      tasks:
+        - task_key: t
+          job_cluster_key: small
+          spark_python_task:
+            python_file: ./hello_world.py

acceptance/bundle/resources/cluster_policies/fixed_values_applied/hello_world.py

@@ -0,0 +1 @@
+print("hello")

acceptance/bundle/resources/cluster_policies/fixed_values_applied/output.txt

@@ -0,0 +1,50 @@
+
+>>> [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files...
+Created cluster_policies.pol
+Created jobs.j
+Files: 6 uploaded, 0 deleted
+Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
+
+=== Did the fixed policy SUPPLY spark_version and custom_tags, or only validate them?
+The bundle declares neither, and does not set apply_policy_default_values.
+{
+  "spark_version": "13.3.x-snapshot-scala2.12",
+  "custom_tags": {
+    "CostCenter": "policy-supplied"
+  }
+}
+
+=== Is the policy-supplied value reported as drift?
+
+>>> [CLI] bundle plan
+Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
+
+=== spark_version is not covered by any backend_defaults rule, so show its verdict
+
+>>> [CLI] bundle plan -o json
+[
+  {
+    "job_clusters[job_cluster_key='small'].new_cluster.custom_tags": {
+      "action": "skip",
+      "reason": "policy_managed",
+      "remote": {
+        "CostCenter": "policy-supplied"
+      }
+    },
+    "job_clusters[job_cluster_key='small'].new_cluster.spark_version": {
+      "action": "skip",
+      "reason": "policy_managed",
+      "remote": "13.3.x-snapshot-scala2.12"
+    }
+  }
+]
+
+>>> [CLI] bundle destroy --auto-approve
+The following resources will be deleted:
+  delete resources.cluster_policies.pol
+  delete resources.jobs.j
+
+All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]
+
+Destroy: 2 deleted

acceptance/bundle/resources/cluster_policies/fixed_values_applied/script

@@ -0,0 +1,26 @@
+envsubst < databricks.yml.tmpl > databricks.yml
+
+cleanup() {
+    trace $CLI bundle destroy --auto-approve
+}
+trap cleanup EXIT
+
+trace $CLI bundle deploy
+
+# Mask the server-assigned policy id (real backend returns a hex id the built-in
+# UUID replacement misses).
+read_id.py pol > /dev/null
+job_id=$(read_id.py j)
+
+title "Did the fixed policy SUPPLY spark_version and custom_tags, or only validate them?\n"
+echo "The bundle declares neither, and does not set apply_policy_default_values."
+$CLI jobs get "$job_id" | jq '.settings.job_clusters[0].new_cluster | {spark_version, custom_tags}'
+
+title "Is the policy-supplied value reported as drift?\n"
+trace $CLI bundle plan
+
+title "spark_version is not covered by any backend_defaults rule, so show its verdict\n"
+trace $CLI bundle plan -o json | jq -S '
+  [.plan | to_entries[] | (.value.changes // {})
+   | with_entries(select(.key | endswith("spark_version") or endswith("custom_tags")))]
+  | map(select(length > 0))'

acceptance/bundle/resources/cluster_policies/fixed_values_applied/test.toml

@@ -0,0 +1,5 @@
+RecordRequests = false
+
+Ignore = [
+    "databricks.yml",
+]

acceptance/bundle/resources/cluster_policies/policy_drift/databricks.yml.tmpl

@@ -0,0 +1,68 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+workspace:
+  root_path: ~/.bundle/$UNIQUE_NAME
+
+resources:
+  cluster_policies:
+    pol:
+      name: test-policy-$UNIQUE_NAME
+      definition: |-
+        {
+          "custom_tags.CostCenter": {
+            "type": "fixed",
+            "value": "from-policy"
+          }
+        }
+
+  jobs:
+    # policy attached: the tag the policy adds must not be drift.
+    with_policy:
+      name: test-with-policy-$UNIQUE_NAME
+      job_clusters:
+        - job_cluster_key: small
+          new_cluster:
+            policy_id: ${resources.cluster_policies.pol.id}
+            spark_version: $DEFAULT_SPARK_VERSION
+            node_type_id: $NODE_TYPE_ID
+            num_workers: 1
+      tasks:
+        - task_key: t
+          job_cluster_key: small
+          spark_python_task:
+            python_file: ./hello_world.py
+
+    # policy attached, and the config owns a tag of its own. An out-of-band change to that
+    # tag must still be reported: a policy suppresses additions, never disagreements.
+    owned_tag:
+      name: test-owned-tag-$UNIQUE_NAME
+      job_clusters:
+        - job_cluster_key: small
+          new_cluster:
+            policy_id: ${resources.cluster_policies.pol.id}
+            spark_version: $DEFAULT_SPARK_VERSION
+            node_type_id: $NODE_TYPE_ID
+            num_workers: 1
+            custom_tags:
+              Mine: mine
+      tasks:
+        - task_key: t
+          job_cluster_key: small
+          spark_python_task:
+            python_file: ./hello_world.py
+
+    # no policy attached: a remote-only tag is still drift. edit_resource.py injects it below.
+    no_policy:
+      name: test-no-policy-$UNIQUE_NAME
+      job_clusters:
+        - job_cluster_key: small
+          new_cluster:
+            spark_version: $DEFAULT_SPARK_VERSION
+            node_type_id: $NODE_TYPE_ID
+            num_workers: 1
+      tasks:
+        - task_key: t
+          job_cluster_key: small
+          spark_python_task:
+            python_file: ./hello_world.py

acceptance/bundle/resources/cluster_policies/policy_drift/hello_world.py

@@ -0,0 +1 @@
+print("hello")

acceptance/bundle/resources/cluster_policies/policy_drift/output.txt

@@ -0,0 +1,64 @@
+
+>>> [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files...
+Created cluster_policies.pol
+Created jobs.no_policy
+Created jobs.owned_tag
+Created jobs.with_policy
+Files: 6 uploaded, 0 deleted
+Resources: 4 created, 0 changed, 0 deleted, 0 unchanged
+
+=== A tag the cluster policy added is not drift
+
+>>> [CLI] bundle plan
+Plan: 0 to add, 0 to change, 0 to delete, 4 unchanged
+
+=== Without a policy_id the same remote-only tag IS drift
+
+>>> [CLI] bundle plan
+update jobs.no_policy
+
+Plan: 0 to add, 1 to change, 0 to delete, 3 unchanged
+
+=== A policy does not mask a change to a tag the config owns
+
+>>> [CLI] bundle plan
+update jobs.no_policy
+update jobs.owned_tag
+
+Plan: 0 to add, 2 to change, 0 to delete, 2 unchanged
+
+=== Removing a tag from config is a change the user asked for, not an addition
+
+>>> update_file.py databricks.yml Mine: mine Other: other
+
+>>> [CLI] bundle plan -o json
+[
+  {
+    "job_clusters[job_cluster_key='small'].new_cluster.custom_tags['CostCenter']": {
+      "action": "skip",
+      "reason": "policy_managed",
+      "remote": "from-policy"
+    },
+    "job_clusters[job_cluster_key='small'].new_cluster.custom_tags['Mine']": {
+      "action": "update",
+      "old": "mine",
+      "remote": "changed-out-of-band"
+    },
+    "job_clusters[job_cluster_key='small'].new_cluster.custom_tags['Other']": {
+      "action": "update",
+      "new": "other"
+    }
+  }
+]
+
+>>> [CLI] bundle destroy --auto-approve
+The following resources will be deleted:
+  delete resources.cluster_policies.pol
+  delete resources.jobs.no_policy
+  delete resources.jobs.owned_tag
+  delete resources.jobs.with_policy
+
+All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]
+
+Destroy: 4 deleted

acceptance/bundle/resources/cluster_policies/policy_drift/script

@@ -0,0 +1,35 @@
+envsubst < databricks.yml.tmpl > databricks.yml
+
+cleanup() {
+    trace $CLI bundle destroy --auto-approve
+}
+trap cleanup EXIT
+
+trace $CLI bundle deploy
+read_id.py pol > /dev/null
+no_policy_id="$(read_id.py no_policy)"
+owned_tag_id="$(read_id.py owned_tag)"
+
+title "A tag the cluster policy added is not drift\n"
+trace $CLI bundle plan | contains.py "0 to change"
+
+title "Without a policy_id the same remote-only tag IS drift\n"
+edit_resource.py jobs "$no_policy_id" <<'EOF'
+for jc in r["job_clusters"]:
+    jc["new_cluster"]["custom_tags"] = {"CostCenter": "added-out-of-band"}
+EOF
+trace $CLI bundle plan | contains.py "1 to change"
+
+title "A policy does not mask a change to a tag the config owns\n"
+edit_resource.py jobs "$owned_tag_id" <<'EOF'
+for jc in r["job_clusters"]:
+    jc["new_cluster"]["custom_tags"]["Mine"] = "changed-out-of-band"
+EOF
+trace $CLI bundle plan | contains.py "2 to change"
+
+title "Removing a tag from config is a change the user asked for, not an addition\n"
+trace update_file.py databricks.yml "Mine: mine" "Other: other"
+trace $CLI bundle plan -o json | jq -S '
+  [.plan | to_entries[]
+   | select(.key | endswith("owned_tag"))
+   | .value.changes | with_entries(select(.key | contains("custom_tags")))]'

acceptance/bundle/resources/cluster_policies/policy_drift/test.toml

@@ -0,0 +1,5 @@
+RecordRequests = false
+
+Ignore = [
+    "databricks.yml",
+]