direct: permanent unconvergeable update on cluster fields supplied by a cluster policy (custom_tags, runtime_engine, spark_version)

[hasanakhuy]

Describe the issue

On the direct engine, a job whose job_clusters[].new_cluster relies on a cluster policy to supply cluster settings reports a permanent, unconvergeable update. Every bundle deploy plans an update, and the next one plans the same update again.

The plan's own changes output shows the CLI already classifies most of this correctly. Of 16 entries, 13 are correctly skipped with reason: backend_default, empty or managed. Only three are reported as update — and all three are values the cluster policy fills in server-side, not backend defaults in the usual sense:

field	remote value	in bundle config?
new_cluster.custom_tags	{"<tag>": "<policy-name>"}	no — set by the policy
new_cluster.runtime_engine	STANDARD	no — set by the policy
new_cluster.spark_version	17.3.x-scala2.13	no — set by the policy

This looks like the same class of problem already fixed for other fields, e.g. in 1.14.0:

bundle plan no longer reports a permanent, unconvergeable update on securable_kind … The field is computed by the backend (output-only), so it is now ignored during drift detection.

Policy-supplied cluster fields appear not to be covered yet.

Two other data products in the same bundle converge correctly and act as a control:

compute	2nd deploy, no source change
serverless	unchanged ✅
own complete new_cluster block	unchanged ✅
new_cluster relying on a policy	update ❌

Configuration

A complex variable holds the cluster shape. It declares only policy_id and autoscale; everything else is expected to come from the policy:

variables:
  policy_s:
    lookup:
      cluster_policy: <policy-name>

  cluster_s:
    type: complex
    default:
      policy_id: ${var.policy_s}
      autoscale:
        min_workers: 1
        max_workers: 2

The data product references it:

resources:
  jobs:
    probe:
      name: probe
      job_clusters:
        - job_cluster_key: small
          new_cluster: ${var.cluster_s}
      tasks:
        - task_key: probe
          job_cluster_key: small
          notebook_task:
            notebook_path: ../src/probe

The policy sets spark_version, runtime_engine and a custom_tags entry as policy values. Declaring them in the bundle as well is not desired here — the policy is meant to be the single source for them.

Steps to reproduce the behavior

1. Create a cluster policy that supplies spark_version, runtime_engine and a custom_tags entry.
2. databricks bundle deploy -t <target> — resource is created.
3. Change nothing.
4. databricks bundle deploy -t <target> again — plans 1 changed instead of unchanged.
5. Repeat: it never converges.

Expected Behavior

The second deploy plans no change. Fields a cluster policy fills in server-side are not treated as drift, the same way backend defaults already are.

Actual Behavior

Every deploy plans 1 changed on the policy-managed job, forever.

OS and CLI version

CLI v1.13.0, Linux (self-hosted GitHub Actions runner), engine: direct.

Is this a regression?

Unknown — this configuration has only ever run on the direct engine with 1.13.0.

Detailed plan

Redacted bundle plan -o json, changes only. Note the three update entries versus the thirteen correctly skipped ones:

"changes": {
  "email_notifications":                                                   { "action": "skip", "reason": "empty",            "remote": {} },
  "timeout_seconds":                                                       { "action": "skip", "reason": "empty",            "remote": 0 },
  "webhook_notifications":                                                 { "action": "skip", "reason": "empty",            "remote": {} },
  "tasks[task_key='probe'].disabled":                                      { "action": "skip", "reason": "empty",            "remote": false },
  "tasks[task_key='probe'].email_notifications":                           { "action": "skip", "reason": "empty",            "remote": {} },
  "tasks[task_key='probe'].timeout_seconds":                               { "action": "skip", "reason": "empty",            "remote": 0 },
  "tasks[task_key='probe'].notebook_task.source":                          { "action": "skip", "reason": "backend_default",  "remote": "WORKSPACE" },
  "tasks[task_key='probe'].run_if":                                        { "action": "skip", "reason": "backend_default",  "remote": "ALL_SUCCESS" },
  "job_clusters[job_cluster_key='small'].new_cluster.azure_attributes":    { "action": "skip", "reason": "managed",          "remote": { "availability": "ON_DEMAND_AZURE" } },
  "job_clusters[job_cluster_key='small'].new_cluster.data_security_mode":  { "action": "skip", "reason": "backend_default",  "remote": "SINGLE_USER" },
  "job_clusters[job_cluster_key='small'].new_cluster.driver_node_type_id": { "action": "skip", "reason": "backend_default",  "remote": "<node-type>" },
  "job_clusters[job_cluster_key='small'].new_cluster.enable_elastic_disk": { "action": "skip", "reason": "backend_default",  "remote": true },
  "job_clusters[job_cluster_key='small'].new_cluster.node_type_id":        { "action": "skip", "reason": "backend_default",  "remote": "<node-type>" },

  "job_clusters[job_cluster_key='small'].new_cluster.custom_tags":         { "action": "update", "remote": { "<tag>": "<policy-name>" } },
  "job_clusters[job_cluster_key='small'].new_cluster.runtime_engine":      { "action": "update", "remote": "STANDARD" },
  "job_clusters[job_cluster_key='small'].new_cluster.spark_version":       { "action": "update", "remote": "17.3.x-scala2.13" }
}

new_state for that cluster contains only what the bundle declares:

"new_cluster": {
  "autoscale": { "max_workers": 2, "min_workers": 1 },
  "policy_id": "<redacted>"
}

while remote_state additionally carries the policy-supplied custom_tags, runtime_engine, spark_version, plus the fields that are correctly skipped.

Impact

For job clusters the practical cost is low — the cluster is created per run, so the repeated update only rewrites the job definition. The real cost is that bundle plan stops being usable as a review artifact: in a change-approval flow, a reviewer reading the plan before approving a production deployment cannot tell real changes from this constant noise. It also affects every data product that uses a shared, policy-based cluster definition, which is the pattern we would like to standardise on.

Related in effect but different in cause: #4933 (release_version, closed as not planned).

[denik]

Thanks for the repo, it's a known issue, see my comment there on who we plan to tackle it #5179 (comment)

[denik]

one question @hasanakhuy in your example config you don't set apply_policy_default_values to true. Is that intentional? My understand that you need that set in order for cluster policy to modify your cluster.

Update: Nevermind, actually it seems that "fixed" policy not just enforces the value but also sets it and it does it regardless of apply_policy_default_values.