Skip to content

direct: job with a table_update trigger never converges — trigger.table_update.condition is input-only and not registered as such #6315

Description

@ajheunis

Summary

On the direct deployment engine, a job with a table_update trigger that sets
condition never converges. Every bundle plan after a successful
bundle deploy reports 1 to change, forever.

The Jobs API accepts trigger.table_update.condition on create/update but never
returns it from GET /api/2.2/jobs/get. condition is therefore an input-only
field, and it is not registered as one, so the direct engine diffs it against a
missing remote value and plans an update on every run.

This is the same class of bug as #6076 (allow_duplicate_names on pipelines)
and the webhook_notifications.on_* phantom diff fixed in #6060. It is not
fixed in v1.11.0.

CLI version

v1.10.0 (also reproduces the same way before a resource rename, so it is not
name-related).

Repro

resources:
  jobs:
    render_files:
      name: render_files
      trigger:
        table_update:
          table_names:
            - main.default.some_table
          condition: ANY_UPDATED
          wait_after_last_change_seconds: 60
      tasks:
        - task_key: t
          notebook_task:
            notebook_path: ../src/nb.ipynb
          environment_key: default
      environments:
        - environment_key: default
          spec:
            environment_version: "4"
databricks bundle deploy -t dev   # succeeds
databricks bundle plan -t dev     # Plan: 0 to add, 1 to change, 0 to delete
databricks bundle deploy -t dev   # succeeds
databricks bundle plan -t dev     # Plan: 0 to add, 1 to change, 0 to delete

Evidence

bundle plan -t dev --output json, plan["resources.jobs.render_files"]:

"changes": {
  "trigger.table_update.condition": {
    "action": "update",
    "old": "ANY_UPDATED",
    "new": "ANY_UPDATED"
  }
}

old and new are identical and the action is still update. old comes from
the bundle's own saved state (state/resources.json), which records:

"trigger": {
  "pause_status": "PAUSED",
  "table_update": {
    "condition": "ANY_UPDATED",
    "table_names": ["<catalog>.<schema>.matched_files"],
    "wait_after_last_change_seconds": 60
  }
}

while remote_state in the same plan — and the raw
GET /api/2.2/jobs/get?job_id=... — has no condition at all:

"trigger": {
  "pause_status": "PAUSED",
  "table_update": {
    "table_names": ["<catalog>.<schema>.matched_files"],
    "wait_after_last_change_seconds": 60
  }
}

Two sibling jobs in the same bundle with file_arrival triggers, and a pipeline,
all report as unchanged; only the job with table_update + condition drifts.

Expected

trigger.table_update.condition should be treated as an input-only field so the
plan converges, the same way #6076 handled allow_duplicate_names.

Workaround

Omit condition from the bundle config. Where a trigger names a single table,
ANY_UPDATED and ALL_UPDATED are equivalent, so this costs nothing there — but
it is not an option for a multi-table trigger that needs ALL_UPDATED.

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugSomething isn't workingDABsDABs related issuesengine/directSpecific to direct deployment engine in Databricks Asset Bundles

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions