Fix use_job_schedule=False disabling asset-triggered Dag runs#67764
Fix use_job_schedule=False disabling asset-triggered Dag runs#67764Vamsi-klu wants to merge 2 commits into
Conversation
3bd15a8 to
ff8ce38
Compare
|
Gentle ping for SME eyes when you have a moment, @Lee-W @uranusjr 🙏 Small scheduler fix for #62929: @Lee-W — you reviewed the earlier (now closed-for-inactivity) #62931 and called the approach "a valid one", so flagging this here. Includes three regression tests (asset, partition, and a flag-on companion) plus a |
| @@ -0,0 +1 @@ | |||
| Fix ``[scheduler] use_job_schedule=False`` wrongly disabling asset- and partitioned-asset-triggered Dag run creation. The flag now disables only time-based (cron) scheduling, as documented; manually triggered and asset-triggered Dag runs are still created. | |||
There was a problem hiding this comment.
We do not add new fragments for bug fixes in this format, please revert or remove them.
[scheduler] use_job_schedule is documented to turn off only cron/time-based scheduling, but it gated the entire _create_dagruns_for_dags call, which also creates asset- and partitioned-asset-triggered runs. As a result, setting the flag to False silently stopped all asset-driven scheduling. Move the flag check so it gates only time-based run creation; asset- and partition-triggered runs are now always created, matching the documented behavior. Also update the "Disable the scheduler" maintenance docs, which previously claimed this flag stops all new Dag runs. closes: apache#62929
ff8ce38 to
31f3c71
Compare
What / impact
Setting
[scheduler] use_job_schedule=Falsesilently stopped all asset- and partitioned-asset-triggered Dag runs — not just cron/time-based runs. Event- and data-driven pipelines that rely on asset scheduling would stop firing with no error, even though the flag is documented to turn off only cron interval scheduling. After this change the flag disables only time-based (cron) scheduling; asset-, partitioned-asset-, and manually-triggered runs are still created.Why
use_job_schedulegated the entire_create_dagruns_for_dagscall in_do_scheduling, but that method creates three kinds of runs: partitioned-asset (top of the method), time-based, and asset-triggered. Only the time-based path should be governed by the flag. This moves the gate so it wraps only the time-based_create_dag_runs(non_asset_dags, …)call; the orchestrator — and therefore the asset/partition paths — always runs. The "Disable the scheduler" maintenance docs, which previously claimed this flag stops all new Dag runs, are updated to match.How it unblocks
Restores the documented behavior for deployments that turn off cron scheduling while still relying on asset-driven scheduling. Supersedes #62931, which was closed for author inactivity (not rejection) — the approach was never disputed and member Lee-W called it "a valid one".
Tests
Three new regression tests in
test_scheduler_job.py, each verified to fail on the unfixed code: asset-triggered runs created with the flag off, partitioned-asset runs created with the flag off, and a flag-on companion proving the time-based gate still suppresses cron runs. Full scheduler suite and the exact-query-count tests pass; mypy + pre-commit + manual static checks are green.closes: #62929
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines