Skip to content

Don't re-emit logical_date when previous data_interval is zero-length#66132

Merged
uranusjr merged 3 commits into
apache:mainfrom
astronomer:fix-skip-to-latest
May 2, 2026
Merged

Don't re-emit logical_date when previous data_interval is zero-length#66132
uranusjr merged 3 commits into
apache:mainfrom
astronomer:fix-skip-to-latest

Conversation

@jedcunningham
Copy link
Copy Markdown
Member

What's the bug

_DataIntervalTimetable.next_dagrun_info aligns the next run's start onto data_interval.end of the previous run. When that previous interval is zero-length (start == end), start lands on the same point as the existing run's logical_date, so the scheduler tries to create a run that collides with the existing one and logs:

run already exists; skipping dagrun creation

It then calls calculate_dagrun_date_fields(last_automated_run=existing_run) to advance — which feeds the same zero-length interval back in and recomputes the same value. The scheduler loops on the skip warning until the next cron boundary nudges _skip_to_latest to a different result, at which point the gate may have already rolled past the slot — so the intended run is silently skipped.

When does this happen?

Zero-length data_intervals are the normal output of CronTriggerTimetable (point-in-time) but never of CronDataIntervalTimetable (windowed, validated > 0). They appear on a CronDataIntervalTimetable DAG when its scheduling history was produced by CronTriggerTimetable and the DAG later switches to CronDataIntervalTimetable — for example, when [scheduler] create_cron_data_intervals is flipped from False (Airflow 3 default) to True after some runs have already been created.

The fix

In _DataIntervalTimetable.next_dagrun_info, after computing start, if the previous interval was zero-length and start falls on it, advance start by one period so the new run gets a fresh logical_date.

The guard only fires for _DataIntervalTimetable subclasses (CronDataIntervalTimetable, DeltaDataIntervalTimetable). CronTriggerTimetable / DeltaTriggerTimetable extend a separate _TriggerTimetable base with its own next_dagrun_info and never go through this code path, so their normal zero-length output is unaffected.

Tests

Added test_zero_length_last_interval_does_not_re_emit_logical_date parametrized on catchup ∈ {True, False} — both branches converge on the same expected interval, demonstrating the guard absorbs the difference between the two paths into the previous-run handling.

related: #59618

Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Anthropic)

Copy link
Copy Markdown
Member

@Lee-W Lee-W left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, looks good, but will need to take a look at the failed tests

jedcunningham and others added 2 commits April 30, 2026 09:55
…ngth

When a DAG is migrated from CronTriggerTimetable (point-in-time) to
CronDataIntervalTimetable, the latest run's data_interval has
``start == end``. ``_DataIntervalTimetable.next_dagrun_info`` would then
align ``start`` onto that same point, producing a new run whose
``logical_date`` collides with the existing one. The scheduler logged
"run already exists; skipping dagrun creation" and looped until the
next cron boundary, sometimes silently skipping a scheduled run.

Detect the zero-length previous interval and advance ``start`` by one
period so the next run gets a fresh ``logical_date``.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- test_bulk_write_to_db_max_active_runs and test_next_dagrun_after_fake_scheduled_previous:
  switch synthetic zero-length data_intervals to realistic non-zero ones and update
  expected values.
- test_schedule_tis_start_trigger: drop stale xfail (XPASS)
@Lee-W Lee-W force-pushed the fix-skip-to-latest branch from 9827701 to fe0103e Compare April 30, 2026 02:08
Comment thread airflow-core/tests/unit/models/test_dag.py
Comment thread airflow-core/src/airflow/timetables/interval.py Outdated
@jason810496 jason810496 self-requested a review May 1, 2026 15:45
@kaxil kaxil added this to the Airflow 3.2.2 milestone May 1, 2026
@kaxil kaxil added the backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch label May 1, 2026
Expand the inline comment in `_DataIntervalTimetable.next_dagrun_info`
to explain the CronTrigger → CronDataInterval transition that produces
a zero-length previous interval, and note in the
`create_cron_data_intervals` config description that switching from
CronTrigger to CronDataInterval skips one period past the most recent
CronTrigger run.
@uranusjr uranusjr merged commit 4a9b5a0 into apache:main May 2, 2026
79 checks passed
@uranusjr uranusjr deleted the fix-skip-to-latest branch May 2, 2026 10:35
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 2, 2026

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-2-test PR Link

potiuk pushed a commit that referenced this pull request May 2, 2026
… zero-length (#66132)

(cherry picked from commit 4a9b5a0)

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
potiuk pushed a commit that referenced this pull request May 2, 2026
… zero-length (#66132) (#66263)

(cherry picked from commit 4a9b5a0)

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@jedcunningham
Copy link
Copy Markdown
Member Author

And just for posterity, why formalizing this is better than blowing errors through the whole overlapping period? Well, if you have more dags in this scenario than max_dagruns_to_create_per_loop (which is likely if you've toggled create_cron_data_intervals), then you get no runs scheduled at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants