Skip to content

Optimize scheduler backfill completion updates - #71216

Open
fat-catTW wants to merge 1 commit into
apache:mainfrom
fat-catTW:fix-backfill-completion-update
Open

Optimize scheduler backfill completion updates#71216
fat-catTW wants to merge 1 commit into
apache:mainfrom
fat-catTW:fix-backfill-completion-update

Conversation

@fat-catTW

Copy link
Copy Markdown
Contributor

Why:

The scheduler checks whether backfills are complete as part of its regular loop. The previous implementation loaded every matching Backfill row into the ORM session and then updated each object in Python, even though the completion condition can be expressed entirely in SQL.

Using a bulk update keeps this scheduler path lighter while preserving the existing AIP-78 backfill lifecycle guards.

Solution:

Update _mark_backfills_complete() to issue a single bulk UPDATE for backfills that are ready to complete.

The existing guards are preserved:

  • initializing backfills without associations are skipped
  • orphaned backfills older than the cutoff can still be completed
  • backfills with queued or running Dag runs stay active

Add a regression test that verifies the completion path uses a single update statement.

Was generative AI tooling used to co-author this PR?
  • [X] Yes (please specify the tool below)

Generated-by: [Codex] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@fat-catTW
fat-catTW requested review from XD-DENG and ashb as code owners August 6, 2026 07:18
@boring-cyborg boring-cyborg Bot added the area:Scheduler including HA (high availability) scheduler label Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant