Skip to content

feat(cli): add airflow dags clear for partition-range reprocessing#66004

Open
Lee-W wants to merge 2 commits intoapache:mainfrom
astronomer:dags-clear-partition-range
Open

feat(cli): add airflow dags clear for partition-range reprocessing#66004
Lee-W wants to merge 2 commits intoapache:mainfrom
astronomer:dags-clear-partition-range

Conversation

@Lee-W
Copy link
Copy Markdown
Member

@Lee-W Lee-W commented Apr 28, 2026

Why

airflow tasks clear --start-date / --end-date filters by logical_date, so it cannot reach DagRuns whose partition_date diverges from their logical_date (rollup or offset partitions). Today the only way to bulk-reprocess such partitioned runs is N individual API calls or N clicks in the UI.

What

  • New airflow dags clear <dag_id> subcommand. Filters DagRuns by DagRun.partition_date against an inclusive [--partition-start, --partition-end] window.
  • At least one bound must be provided; an inverted window is rejected. Runs with NULL partition_date are never matched.
  • Reuses the existing Dag.clear() engine per matched run_id, so the resulting state transitions (TIs reset, DagRun re-queued) match every other clear path.
  • Supports --yes, --only-failed, --only-running, --verbose. No --start-date / --end-date, task-regex, or upstream/downstream — those remain on tasks clear for TI-level workflows.

closes: #65921


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Opus 4.7 (1M context)

Generated-by: Claude Opus 4.7 (1M context) following the
guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@Lee-W Lee-W force-pushed the dags-clear-partition-range branch from ca20676 to fcdbfc5 Compare April 28, 2026 09:25
@Lee-W Lee-W marked this pull request as ready for review April 28, 2026 09:27
@uranusjr
Copy link
Copy Markdown
Member

I wonder what exactly we are looking for when we clear in this context. Do we want to make the old dag run disappear (basically what this does), or do we simply want to mark the partitions as unfulfilled? If the latter, I wonder if it’d be better to have airflow partitions clear instead.

Implementation-wise, I also wonder if it’d make sense to disassociate a dag run with a partition (to “clear” the partition), instead of clearing the run directly.

cc @cmarteepants

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

airflow dags clear: add --partition-start / --partition-end range flags

2 participants