Skip to content

Remove all_map_indices from task_state_store.clear() in context#68880

Merged
amoghrajesh merged 2 commits into
apache:mainfrom
astronomer:aip-103-nuke-clear-on-success-on-context
Jun 23, 2026
Merged

Remove all_map_indices from task_state_store.clear() in context#68880
amoghrajesh merged 2 commits into
apache:mainfrom
astronomer:aip-103-nuke-clear-on-success-on-context

Conversation

@amoghrajesh

@amoghrajesh amoghrajesh commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Was generative AI tooling used to co-author this PR?
  • Yes - claude sonnet 4.6

What

task_state_store.clear(all_map_indices=True) was exposed on the context object available inside running tasks. Mapped tasks run concurrently, so any sibling that writes after the clear survives it and the result is unpredictable and impossible to reason about if you go down a debugging path as to why it's not cleared :). There is no practical use case for this inside a running task: the only candidate is of a designated "leader" index doing post-fleet cleanup which would require polling sibling state, which is not available on the context object.

Current behaviour

A task could call task_state_store.clear(all_map_indices=True) to attempt a fleet-wide wipe of state for all map indices of the task. Due to concurrent execution, the outcome could have a race condition and unreliable.

Proposed change

Remove all_map_indices from TaskStateStore.clear() in the task context, and also remove out the entire plumbing: the field on ClearTaskStateStore comms message, the supervisor forwarding, the client query param, and the execution API route query param.

The flag is preserved at the backend and core API layer - external callers (UI, CLI) acting on a completed task group can still wipe all indices cleanly via the core API.

Alternative

If you need to clear state across all map indices of a mapped task group, do it from a downstream task using the CLI / core API which supports it.

Changes of Note

The removal touches the full stack from context to wire: context.py, comms.py, supervisor.py, client.py, and the execution API route. The Core API route and MetastoreStateBackend are intentionally untouched.

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Comment thread shared/state/src/airflow_shared/state/__init__.py Outdated
@amoghrajesh amoghrajesh requested a review from Lee-W June 23, 2026 06:23
@amoghrajesh

Copy link
Copy Markdown
Contributor Author

Failures are unrelated, merging.

@amoghrajesh amoghrajesh merged commit aa28f27 into apache:main Jun 23, 2026
99 of 100 checks passed
@amoghrajesh amoghrajesh deleted the aip-103-nuke-clear-on-success-on-context branch June 23, 2026 08:57
@github-project-automation github-project-automation Bot moved this from In progress to Done in AIP-103: Task State Management Jun 23, 2026
cetingokhan pushed a commit to cetingokhan/airflow that referenced this pull request Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

2 participants