Skip to content

Document effect of state-store cleanup for operators with durable execution - #70721

Merged
potiuk merged 1 commit into
apache:mainfrom
astronomer:warn-cleanup-risk-for-resumablemixin-providers
Jul 30, 2026
Merged

Document effect of state-store cleanup for operators with durable execution#70721
potiuk merged 1 commit into
apache:mainfrom
astronomer:warn-cleanup-risk-for-resumablemixin-providers

Conversation

@amoghrajesh

@amoghrajesh amoghrajesh commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Companion of #70718 but for providers.

Covers RedshiftDataOperator, BigQueryInsertJobOperator, DatabricksRunNowOperator, DatabricksSubmitRunOperator, and SnowflakeSqlApiOperator -- every operator built on ResumableJobMixin that doesn't pass an explicit retention override.


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added area:providers kind:documentation provider:amazon AWS/Amazon - related issues provider:databricks provider:google Google (including GCP) related issues provider:snowflake Issues related to Snowflake provider labels Jul 30, 2026
@amoghrajesh amoghrajesh changed the title Document effect of state-store cleanup for operators with durable exe… Document effect of state-store cleanup for operators with durable execution Jul 30, 2026
@amoghrajesh amoghrajesh self-assigned this Jul 30, 2026
@amoghrajesh
amoghrajesh requested a review from eladkal July 30, 2026 07:53
@potiuk potiuk added the backport-to-v3-3-test Backport to v3-3-test label Jul 30, 2026
@potiuk potiuk added this to the Airflow 3.3.1 milestone Jul 30, 2026
@potiuk
potiuk merged commit f5f4313 into apache:main Jul 30, 2026
69 of 70 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-3-test Commit Link

You can attempt to backport this manually by running:

cherry_picker f5f4313 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

@vatsrahul1001

vatsrahul1001 commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Going to move this to the 3.4.0 milestone rather than backporting it to v3-3-test.

These paragraphs append into the "Durable execution" sections of the Redshift Data / Databricks / BigQuery / Snowflake operator docs, but none of those sections exist on v3-3-test — the durable= parameter on these operators only landed on main, not in the 3.3.x providers. So the cleanup caveats have nothing to attach to on the release branch (the cherry-pick conflicts for the same reason).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers backport-to-v3-3-test Backport to v3-3-test kind:documentation provider:amazon AWS/Amazon - related issues provider:databricks provider:google Google (including GCP) related issues provider:snowflake Issues related to Snowflake provider

Projects

Development

Successfully merging this pull request may close these issues.

3 participants