From 1dc622cddbd8db146ae9de489c3c7a7bc874391a Mon Sep 17 00:00:00 2001 From: Amogh Desai Date: Thu, 30 Jul 2026 13:21:39 +0530 Subject: [PATCH] Document effect of state-store cleanup for operators with durable execution --- providers/amazon/docs/operators/redshift/redshift_data.rst | 7 +++++++ providers/databricks/docs/operators/run_now.rst | 6 ++++++ providers/databricks/docs/operators/submit_run.rst | 6 ++++++ providers/google/docs/operators/cloud/bigquery.rst | 6 ++++++ providers/snowflake/docs/operators/snowflake.rst | 7 +++++++ 5 files changed, 32 insertions(+) diff --git a/providers/amazon/docs/operators/redshift/redshift_data.rst b/providers/amazon/docs/operators/redshift/redshift_data.rst index e48b09ce60dec..c9e64aeb66271 100644 --- a/providers/amazon/docs/operators/redshift/redshift_data.rst +++ b/providers/amazon/docs/operators/redshift/redshift_data.rst @@ -92,6 +92,13 @@ earlier Airflow versions the flag is a no-op and the operator always submits fre exactly as before. If the task state store is unavailable at runtime, the operator logs that crash recovery is disabled and behaves the same way. +Like the persisted state itself, the stored statement id isn't deleted automatically, that only +happens when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer +than ``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, +the statement id won't be there for the next retry, and the operator will submit the SQL fresh +instead of reconnecting. Avoid running cleanup on a schedule shorter than your longest +``retry_delay``. + To opt out and always submit fresh SQL on retry, set ``durable=False``: .. code-block:: python diff --git a/providers/databricks/docs/operators/run_now.rst b/providers/databricks/docs/operators/run_now.rst index 78e5a1788aac2..00ee6ed0fecfe 100644 --- a/providers/databricks/docs/operators/run_now.rst +++ b/providers/databricks/docs/operators/run_now.rst @@ -99,6 +99,12 @@ Airflow versions the flag is a no-op and the operator always triggers a fresh ru exactly as before. If the task state store is unavailable at runtime, the operator logs that crash recovery is disabled and behaves the same way. +Like the persisted state itself, the stored run id isn't deleted automatically, that only happens +when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than +``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the +run id won't be there for the next retry, and the operator will trigger a fresh run instead of +reconnecting. Avoid running cleanup on a schedule shorter than your longest ``retry_delay``. + To opt out and always trigger a fresh run on retry, set ``durable=False``: .. code-block:: python diff --git a/providers/databricks/docs/operators/submit_run.rst b/providers/databricks/docs/operators/submit_run.rst index 808f7534995d0..f3afd1046199f 100644 --- a/providers/databricks/docs/operators/submit_run.rst +++ b/providers/databricks/docs/operators/submit_run.rst @@ -184,6 +184,12 @@ earlier Airflow versions the flag is a no-op and the operator always submits a f run on retry, exactly as before. If the task state store is unavailable at runtime, the operator logs that crash recovery is disabled and behaves the same way. +Like the persisted state itself, the stored run id isn't deleted automatically, that only happens +when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than +``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the +run id won't be there for the next retry, and the operator will submit a fresh run instead of +reconnecting. Avoid running cleanup on a schedule shorter than your longest ``retry_delay``. + To opt out and always submit a fresh run on retry, set ``durable=False``: .. code-block:: python diff --git a/providers/google/docs/operators/cloud/bigquery.rst b/providers/google/docs/operators/cloud/bigquery.rst index 9f46ef351fac2..6f42b4e841aa4 100644 --- a/providers/google/docs/operators/cloud/bigquery.rst +++ b/providers/google/docs/operators/cloud/bigquery.rst @@ -408,6 +408,12 @@ exactly as before -- including the pre-existing ``reattach_states``/``Conflict`` unchanged. If the task state store is unavailable at runtime, the operator logs that crash recovery is disabled and behaves the same way. +Like the persisted state itself, the stored job id isn't deleted automatically, that only happens +when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than +``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the +job id won't be there for the next retry, and the operator will submit a fresh job instead of +reconnecting. Avoid running cleanup on a schedule shorter than your longest ``retry_delay``. + To opt out and always submit a fresh job on retry, set ``durable=False``: .. code-block:: python diff --git a/providers/snowflake/docs/operators/snowflake.rst b/providers/snowflake/docs/operators/snowflake.rst index 5bb2beb8daca0..6f12197cd12b7 100644 --- a/providers/snowflake/docs/operators/snowflake.rst +++ b/providers/snowflake/docs/operators/snowflake.rst @@ -182,6 +182,13 @@ earlier Airflow versions the flag is a no-op and the operator always submits fre exactly as before. If the task state store is unavailable at runtime, the operator logs that crash recovery is disabled and behaves the same way. +Like the persisted state itself, the stored statement handles aren't deleted automatically, that +only happens when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is +longer than ``[state_store] default_retention_days`` (30 days by default) and cleanup runs in +between, the handles won't be there for the next retry, and the operator will submit the SQL +fresh instead of reconnecting. Avoid running cleanup on a schedule shorter than your longest +``retry_delay``. + To opt out and always submit fresh SQL on retry, set ``durable=False``: .. code-block:: python