From 1fffb9af8db0b8a15511f8a1972f74de7955dd16 Mon Sep 17 00:00:00 2001 From: Amogh Desai Date: Tue, 14 Jul 2026 11:38:12 +0530 Subject: [PATCH] Link task state store docs in durable execution across providers --- providers/amazon/docs/operators/redshift/redshift_data.rst | 6 +++--- providers/apache/spark/docs/operators.rst | 6 +++--- providers/databricks/docs/operators/run_now.rst | 7 ++++--- providers/databricks/docs/operators/submit_run.rst | 7 ++++--- providers/google/docs/operators/cloud/bigquery.rst | 7 ++++--- providers/snowflake/docs/operators/snowflake.rst | 7 ++++--- 6 files changed, 22 insertions(+), 18 deletions(-) diff --git a/providers/amazon/docs/operators/redshift/redshift_data.rst b/providers/amazon/docs/operators/redshift/redshift_data.rst index 7720e304b76bc..e48b09ce60dec 100644 --- a/providers/amazon/docs/operators/redshift/redshift_data.rst +++ b/providers/amazon/docs/operators/redshift/redshift_data.rst @@ -71,9 +71,9 @@ Durable execution ``RedshiftDataOperator`` submits a statement and then polls it to completion on the worker. By default the operator runs in a *durable* mode that makes this crash-safe: the Redshift statement -id is persisted to task state before polling begins, so if the worker crashes or is preempted and -the task is retried, the operator reconnects to the statement that is already executing in -Redshift instead of resubmitting the SQL. +id is persisted to :doc:`task state store ` before +polling begins, so if the worker crashes or is preempted and the task is retried, the operator +reconnects to the statement that is already executing in Redshift instead of resubmitting the SQL. On retry the operator checks the prior statement's state: diff --git a/providers/apache/spark/docs/operators.rst b/providers/apache/spark/docs/operators.rst index d20c1da5cf7fe..9705e505b57e0 100644 --- a/providers/apache/spark/docs/operators.rst +++ b/providers/apache/spark/docs/operators.rst @@ -190,9 +190,9 @@ independently on the cluster. If the Airflow worker dies while the Spark job is Airflow loses track of it and the behaviour to submit a brand new job would be wasting the compute already done or even cause conflicts if the Spark job itself is not designed to be idempotent. -Now, the ``SparkSubmitOperator`` solves this by persisting the driver ID to ``task_state_store`` immediately after -submission. On retry, it reads the ID back and reconnects to the already-running driver instead of -resubmitting. +Now, the ``SparkSubmitOperator`` solves this by persisting the driver ID to :doc:`task state store +` immediately after submission. On retry, it reads +the ID back and reconnects to the already-running driver instead of resubmitting. This is the **synchronous path** — the worker holds a slot for the duration of polling. This is a crash-safety net for teams running sync operators for log observability, org constraints, or diff --git a/providers/databricks/docs/operators/run_now.rst b/providers/databricks/docs/operators/run_now.rst index 30f64139b9cbe..78e5a1788aac2 100644 --- a/providers/databricks/docs/operators/run_now.rst +++ b/providers/databricks/docs/operators/run_now.rst @@ -79,9 +79,10 @@ Durable execution ``DatabricksRunNowOperator`` triggers a run of an existing job and then polls it to completion on the worker. By default the operator runs in a *durable* mode that makes this crash-safe: the -Databricks run id is persisted to Airflow's task state store before polling begins, so if the -worker crashes or is preempted and the task is retried, the operator reconnects to the run that is -already executing on Databricks instead of triggering a duplicate run of the same job. +Databricks run id is persisted to :doc:`task state store +` before polling begins, so if the worker crashes +or is preempted and the task is retried, the operator reconnects to the run that is already +executing on Databricks instead of triggering a duplicate run of the same job. On retry the operator checks the prior run's state: diff --git a/providers/databricks/docs/operators/submit_run.rst b/providers/databricks/docs/operators/submit_run.rst index f6dbffa5ef16d..808f7534995d0 100644 --- a/providers/databricks/docs/operators/submit_run.rst +++ b/providers/databricks/docs/operators/submit_run.rst @@ -165,9 +165,10 @@ Durable execution ``DatabricksSubmitRunOperator`` submits a run and then polls it to completion on the worker. By default the operator runs in a *durable* mode that makes the runs crash-safe: the Databricks -run id is persisted to task state store before polling begins, so if the worker crashes or is -preempted and the task is retried, the operator reconnects to the run that is already executing -on Databricks instead of submitting a duplicate. +run id is persisted to :doc:`task state store ` +before polling begins, so if the worker crashes or is preempted and the task is retried, the +operator reconnects to the run that is already executing on Databricks instead of submitting a +duplicate. On retry the operator checks the prior run's state: diff --git a/providers/google/docs/operators/cloud/bigquery.rst b/providers/google/docs/operators/cloud/bigquery.rst index 1513a95867015..9f46ef351fac2 100644 --- a/providers/google/docs/operators/cloud/bigquery.rst +++ b/providers/google/docs/operators/cloud/bigquery.rst @@ -374,9 +374,10 @@ Durable execution ``BigQueryInsertJobOperator`` submits a job and then polls it to completion on the worker. By default the operator runs in a *durable* mode that makes this crash-safe: the submitted BigQuery -job id is persisted to task state before polling begins, so if the worker crashes or is preempted -and the task is retried, the operator reconnects to the job that is already running in BigQuery -instead of submitting a duplicate. +job id is persisted to :doc:`task state store ` +before polling begins, so if the worker crashes or is preempted and the task is retried, the +operator reconnects to the job that is already running in BigQuery instead of submitting a +duplicate. On retry the operator checks the prior job's state: diff --git a/providers/snowflake/docs/operators/snowflake.rst b/providers/snowflake/docs/operators/snowflake.rst index 85c1df3b2eedc..5bb2beb8daca0 100644 --- a/providers/snowflake/docs/operators/snowflake.rst +++ b/providers/snowflake/docs/operators/snowflake.rst @@ -160,9 +160,10 @@ Durable execution ``SnowflakeSqlApiOperator`` submits one or more SQL statements and then polls their statement handles to completion on the worker. By default the operator runs in a *durable* mode that makes -this crash-safe: the statement handles are persisted to task state store before polling begins, so -if the worker crashes or is preempted and the task is retried, the operator reconnects to the -statements that are already executing in Snowflake instead of resubmitting the SQL. +this crash-safe: the statement handles are persisted to :doc:`task state store +` before polling begins, so if the worker crashes +or is preempted and the task is retried, the operator reconnects to the statements that are +already executing in Snowflake instead of resubmitting the SQL. On retry the operator checks the prior statements' state: