Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions providers/amazon/docs/operators/redshift/redshift_data.rst
Original file line number Diff line number Diff line change
Expand Up @@ -71,9 +71,9 @@ Durable execution

``RedshiftDataOperator`` submits a statement and then polls it to completion on the worker. By
default the operator runs in a *durable* mode that makes this crash-safe: the Redshift statement
id is persisted to task state before polling begins, so if the worker crashes or is preempted and
the task is retried, the operator reconnects to the statement that is already executing in
Redshift instead of resubmitting the SQL.
id is persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
polling begins, so if the worker crashes or is preempted and the task is retried, the operator
reconnects to the statement that is already executing in Redshift instead of resubmitting the SQL.

On retry the operator checks the prior statement's state:

Expand Down
6 changes: 3 additions & 3 deletions providers/apache/spark/docs/operators.rst
Original file line number Diff line number Diff line change
Expand Up @@ -190,9 +190,9 @@ independently on the cluster. If the Airflow worker dies while the Spark job is
Airflow loses track of it and the behaviour to submit a brand new job would be wasting
the compute already done or even cause conflicts if the Spark job itself is not designed to be idempotent.

Now, the ``SparkSubmitOperator`` solves this by persisting the driver ID to ``task_state_store`` immediately after
submission. On retry, it reads the ID back and reconnects to the already-running driver instead of
resubmitting.
Now, the ``SparkSubmitOperator`` solves this by persisting the driver ID to :doc:`task state store
<apache-airflow:core-concepts/task-state-store>` immediately after submission. On retry, it reads
the ID back and reconnects to the already-running driver instead of resubmitting.

This is the **synchronous path** — the worker holds a slot for the duration of polling. This is
a crash-safety net for teams running sync operators for log observability, org constraints, or
Expand Down
7 changes: 4 additions & 3 deletions providers/databricks/docs/operators/run_now.rst
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,10 @@ Durable execution

``DatabricksRunNowOperator`` triggers a run of an existing job and then polls it to completion on
the worker. By default the operator runs in a *durable* mode that makes this crash-safe: the
Databricks run id is persisted to Airflow's task state store before polling begins, so if the
worker crashes or is preempted and the task is retried, the operator reconnects to the run that is
already executing on Databricks instead of triggering a duplicate run of the same job.
Databricks run id is persisted to :doc:`task state store
<apache-airflow:core-concepts/task-state-store>` before polling begins, so if the worker crashes
or is preempted and the task is retried, the operator reconnects to the run that is already
executing on Databricks instead of triggering a duplicate run of the same job.

On retry the operator checks the prior run's state:

Expand Down
7 changes: 4 additions & 3 deletions providers/databricks/docs/operators/submit_run.rst
Original file line number Diff line number Diff line change
Expand Up @@ -165,9 +165,10 @@ Durable execution

``DatabricksSubmitRunOperator`` submits a run and then polls it to completion on the worker.
By default the operator runs in a *durable* mode that makes the runs crash-safe: the Databricks
run id is persisted to task state store before polling begins, so if the worker crashes or is
preempted and the task is retried, the operator reconnects to the run that is already executing
on Databricks instead of submitting a duplicate.
run id is persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>`
before polling begins, so if the worker crashes or is preempted and the task is retried, the
operator reconnects to the run that is already executing on Databricks instead of submitting a
duplicate.

On retry the operator checks the prior run's state:

Expand Down
7 changes: 4 additions & 3 deletions providers/google/docs/operators/cloud/bigquery.rst
Original file line number Diff line number Diff line change
Expand Up @@ -374,9 +374,10 @@ Durable execution

``BigQueryInsertJobOperator`` submits a job and then polls it to completion on the worker. By
default the operator runs in a *durable* mode that makes this crash-safe: the submitted BigQuery
job id is persisted to task state before polling begins, so if the worker crashes or is preempted
and the task is retried, the operator reconnects to the job that is already running in BigQuery
instead of submitting a duplicate.
job id is persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>`
before polling begins, so if the worker crashes or is preempted and the task is retried, the
operator reconnects to the job that is already running in BigQuery instead of submitting a
duplicate.

On retry the operator checks the prior job's state:

Expand Down
7 changes: 4 additions & 3 deletions providers/snowflake/docs/operators/snowflake.rst
Original file line number Diff line number Diff line change
Expand Up @@ -160,9 +160,10 @@ Durable execution

``SnowflakeSqlApiOperator`` submits one or more SQL statements and then polls their statement
handles to completion on the worker. By default the operator runs in a *durable* mode that makes
this crash-safe: the statement handles are persisted to task state store before polling begins, so
if the worker crashes or is preempted and the task is retried, the operator reconnects to the
statements that are already executing in Snowflake instead of resubmitting the SQL.
this crash-safe: the statement handles are persisted to :doc:`task state store
<apache-airflow:core-concepts/task-state-store>` before polling begins, so if the worker crashes
or is preempted and the task is retried, the operator reconnects to the statements that are
already executing in Snowflake instead of resubmitting the SQL.

On retry the operator checks the prior statements' state:

Expand Down
Loading