Context
PR #67118 is attempting to add sync resumable support to SparkSubmitOperator via ResumableJobMixin. During review, @ashb suggested also supporting a deferrable=True mode in the same operator.
What this issue tracks
Add deferrable: bool = False parameter to SparkSubmitOperator:
Relationship to #67118: The two modes share spark_job_id in task_state. A user can switch from sync to deferrable without any state migration.
Context
PR #67118 is attempting to add sync resumable support to
SparkSubmitOperatorviaResumableJobMixin. During review, @ashb suggested also supporting adeferrable=Truemode in the same operator.What this issue tracks
Add
deferrable: bool = Falseparameter toSparkSubmitOperator:deferrable=False(default): sync path withResumableJobMixin— worker slot occupied during polling, reconnects to existing driver on infrastructure failure (implemented in AddResumableJobMixinwithSparkSubmitOperatoras a case study for surviving worker failures (standalone) #67118)deferrable=True: submit job,defer()toSparkDriverTrigger, worker slot freed during polling. Whenexecute()is called again (only happens on user clear), resubmit fresh — no reconnect needed since crashes are handled by Trigger row persistence.Relationship to #67118: The two modes share
spark_job_idintask_state. A user can switch from sync to deferrable without any state migration.