Skip to content

Consider adding deferrable support to SparkSubmitOperator #67168

@amoghrajesh

Description

@amoghrajesh

Context

PR #67118 is attempting to add sync resumable support to SparkSubmitOperator via ResumableJobMixin. During review, @ashb suggested also supporting a deferrable=True mode in the same operator.

What this issue tracks

Add deferrable: bool = False parameter to SparkSubmitOperator:

Relationship to #67118: The two modes share spark_job_id in task_state. A user can switch from sync to deferrable without any state migration.

Metadata

Metadata

Assignees

Labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions