Skip to content

Drop unused post_submit_commands copy from SparkSubmitOperator init - #70323

Merged
shahar1 merged 1 commit into
apache:mainfrom
1fanwang:fix-sparksubmit-init-template-field
Jul 24, 2026
Merged

Drop unused post_submit_commands copy from SparkSubmitOperator init#70323
shahar1 merged 1 commit into
apache:mainfrom
1fanwang:fix-sparksubmit-init-template-field

Conversation

@1fanwang

@1fanwang 1fanwang commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

post_submit_commands is a template field, so it is rendered after __init__ runs. The constructor also kept a private self._post_submit_commands = list(post_submit_commands) if post_submit_commands else [], built from the un-rendered value. Nothing reads that copy: the operator forwards the rendered self.post_submit_commands to SparkSubmitHook, which builds its own list and runs the commands. Removing it stops the constructor reading the template field and clears its validate-operators-init exemption.

related: #70296

Testing Done

validate_operators_init.py on the operator, before (with the exemption removed):

SparkSubmitOperator's constructor applies logic to template fields. Template fields are
rendered after the constructor runs, so validation or transformation here acts on the
un-rendered Jinja expression and should move to execute():
  line 266: self._post_submit_commands = list(post_submit_commands) if post_submit_commands else []  (post_submit_commands)

After removing the line, the validator exits 0.

test_spark_submit.py (operator suite): 69 passed. The removed attribute had no readers, so behaviour is unchanged.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: GitHub Copilot CLI following the guidelines

post_submit_commands is a template field, so it is rendered after __init__
runs. The constructor kept a second, private list copy
(self._post_submit_commands) built from the un-rendered value, but nothing
reads it: the operator forwards the rendered self.post_submit_commands to
SparkSubmitHook, which builds its own list. Removing the dead copy stops the
constructor reading the template field and lets its validate-operators-init
exemption go.

related: apache#70296
Signed-off-by: 1fanwang <1fannnw@gmail.com>

@shahar1 shahar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!

@shahar1
shahar1 merged commit dca1d97 into apache:main Jul 24, 2026
152 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-3-test Commit Link

You can attempt to backport this manually by running:

cherry_picker dca1d97 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants