Fix SparkKubernetesOperator with deferrable=True falling through to KubernetesPodOperator.execute() creating a spurious pod

[Pranaykarvi]

Fixes #66437

What does this PR do?

When SparkKubernetesOperator was used with deferrable=True, after calling
execute_async(context) there was no return statement, so execution fell
through to super().execute(context) (i.e. KubernetesPodOperator.execute()).
That path attempted to create a bare Kubernetes Pod with no container image,
producing a 422 Unprocessable Entity error from the API even though the
SparkApplication CR had already been submitted successfully.

Root cause

SparkKubernetesOperator.execute() called self.execute_async(context) in the
deferrable=True branch but had no return after it, causing fall-through to
return super().execute(context).

Fix

Added a single return statement immediately after self.execute_async(context)
in the if self.deferrable: branch. The non-deferrable path is unchanged.

Tests

• test_deferrable_does_not_fall_through_to_kpo_execute: asserts that with
  deferrable=True, execute_async is called and KubernetesPodOperator.execute
  is never called.

• test_non_deferrable_calls_kpo_execute: asserts that with deferrable=False,
  KubernetesPodOperator.execute is still called as before.

• Tests added

• Newsfragment added (providers/cncf/kubernetes/newsfragments/66437.bugfix.rst)

• Closes SparkKubernetesOperator with deferrable=True falls through to KubernetesPodOperator.execute() creating a spurious pod #66437

[SameerMesiah97]

Left a few comments. Also, it would be great if you could confirm that you were able to reproduce the bug and verify that your fix works. This is just for due diligence.

[Pranaykarvi]

@SameerMesiah97 Updated based on your review feedback:

• Removed newsfragments/66448.bugfix.rst
• Moved test_execute_deferrable_does_not_call_super and test_execute_non_deferrable_calls_super into TestSparkKubernetesOperator, using the class's existing fixtures instead of standalone setup helpers
• Removed _spark_kubernetes_test_connections and the two old standalone test functions

CI running — please re-review when ready.

[nikhilkarve]

@shahar1 @Pranaykarvi

Thanks for the fix. I have a question regarding this issue:

#66437

I encountered this issue in the deployed environment, but I was unable to reproduce it locally. Because of that, I created the issue to investigate further and have a discussion around it.

I’m not fully clear on how this PR addresses the issue, especially since it could not be reproduced in a local environment.

Also, as @SameerMesiah97 asked earlier, could you please confirm whether the author of the PR was able to reproduce the issue before working on the fix? I don’t think there was a response to that question earlier.