[SPARK-53780] spark-submission-worker should depend on io.fabric8 directly

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to make spark-submission-worker depend on io.fabric8 directly like spark-operator-api module.

Why are the changes needed?

Currently, spark-submission-worker are using the transitive dependency from spark-kubernetes module when it uses io.fabric8 library. This is inconsistent and a wrong use case.

BEFORE

$ gradle :spark-submission-worker:dependencies --configuration compileClasspath
...
+--- org.projectlombok:lombok:1.18.42
+--- project :spark-operator-api
\--- org.apache.spark:spark-kubernetes_2.13:4.0.1
     +--- org.apache.spark:spark-core_2.13:4.0.1
     |   ...
     +--- io.fabric8:kubernetes-client:7.1.0

AFTER

$ gradle :spark-submission-worker:dependencies --configuration compileClasspath
...
+--- org.projectlombok:lombok:1.18.42
+--- project :spark-operator-api
+--- io.fabric8:kubernetes-client:7.4.0

Does this PR introduce any user-facing change?

No behavior change.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

cc @jiangzho , too.

[dongjoon-hyun]

Thank you, @viirya ! Merged to main.