[#1805][part-2] Support Spark4 (Scala 2.13)

[EnricoMi]

What changes were proposed in this pull request?

Adds Spark4 client.

Why are the changes needed?

Spark4 is about to be released. This tests that the existing Spark3 client is compatible with Spark4, and introduces a Spark3 client, which is a mere copy of the Spark3 client. Releasing an explicit Spark4 client will make it more obvious for users to pick the right client for their Spark4 environment.

Fix: #1805

Does this PR introduce any user-facing change?

It adds a Spark4 client, shaded client and integration tests.

How was this patch tested?

Existing Spark3 tests.

[EnricoMi]

What do you think about the plain copy of Spark3 client?

[EnricoMi]

How can we fix those nasty Kerberos failures with JDK 17?

java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Message stream modified (41) - Message stream modified)]; Host Details : local host is: "action-host/127.0.0.1"; destination host is: "localhost":40741;

https://github.com/apache/incubator-uniffle/runs/26438380881

[github-actions]

Test Results

 2 703 files  + 62   2 703 suites  +62   5h 30m 21s ⏱️ + 3m 42s
   948 tests +  5     941 ✅  -   1   1 💤 ±0  0 ❌ ±0  6 🔥 +6 
12 164 runs  +399  12 142 ✅ +392  16 💤 +1  0 ❌ ±0  6 🔥 +6 

For more details on these errors, see this check.

Results for commit a6ebf57. ± Comparison against base commit ea1070c.

[jerqi]

What do you think about the plain copy of Spark3 client?

What do you think about the plain copy of Spark3 client?

It's ok for me.

[jerqi]

What do you think about the plain copy of Spark3 client?

How can we fix those nasty Kerberos failures with JDK 17?

java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Message stream modified (41) - Message stream modified)]; Host Details : local host is: "action-host/127.0.0.1"; destination host is: "localhost":40741;

https://github.com/apache/incubator-uniffle/runs/26438380881

cc @zuston

[zuston]

From my sight, kerbreos is not ready for the JDK17. Maybe we could disble this in the java17