[SPARK-45046][BUILD] Set shadeTestJar of core module to false

[LuciferYang]

What changes were proposed in this pull request?

This pr aims to set shadeTestJar of core module to false to skip shade spark-core**-tests.jar process.

Why are the changes needed?

Before SPARK-41244, the core module used the maven-shade-plugin configuration in the parent pom.xml for shading, where the shadeTestJar configuration item was set to the default false in the maven-shade-plugin in parent pom.xml.

In SPARK-41244 | SPARK-41244 #38779, the core module started using a separate maven-shade-plugin configuration and set the shadeTestJar configuration item to true.

And when shadeTestJar is true, maven-shade-plugin always try to find(sometime is downloading) some non-existent jars:

[INFO] --- maven-shade-plugin:3.5.0:shade (default) @ spark-core_2.12 ---
[INFO] Including org.eclipse.jetty:jetty-plus:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-plus/9.4.51.v20230217/jetty-plus-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-plus/9.4.51.v20230217/jetty-plus-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-plus:jar:9.4.51.v20230217:compile
[INFO] Including org.eclipse.jetty:jetty-security:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-security/9.4.51.v20230217/jetty-security-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-security/9.4.51.v20230217/jetty-security-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-security:jar:9.4.51.v20230217:compile
[INFO] Including org.eclipse.jetty:jetty-util:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-util/9.4.51.v20230217/jetty-util-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-util/9.4.51.v20230217/jetty-util-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-util:jar:9.4.51.v20230217:compile
[INFO] Including org.eclipse.jetty:jetty-server:jar:9.4.51.v20230217 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-io:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-io/9.4.51.v20230217/jetty-io-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-io/9.4.51.v20230217/jetty-io-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-io:jar:9.4.51.v20230217:compile
[INFO] Including org.eclipse.jetty:jetty-http:jar:9.4.51.v20230217 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-continuation:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-continuation/9.4.51.v20230217/jetty-continuation-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-continuation/9.4.51.v20230217/jetty-continuation-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-continuation:jar:9.4.51.v20230217:compile
[INFO] Including org.eclipse.jetty:jetty-servlet:jar:9.4.51.v20230217 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-proxy:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-proxy/9.4.51.v20230217/jetty-proxy-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-proxy/9.4.51.v20230217/jetty-proxy-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-proxy:jar:9.4.51.v20230217:compile
[INFO] Including org.eclipse.jetty:jetty-client:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-client/9.4.51.v20230217/jetty-client-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-client/9.4.51.v20230217/jetty-client-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-client:jar:9.4.51.v20230217:compile
[INFO] Including org.eclipse.jetty:jetty-servlets:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-servlets/9.4.51.v20230217/jetty-servlets-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-servlets/9.4.51.v20230217/jetty-servlets-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-servlets:jar:9.4.51.v20230217:compile
[INFO] Including com.google.protobuf:protobuf-java:jar:3.23.4 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/com/google/protobuf/protobuf-java/3.23.4/protobuf-java-3.23.4-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/3.23.4/protobuf-java-3.23.4-tests.jar
[WARNING] Could not get tests for com.google.protobuf:protobuf-java:jar:3.23.4:compile
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/spark-project/spark/unused/1.0.0/unused-1.0.0-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/spark-project/spark/unused/1.0.0/unused-1.0.0-tests.jar
[WARNING] Could not get tests for org.spark-project.spark:unused:jar:1.0.0:compile

Under poor network conditions, the time for Maven compilation will increase.

On the other hand, setting shadeTestJar back to false has not resulted in any Maven test failures, so this pr reset shadeTestJar to false.

Does this PR introduce any user-facing change?

No

How was this patch tested?

• Pass GitHub Action

Was this patch authored or co-authored using generative AI tooling?

No

[LuciferYang]

Set shadeTestJar to false to check Maven tests: https://github.com/LuciferYang/spark/runs/16407130667

All maven tests passed.

[HyukjinKwon]

cc @gengliangwang FYI

[LuciferYang]

@gengliangwang Do you still remember why changed shadeTestJar to false?

[gengliangwang]

@gengliangwang Do you still remember why changed shadeTestJar to false?

I guess it was a mistake...

[gengliangwang]

BTW, there were multiple places setting shadeTestJar as true. @LuciferYang could you update the unnecessary ones as well?

$ git grep shadeTestJar
connector/protobuf/pom.xml:          <shadeTestJar>false</shadeTestJar>
core/pom.xml:          <shadeTestJar>true</shadeTestJar>
streaming/pom.xml:          <shadeTestJar>true</shadeTestJar>

[LuciferYang]

@gengliangwang The connector module has already been set to false, and the core module is changed to false in the current PR. The streaming module is a historical legacy and it seems to be intentionally retained(37a5e27).

[gengliangwang]

@LuciferYang Got it, thanks!

[LuciferYang]

Merged into master. Thanks @gengliangwang @HyukjinKwon

[LuciferYang]

@gengliangwang After further investigation, there are three modules that depend on spark-streaming_2.12:test-jar: mllib, streaming-kafka-0-10 and kinesis-asl. As I lack the necessary credentials to access AWS, I'm unable to verify all Unit Tests of the kinesis-asl module, so I've chosen to retain the configuration of <shadeTestJar>true</shadeTestJar> in the streaming module.