[SPARK-51600][CORE] Prepend classes of sql/hive and sql/hive-thriftserver when isTesting || isTestingSql is true

[LuciferYang]

What changes were proposed in this pull request?

This pr aims to add a condition check for isTesting || isTestingSql to shouldPrePendSparkHive and shouldPrePendSparkHiveThriftServer. When running Maven tests, prepend classes should be performed for "sql/hive" and "sql/hive-thriftserver" modules.

Why are the changes needed?

After SPARK-49534 was merged, when spark-hive_xxx.jar is not present in the assembly/target/scala-2.13/jars directory, prepend classes will no longer be executed for sql/hive. Similar handling has been applied to sql/hive-thriftserver.

Although this resolves the issue described in #48015, it introduces another problem:

When we execute mvn test, if the dependent JARs are not pre-collected into the assembly/target/scala-2.13/jars directory and we directly run Maven tests on the sql/hive and sql/hive-thriftserver modules, some tests will fail.

Consider the following testing approach:

build/mvn clean -Phive -Phive-thriftserver
build/mvn clean install -DskipTests -pl sql/hive-thriftserver -am -Phive -Phive-thriftserver
build/mvn clean install -pl sql/hive-thriftserver -Phive -Phive-thriftserver
build/mvn clean install -pl sql/hive -Phive   

The tests for the sql/hive-thriftserver module *** RUN ABORTED *** due to the following reasons:

HiveThriftBinaryServerSuite:
18:48:19.595 ERROR org.apache.spark.sql.hive.thriftserver.HiveThriftBinaryServerSuite: 
=====================================
HiveThriftServer2Suite failure output
=====================================

### Attempt 0 ###
HiveThriftServer2 command line: ArraySeq(../../sbin/start-thriftserver.sh, --master, local, --hiveconf, javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=/home/runner/work/spark/spark/sql/hive-thriftserver/target/tmp/spark-2bca44a3-c220-485c-b2a4-289262293652;create=true, --hiveconf, hive.metastore.warehouse.dir=/home/runner/work/spark/spark/sql/hive-thriftserver/target...

18:48:22.634 WARN org.apache.spark.sql.hive.thriftserver.HiveThriftBinaryServerSuite: 

===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.hive.thriftserver.HiveThriftBinaryServerSuite, threads: Thread-10 (daemon=true), Thread-11 (daemon=true) =====


*** RUN ABORTED ***
An exception or error caused a run to abort: Future timed out after [3 minutes] 
  java.util.concurrent.TimeoutException: Future timed out after [3 minutes]
  at scala.concurrent.impl.Promise$DefaultPromise.tryAwait0(Promise.scala:248)
  at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:261)
  at org.apache.spark.util.SparkThreadUtils$.awaitResultNoSparkExceptionConversion(SparkThreadUtils.scala:61)
  at org.apache.spark.util.SparkThreadUtils$.awaitResult(SparkThreadUtils.scala:45)
  at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:342)
  at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2TestBase.startThriftServer(HiveThriftServer2Suites.scala:1345)
  at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2TestBase.$anonfun$beforeAll$4(HiveThriftServer2Suites.scala:1403)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
  at scala.util.Try$.apply(Try.scala:217)
  at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2TestBase.$anonfun$beforeAll$3(HiveThriftServer2Suites.scala:1402)
  ...

Error: Failed to load class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.
Failed to load main class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.

HiveSparkSubmitSuite will have 15 failed tests due to the following reasons:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/hive/HiveUtils$
  	at org.apache.spark.sql.hive.SetMetastoreURLTest$.main(HiveSparkSubmitSuite.scala:390)
  	at org.apache.spark.sql.hive.SetMetastoreURLTest.main(HiveSparkSubmitSuite.scala)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
  	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
  	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1027)
  	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:204)
  	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:227)
  	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:96)
  	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1132)
  	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1141)
  	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.hive.HiveUtils$
  	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
  	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
  	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
  	... 14 more

The reason why the issue is not triggered by the Maven daily test is that a full build is executed before the test, which completes the process of collecting JARs into the assembly/target/scala-2.13/jars directory.

Does this PR introduce any user-facing change?

No

How was this patch tested?

• Pass GitHub Actions
• Pass Maven test: https://github.com/LuciferYang/spark/runs/39370781864

• re-check the test in [SPARK-49534][CORE] No longer prepend sql/hiveand sql/hive-thriftserver when spark-hive_xxx.jar is not in the classpath #48015, the changes in this pr will not break it.

• Locally test

build/mvn clean -Phive -Phive-thriftserver
build/mvn clean install -DskipTests -pl sql/hive-thriftserver -am -Phive -Phive-thriftserver
build/mvn clean install -pl sql/hive-thriftserver -Phive -Phive-thriftserver
build/mvn clean install -pl sql/hive -Phive   

sql/hive-thriftserver

Run completed in 12 minutes, 55 seconds.
Total number of tests run: 640
Suites: completed 20, aborted 0
Tests: succeeded 640, failed 0, canceled 0, ignored 26, pending 0
All tests passed.

sql/hive

Run completed in 1 hour, 17 minutes, 15 seconds.
Total number of tests run: 3987
Suites: completed 148, aborted 0
Tests: succeeded 3987, failed 0, canceled 2, ignored 606, pending 0
All tests passed.

Was this patch authored or co-authored using generative AI tooling?

No

[LuciferYang]

Actually, we can reproduce the issue described in this PR (Pull Request) in GA by change the execution process of the Maven daily test.

• https://github.com/apache/spark/pull/50387/files

sql/hive-thriftserver

• https://github.com/LuciferYang/spark/actions/runs/14066164624/job/39389068986

• https://github.com/LuciferYang/spark/actions/runs/14066164624/job/39389068997

Afterwards, I will turn #50387 into a formal pr to enhance the Maven daily test.

[LuciferYang]

cc @HyukjinKwon FYI

[dongjoon-hyun]

+1, LGTM. Thank you, @LuciferYang .

[LuciferYang]

Merged into master/branch-4.0. Thanks @dongjoon-hyun