[SPARK-55682][CORE][SQL] ServiceLoader returned iterator may throw NoClassDefFoundError on hasNext()

[pan3793]

What changes were proposed in this pull request?

JDK 24 introduces a behavior change on ServiceLoader's returned iterator, iterator.hasNext() might throw
NoClassDefFoundError, which previously only happened on iterator.next(). See more details at OpenJDK
bug report JDK-8350481

Another issue is even the Javadoc of ServiceLoader say

Its hasNext and next methods can therefore throw a ServiceConfigurationError for any of the reasons specified
in the Errors section above. To write robust code it is only necessary to catch ServiceConfigurationError when
using the iterator. If an error is thrown then subsequent invocations of the iterator will make a best effort
to locate and instantiate the next available provider, but in general such recovery cannot be guaranteed.

but it actually might throw NoClassDefFoundError. See more details at OpenJDK bug report JDK-8196182

Why are the changes needed?

Enable Java 25 support.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Changes in HadoopDelegationTokenManager are covered by UT, other 2 places are not, need to review carefully.

$ export JAVA_HOME=/path/of/openjdk-25
$ build/sbt -Phive "hive/testOnly *HiveHadoopDelegationTokenManagerSuite"

Before

[info]   Cause: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
[info]   at org.apache.spark.sql.hive.security.HiveHadoopDelegationTokenManagerSuite$$anon$1.loadClass(HiveHadoopDelegationTokenManagerSuite.scala:51)
[info]   at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:490)
[info]   at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
[info]   at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:2985)
[info]   at java.base/java.lang.Class.getConstructor0(Class.java:3180)
[info]   at java.base/java.lang.Class.getConstructor(Class.java:2199)
[info]   at java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:623)
[info]   at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1111)
[info]   at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1142)
[info]   at java.base/java.util.ServiceLoader$1.hasNext(ServiceLoader.java:1164)
[info]   at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1246)
[info]   at org.apache.spark.deploy.security.HadoopDelegationTokenManager.loadProviders(HadoopDelegationTokenManager.scala:273)
[info]   at org.apache.spark.deploy.security.HadoopDelegationTokenManager.<init>(HadoopDelegationTokenManager.scala:79)
...
[info] Run completed in 1 second, 186 milliseconds.
[info] Total number of tests run: 3
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 2, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***
[error] Failed tests:
[error] 	org.apache.spark.sql.hive.security.HiveHadoopDelegationTokenManagerSuite
[error] (hive / Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 192 s (0:03:12.0), completed Feb 25, 2026, 4:40:21 PM

After

[info] HiveHadoopDelegationTokenManagerSuite:
[info] - default configuration (44 milliseconds)
16:41:38.695 WARN org.apache.spark.deploy.security.HadoopDelegationTokenManager: spark.yarn.security.credentials.hive.enabled is deprecated. Please use spark.security.credentials.hive.enabled instead.
[info] - using deprecated configurations (6 milliseconds)
[info] - SPARK-23209: obtain tokens when Hive classes are not available (284 milliseconds)
[info] Run completed in 1 second, 200 milliseconds.
[info] Total number of tests run: 3
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 33 s, completed Feb 25, 2026, 4:41:43 PM

A similar issue was fixed in Hadoop too, see HADOOP-19821. I created a simple project to help reviewers better understand this issue and solution.

Was this patch authored or co-authored using generative AI tooling?

No.

[LuciferYang]

    var keepLoading = true
    while (keepLoading) {
      try {
        if (iterator.hasNext) {
          providers += iterator.next
        } else {
          keepLoading = false
        }
      } catch {
        case t: Throwable =>
          logDebug(s"Failed to load built in provider.", t)
      }
    }

How about this? The Breaks feature will be deprecated in higher versions of Scala 3 (although it won't affect current code).

[pan3793]

The Breaks feature will be deprecated in higher versions of Scala 3

did not know that, let me change it to your suggested pattern

[dongjoon-hyun]

+1, LGTM.

[pan3793]

thanks, merged to master for 4.2