[SPARK-37070][TEST] Pass all UTs in mllib-local and mllib with Java 17

[LuciferYang]

What changes were proposed in this pull request?

Mockito can't mock j.u.Random with Java 17 due to module java.base does not export jdk.internal.util.random to unnamed module and there are some UTs in mllib-local and mllib module failed with Java 17.

So the main change of this pr is add --add-exports=java.base/jdk.internal.util.random=ALL-UNNAMED to extraTestJavaArgs to make all UTs passed in mllib-local and mllib module.

Why are the changes needed?

Pass UT with JDK 17

Does this PR introduce any user-facing change?

No

How was this patch tested?

• Pass the Jenkins or GitHub Action
• Manual test with Java 17

mvn clean install -pl mllib-local
mvn clean install -pl mllib

Before

Run completed in 714 milliseconds.
Total number of tests run: 96
Suites: completed 9, aborted 0
Tests: succeeded 92, failed 4, canceled 0, ignored 0, pending 0
*** 4 TESTS FAILED ***
 
Run completed in 10 minutes, 54 seconds.
Total number of tests run: 1639
Suites: completed 206, aborted 0
Tests: succeeded 1635, failed 4, canceled 0, ignored 7, pending 0
*** 4 TESTS FAILED ***

All failed UTs has similar reasons:

- rand *** FAILED ***
  org.mockito.exceptions.base.MockitoException: Mockito cannot mock this class: class java.util.Random.

Mockito can only mock non-private & non-final classes.
If you're not sure why you're getting this error, please report to the mailing list.

Java               : 17
JVM vendor name    : Azul Systems, Inc.
JVM vendor version : 17+35-LTS
JVM name           : OpenJDK 64-Bit Server VM
JVM version        : 17+35-LTS
JVM info           : mixed mode, sharing
OS name            : Mac OS X
OS version         : 11.4

Underlying exception : java.lang.IllegalStateException: Cannot access annotation property public abstract boolean jdk.internal.util.random.RandomSupport$RandomGeneratorProperties.isStochastic()
  at org.scalatestplus.mockito.MockitoSugar.mock(MockitoSugar.scala:73)
  at org.scalatestplus.mockito.MockitoSugar.mock$(MockitoSugar.scala:72)
  at org.scalatestplus.mockito.MockitoSugar$.mock(MockitoSugar.scala:155)
  at org.apache.spark.ml.linalg.MatricesSuite.$anonfun$new$54(MatricesSuite.scala:807)
  at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  at org.scalatest.Transformer.apply(Transformer.scala:22)
  at org.scalatest.Transformer.apply(Transformer.scala:20)
  at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
  ...
  Cause: java.lang.IllegalStateException: Cannot access annotation property public abstract boolean jdk.internal.util.random.RandomSupport$RandomGeneratorProperties.isStochastic()
  at net.bytebuddy.description.annotation.AnnotationDescription$ForLoadedAnnotation.getValue(AnnotationDescription.java:663)
  at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.handle(AnnotationAppender.java:298)
  at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.doAppend(AnnotationAppender.java:361)
  at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.append(AnnotationAppender.java:338)
  at net.bytebuddy.implementation.attribute.TypeAttributeAppender$ForInstrumentedType.apply(TypeAttributeAppender.java:93)
  at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForCreation.create(TypeWriter.java:5652)
  at net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2166)
  at net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamicTypeBuilder.make(SubclassDynamicTypeBuilder.java:232)
  at net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamicTypeBuilder.make(SubclassDynamicTypeBuilder.java:204)
  at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase.make(DynamicType.java:3595)
  ...
  Cause: java.lang.IllegalAccessException: class net.bytebuddy.description.annotation.AnnotationDescription$ForLoadedAnnotation cannot access interface jdk.internal.util.random.RandomSupport$RandomGeneratorProperties (in module java.base) because module java.base does not export jdk.internal.util.random to unnamed module @e55b13
  at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:392)
  at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:674)
  at java.base/java.lang.reflect.Method.invoke(Method.java:560)
  at net.bytebuddy.description.annotation.AnnotationDescription$ForLoadedAnnotation.getValue(AnnotationDescription.java:642)
  at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.handle(AnnotationAppender.java:298)
  at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.doAppend(AnnotationAppender.java:361)
  at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.append(AnnotationAppender.java:338)
  at net.bytebuddy.implementation.attribute.TypeAttributeAppender$ForInstrumentedType.apply(TypeAttributeAppender.java:93)
  at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForCreation.create(TypeWriter.java:5652)
  at net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2166)
  ...

After

Run completed in 1 second, 7 milliseconds.
Total number of tests run: 96
Suites: completed 9, aborted 0
Tests: succeeded 96, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

Run completed in 10 minutes, 25 seconds.
Total number of tests run: 1639
Suites: completed 206, aborted 0
Tests: succeeded 1639, failed 0, canceled 0, ignored 7, pending 0
All tests passed.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48978/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48978/

[SparkQA]

Test build #144506 has finished for PR 34355 at commit 27d2c24.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

+1, LGTM.
Merged to master For Apache Spark 3.3.

[LuciferYang]

thanks @dongjoon-hyun