Skip to content

Commit

Permalink
[SPARK-44297][CORE][TESTS] Make ClassLoaderIsolationSuite test pass…
Browse files Browse the repository at this point in the history
… with Scala 2.13

### What changes were proposed in this pull request?
The main change of this pr as follows:

1. rename `TestHelloV2.jar` and `TestHelloV3.jar` added in #41789 to `TestHelloV2_2.12.jar` and `TestHelloV3_2.12.jar`
2. Add corresponding `TestHelloV2_2.13.jar` and `TestHelloV3_2.13.jar` which compiled with Scala 2.13
3. Make `ClassLoaderIsolationSuite` use the correct jar in testing

### Why are the changes needed?
Make `ClassLoaderIsolationSuite` test pass with Scala 2.13.

The Scala 2.13 daily test failed after #41789
- https://github.com/apache/spark/actions/runs/5447771717/jobs/9910185372

```
[info] - Executor classloader isolation with JobArtifactSet *** FAILED *** (83 milliseconds)
[info]   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (localhost executor driver): java.lang.NoClassDefFoundError: scala/Serializable
[info] 	at java.lang.ClassLoader.defineClass1(Native Method)
[info] 	at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
[info] 	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
[info] 	at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
[info] 	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
[info] 	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
[info] 	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
[info] 	at java.security.AccessController.doPrivileged(Native Method)
[info] 	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
[info] 	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
[info] 	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
[info] 	at java.lang.Class.forName0(Native Method)
[info] 	at java.lang.Class.forName(Class.java:348)
[info] 	at org.apache.spark.util.SparkClassUtils.classForName(SparkClassUtils.scala:35)
[info] 	at org.apache.spark.util.SparkClassUtils.classForName$(SparkClassUtils.scala:30)
[info] 	at org.apache.spark.util.Utils$.classForName(Utils.scala:94)
[info] 	at org.apache.spark.executor.ClassLoaderIsolationSuite.$anonfun$new$3(ClassLoaderIsolationSuite.scala:53)
[info] 	at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.scala:18)
[info] 	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
[info] 	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
[info] 	at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
[info] 	at org.apache.spark.rdd.RDD.$anonfun$foreach$2(RDD.scala:1028)
[info] 	at org.apache.spark.rdd.RDD.$anonfun$foreach$2$adapted(RDD.scala:1028)
[info] 	at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2406)
[info] 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
[info] 	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
[info] 	at org.apache.spark.scheduler.Task.run(Task.scala:141)
[info] 	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:593)
[info] 	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1478)
[info] 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:596)
[info] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info] 	at java.lang.Thread.run(Thread.java:750)
[info] Caused by: java.lang.ClassNotFoundException: scala.Serializable
[info] 	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
[info] 	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
[info] 	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
[info] 	... 33 more
```

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
- Pass GitHub Actions
- Manual check:

**Scala 2.12**
```
build/sbt "core/testOnly *ClassLoaderIsolationSuite"
```

```
[info] ClassLoaderIsolationSuite:
[info] - Executor classloader isolation with JobArtifactSet (1 second, 394 milliseconds)
[info] Run completed in 2 seconds, 437 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```

**Scala 2.13**

```
dev/change-scala-version.sh 2.13
build/sbt "core/testOnly *ClassLoaderIsolationSuite" -Pscala-2.13
```

```
[info] ClassLoaderIsolationSuite:
[info] - Executor classloader isolation with JobArtifactSet (1 second, 355 milliseconds)
[info] Run completed in 2 seconds, 264 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```

Closes #41852 from LuciferYang/SPARK-44297.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: yangjie01 <yangjie01@baidu.com>
  • Loading branch information
LuciferYang committed Jul 4, 2023
1 parent 0e8a95e commit 16295a3
Show file tree
Hide file tree
Showing 5 changed files with 12 additions and 2 deletions.
File renamed without changes.
Binary file added core/src/test/resources/TestHelloV2_2.13.jar
Binary file not shown.
File renamed without changes.
Binary file added core/src/test/resources/TestHelloV3_2.13.jar
Binary file not shown.
Original file line number Diff line number Diff line change
Expand Up @@ -17,21 +17,31 @@

package org.apache.spark.executor

import scala.util.Properties

import org.apache.spark.{JobArtifactSet, LocalSparkContext, SparkConf, SparkContext, SparkFunSuite}
import org.apache.spark.util.Utils

class ClassLoaderIsolationSuite extends SparkFunSuite with LocalSparkContext {

private val scalaVersion = Properties.versionNumberString
.split("\\.")
.take(2)
.mkString(".")

val jar1 = Thread.currentThread().getContextClassLoader.getResource("TestUDTF.jar").toString

// package com.example
// object Hello { def test(): Int = 2 }
// case class Hello(x: Int, y: Int)
val jar2 = Thread.currentThread().getContextClassLoader.getResource("TestHelloV2.jar").toString
val jar2 = Thread.currentThread().getContextClassLoader
.getResource(s"TestHelloV2_$scalaVersion.jar").toString

// package com.example
// object Hello { def test(): Int = 3 }
// case class Hello(x: String)
val jar3 = Thread.currentThread().getContextClassLoader.getResource("TestHelloV3.jar").toString
val jar3 = Thread.currentThread().getContextClassLoader
.getResource(s"TestHelloV3_$scalaVersion.jar").toString

test("Executor classloader isolation with JobArtifactSet") {
sc = new SparkContext(new SparkConf().setAppName("test").setMaster("local"))
Expand Down

0 comments on commit 16295a3

Please sign in to comment.