[SPARK-44690][SPARK-44376][BUILD] Downgrade Scala to 2.13.8

[LuciferYang]

What changes were proposed in this pull request?

The aim of this PR is to downgrade the Scala 2.13 dependency to 2.13.8 to ensure that Spark can be build with -target:jvm-1.8, and tested with Java 11/17.

Why are the changes needed?

As reported in SPARK-44376, there are issues when maven build and test using Java 11/17 with -target:jvm-1.8:

• run build/mvn clean install -Pscala-2.13 with Java 17

[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first) @ spark-core_2.13 ---
[INFO] Compiler bridge file: /Users/yangjie01/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.8.0-bin_2.13.11__61.0-1.8.0_20221110T195421.jar
[INFO] compiling 602 Scala sources and 77 Java sources to /Users/yangjie01/SourceCode/git/spark-mine-13/core/target/scala-2.13/classes ...
[WARNING] [Warn] : [deprecation @  | origin= | version=] -target is deprecated: Use -release instead to compile against the correct platform API.
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: not found: value sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26: not found: object sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: not found: object sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:206: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:210: not found: type Unsafe
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:212: not found: type Unsafe
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:213: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:216: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:236: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26: Unused import
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: Unused import
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:452: not found: value sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: not found: object sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type SignalHandler
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:83: not found: type Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: type SignalHandler
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: value Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:114: not found: type Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:116: not found: value Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:128: not found: value Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: Unused import
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: Unused import
[WARNING] one warning found
[ERROR] 23 errors found

• run build/mvn clean install -Pscala-2.13 -Djava.version=17 with Java 17

[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first) @ spark-tags_2.13 ---
[INFO] Compiler bridge file: /Users/yangjie01/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.8.0-bin_2.13.11__61.0-1.8.0_20221110T195421.jar
[INFO] compiling 2 Scala sources and 8 Java sources to /Users/yangjie01/SourceCode/git/spark-mine-13/common/tags/target/scala-2.13/classes ...
[WARNING] [Warn] : [deprecation @  | origin= | version=] -target is deprecated: Use -release instead to compile against the correct platform API.
[ERROR] [Error] : target platform version 8 is older than the release version 17
[WARNING] one warning found
[ERROR] one error found

• run build/mvn clean package -Pscala-2.13 -DskipTests or build/mvn clean install -Pscala-2.13 -DskipTests with Java 8 first, then run build/mvn test -Pscala-2.13 with Java 17

[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first) @ spark-core_2.13 ---
[INFO] Compiler bridge file: /Users/yangjie01/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.8.0-bin_2.13.11__61.0-1.8.0_20221110T195421.jar
[INFO] compiling 602 Scala sources and 77 Java sources to /Users/yangjie01/SourceCode/git/spark-mine-13/core/target/scala-2.13/classes ...
[WARNING] [Warn] : [deprecation @  | origin= | version=] -target is deprecated: Use -release instead to compile against the correct platform API.
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: not found: value sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26: not found: object sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: not found: object sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:206: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:210: not found: type Unsafe
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:212: not found: type Unsafe
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:213: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:216: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:236: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26: Unused import
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: Unused import
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:452: not found: value sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: not found: object sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type SignalHandler
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:83: not found: type Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: type SignalHandler
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: value Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:114: not found: type Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:116: not found: value Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:128: not found: value Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: Unused import
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: Unused import
[WARNING] one warning found
[ERROR] 23 errors found

This is inconsistent with the behavior of the released Apache Spark version, so we need to use the previous Scala2.13 version to support this behavior.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

• Pass GitHub Actions
• Manual checked, the above command can run normally after this pr

[LuciferYang]

cc @HyukjinKwon @srowen we can use this one.
also cc @eejbyfeldt for more check.

Since Scala 2.13.11 is a blocker for the Scala 2.13 series to support Java 21, we will temporarily be unable to support Java-21ea version build and testing after this downgrade for Scala 2.13. also cc @dongjoon-hyun to know.

For related discussions, please refer to #41943 and #42362

[srowen]

It looks like it. It of course would be possible to create a custom build that evidently works on 21 with only minor changes (or are we really sure 2.13.8 + Java 21 doesn't work?) I think we have to make this trade if it's Java 8 support or 21 support; Java 8 support is a property on the 3.x line but 21 isn't even out yet.

[HyukjinKwon]

I am good with this change.

[LuciferYang]

Merged into master and branch-3.5

[LuciferYang]

Thanks @HyukjinKwon @srowen

[dongjoon-hyun]

Thank you, @LuciferYang .

Sorry for being late, but I have two minor questions for my understanding without further context except this PR.

• Given that it's reverted to Scala 2.13.8 from 2.13.11, do you happen to know the Scala issue about this?
• When we are going to drop Java 8 support in master for Apache Spark 4.0, do we still need this revert for -target:jvm-1.8 on master branch? Or, does Scala 2.13.11 have any other issues?

[srowen]

There's a big discussion at #41943

I think we can update to Scala 2.13.11 again after we decide to drop Java 8 support. Then, the build targets Java 11 or 17 and we shouldn't face the same issue.

[LuciferYang]

There's a big discussion at #41943

I think we can update to Scala 2.13.11 again after we decide to drop Java 8 support. Then, the build targets Java 11 or 17 and we shouldn't face the same issue.

@srowen You might have missed my previous comments: #41943 (comment)

I think even if we drop Java 8, like support Java 11+, we would still face this issue. It might be necessary to try to resolve this problem by scala-maven-plugin ...

[srowen]

Yeah I think we always can hit this problem, if we want to make 1 release that works over N Java versions. I think the incompatibility issues are far smaller after Java 9.

However, the setup we are contemplating is to (for example) build with Java 11, not to build with 17 and target 11. That's not quite what you tested.

I personally think we should target Java 17 for Spark 4.0. Same problem is possible - does it run on 21? - but I think it's much less likely we face that, or we simply work around as we always did in the source code.

[LuciferYang]

@dongjoon-hyun

Given that it's reverted to Scala 2.13.8 from 2.13.11, do you happen to know the Scala issue about this?

The main reason for the downgrade is that version 3.5 has not been released yet, and the master needs to maintain consistent behavior with previous versions. If there is a better way to solve this problem, we can upgrade again.

Sorry for overlooked this scenario during the last upgrade.

When we are going to drop Java 8 support in master for Apache Spark 4.0, do we still need this revert for -target:jvm-1.8 on master branch? Or, does Scala 2.13.11 have any other issues?

My personal understanding is that we either accept the solution proposed in #41943: use -target:${java.version}, or we can find another solution to keep the -target as a fixed value, fFor example, modifying scala-maven-plugin ...

[dongjoon-hyun]

Thank you for the reply, @LuciferYang .

[LuciferYang]

Let me add two test scenarios:

1. Build with Java 11 and test with Java 17&21-ea with Scala 2.13.11
2. Build with Java 17 and test with Java 21-ea with Scala 2.13.11

[srowen]

I don't think we'd ever maintain two binary releases built with different Java versions. So I think the scenario is always: build with minimum supported Java version, and test versus all supported versions.

[LuciferYang]

Just noticed that all of yesterday's subsequent comments were written here...
#42362 (comment)