[WIP][SPARK-40497][BUILD] Upgrade Scala to 2.13.9

[LuciferYang]

What changes were proposed in this pull request?

This pr aims to update from Scala 2.13.8 to Scala 2.13.9 for Apache Spark 3.4.

Why are the changes needed?

Does this PR introduce any user-facing change?

Yes, this is a Scala version change.

How was this patch tested?

• Pass the GitHub Action Scala 2.13 job
• Manual test (Will add)

[LuciferYang]

Let GA compile first, and I will check all UTs locally

[LuciferYang]

[WARNING] : [deprecation @  | origin= | version=] -target is deprecated: Use -release instead to compile against the correct platform API.
[ERROR] /Users/spark-source/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala:1047: type mismatch;
 found   : (x: Char): builder.type <and> (x: Double): builder.type <and> (x: Float): builder.type <and> (x: Long): builder.type <and> (x: Int): builder.type <and> (x: Short): builder.type <and> (x: Byte): builder.type <and> (x: Boolean): builder.type <and> (s: StringBuilder): builder.type <and> (cs: CharSequence): builder.type <and> (s: String): builder.type <and> (x: Any): builder.type (with underlying type (x: Char): builder.type <and> (x: Double): builder.type <and> (x: Float): builder.type <and> (x: Long): builder.type <and> (x: Int): builder.type <and> (x: Short): builder.type <and> (x: Byte): builder.type <and> (x: Boolean): builder.type <and> (s: StringBuilder): builder.type <and> (cs: CharSequence): builder.type <and> (s: String): builder.type <and> (x: Any): builder.type)
 required: String => ?
[WARNING] /Users/spark-source/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:131: [deprecation @ org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.prepareWrite | origin=org.apache.parquet.hadoop.ParquetOutputFormat.ENABLE_JOB_SUMMARY | version=] value ENABLE_JOB_SUMMARY in class ParquetOutputFormat is deprecated
[ERROR] /Users/spark-source/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowCreateTableExec.scala:69: type mismatch;
 found   : (x: Char): builder.type <and> (x: Double): builder.type <and> (x: Float): builder.type <and> (x: Long): builder.type <and> (x: Int): builder.type <and> (x: Short): builder.type <and> (x: Byte): builder.type <and> (x: Boolean): builder.type <and> (s: StringBuilder): builder.type <and> (cs: CharSequence): builder.type <and> (s: String): builder.type <and> (x: Any): builder.type (with underlying type (x: Char): builder.type <and> (x: Double): builder.type <and> (x: Float): builder.type <and> (x: Long): builder.type <and> (x: Int): builder.type <and> (x: Short): builder.type <and> (x: Byte): builder.type <and> (x: Boolean): builder.type <and> (s: StringBuilder): builder.type <and> (cs: CharSequence): builder.type <and> (s: String): builder.type <and> (x: Any): builder.type)
 required: String => ?
[ERROR] /Users/spark-source/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowCreateTableExec.scala:122: type mismatch;
 found   : (x: Char): builder.type <and> (x: Double): builder.type <and> (x: Float): builder.type <and> (x: Long): builder.type <and> (x: Int): builder.type <and> (x: Short): builder.type <and> (x: Byte): builder.type <and> (x: Boolean): builder.type <and> (s: StringBuilder): builder.type <and> (cs: CharSequence): builder.type <and> (s: String): builder.type <and> (x: Any): builder.type (with underlying type (x: Char): builder.type <and> (x: Double): builder.type <and> (x: Float): builder.type <and> (x: Long): builder.type <and> (x: Int): builder.type <and> (x: Short): builder.type <and> (x: Byte): builder.type <and> (x: Boolean): builder.type <and> (s: StringBuilder): builder.type <and> (cs: CharSequence): builder.type <and> (s: String): builder.type <and> (x: Any): builder.type)
 required: String => ?
[ERROR] /Users/spark-source/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowCreateTableExec.scala:149: type mismatch;
 found   : (x: Char): builder.type <and> (x: Double): builder.type <and> (x: Float): builder.type <and> (x: Long): builder.type <and> (x: Int): builder.type <and> (x: Short): builder.type <and> (x: Byte): builder.type <and> (x: Boolean): builder.type <and> (s: StringBuilder): builder.type <and> (cs: CharSequence): builder.type <and> (s: String): builder.type <and> (x: Any): builder.type (with underlying type (x: Char): builder.type <and> (x: Double): builder.type <and> (x: Float): builder.type <and> (x: Long): builder.type <and> (x: Int): builder.type <and> (x: Short): builder.type <and> (x: Byte): builder.type <and> (x: Boolean): builder.type <and> (s: StringBuilder): builder.type <and> (cs: CharSequence): builder.type <and> (s: String): builder.type <and> (x: Any): builder.type)
 required: String => ?
[WARNING] /Users/spark-source/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetWrite.scala:93: [deprecation @ org.apache.spark.sql.execution.datasources.v2.parquet.ParquetWrite.prepareWrite | origin=org.apache.parquet.hadoop.ParquetOutputFormat.ENABLE_JOB_SUMMARY | version=] value ENABLE_JOB_SUMMARY in class ParquetOutputFormat is deprecated
[WARNING] three warnings found
[ERROR] four errors found

[LuciferYang]

case class TestObj(testValue: Option[String] = None)
def appendToBuilder(obj: TestObj, builder: StringBuilder): Unit = {
  obj.testValue.foreach(builder.append)
}

In Scala 2.13.8:

Welcome to Scala 2.13.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_322).
Type in expressions for evaluation. Or try :help.

scala> :paste
// Entering paste mode (ctrl-D to finish)

case class TestObj(testValue: Option[String] = None)
def appendToBuilder(obj: TestObj, builder: StringBuilder): Unit = {
  obj.testValue.foreach(builder.append)
}


// Exiting paste mode, now interpreting.

class TestObj
def appendToBuilder(obj: TestObj, builder: StringBuilder): Unit

scala>

In Scala 2.13.9

Welcome to Scala 2.13.9 (OpenJDK 64-Bit Server VM, Java 1.8.0_322).
Type in expressions for evaluation. Or try :help.

scala> :paste
// Entering paste mode (ctrl-D to finish)

case class TestObj(testValue: Option[String] = None)
def appendToBuilder(obj: TestObj, builder: StringBuilder): Unit = {
  obj.testValue.foreach(builder.append)
}


// Exiting paste mode, now interpreting.


  obj.testValue.foreach(builder.append)
                                ^
<pastie>:3: error: type mismatch;
 found   : (x: Char): builder.type <and> (x: Double): builder.type <and> (x: Float): builder.type <and> (x: Long): builder.type <and> (x: Int): builder.type <and> (x: Short): builder.type <and> (x: Byte): builder.type <and> (x: Boolean): builder.type <and> (s: StringBuilder): builder.type <and> (cs: CharSequence): builder.type <and> (s: String): builder.type <and> (x: Any): builder.type (with underlying type (x: Char): builder.type <and> (x: Double): builder.type <and> (x: Float): builder.type <and> (x: Long): builder.type <and> (x: Int): builder.type <and> (x: Short): builder.type <and> (x: Byte): builder.type <and> (x: Boolean): builder.type <and> (s: StringBuilder): builder.type <and> (cs: CharSequence): builder.type <and> (s: String): builder.type <and> (x: Any): builder.type)
 required: String => ?

It seems that this is not an understandable change. I think we should skip the upgrade of Scala 2.13.9

WDYT? @dongjoon-hyun @srowen @HyukjinKwon

[HyukjinKwon]

Yes. should ideally report to Scala 2.13.9 too to fix the compatibility issue.

[LuciferYang]

An issue has been created in the scala community

[LuciferYang]

Yes. should ideally report to Scala 2.13.9 too to fix the compatibility issue.

Let me check if there are any other incompatibilities and when upgrading to the next version, we can reuse this Jira

[dongjoon-hyun]

Thank you for checking. +1 for skipping Scala 2.13.9 and reusing this JIRA for 2.13.10.

[LuciferYang]

maven test Java 8 + Scala 2.13.9 with this pr:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.4.0-SNAPSHOT:
[INFO] 
[INFO] Spark Project Parent POM ........................... SUCCESS [  3.765 s]
[INFO] Spark Project Tags ................................. SUCCESS [  8.773 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 15.212 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 10.418 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 58.304 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 14.684 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 12.029 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  7.129 s]
[INFO] Spark Project Core ................................. SUCCESS [28:25 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 49.680 s]
[INFO] Spark Project GraphX ............................... SUCCESS [01:50 min]
[INFO] Spark Project Streaming ............................ SUCCESS [04:58 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [10:59 min]
[INFO] Spark Project SQL .................................. SUCCESS [  01:34 h]
[INFO] Spark Project ML Library ........................... SUCCESS [21:41 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 10.673 s]
[INFO] Spark Project Hive ................................. SUCCESS [  01:37 h]
[INFO] Spark Project REPL ................................. SUCCESS [01:34 min]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 12.431 s]
[INFO] Spark Project YARN ................................. SUCCESS [10:19 min]
[INFO] Spark Project Mesos ................................ SUCCESS [01:45 min]
[INFO] Spark Project Kubernetes ........................... SUCCESS [04:16 min]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [27:16 min]
[INFO] Spark Ganglia Integration .......................... SUCCESS [ 10.064 s]
[INFO] Spark Project Hadoop Cloud Integration ............. SUCCESS [ 23.943 s]
[INFO] Spark Project Assembly ............................. SUCCESS [  4.519 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 38.650 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [02:18 min]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [34:47 min]
[INFO] Spark Kinesis Integration .......................... SUCCESS [01:10 min]
[INFO] Spark Project Examples ............................. SUCCESS [38:18 min]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 14.551 s]
[INFO] Spark Avro ......................................... SUCCESS [04:09 min]
[INFO] Spark Project Kinesis Assembly ..................... SUCCESS [ 17.387 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  06:31 h
[INFO] Finished at: 2022-09-22T19:55:11+08:00
[INFO] ------------------------------------------------------------------------

All passed, seems no other incompatibilities, I will close this pr later

[LuciferYang]

Thank you for checking. +1 for skipping Scala 2.13.9 and reusing this JIRA for 2.13.10.

I found the fix already moved to Scala 2.13.11 milestone , so we may have to wait until 2.13.11 to upgrade

[LuciferYang]

@HyukjinKwon @dongjoon-hyun @panbingkun I checked Scala 2.13.10 with build/sbt "sql/compile" -Pscala-2.13 , confirmed the problem still exists in this version, so I think we need wait until 2.13.11 to upgrade.

[dongjoon-hyun]

Thank you for making this pass by changing the code.

[dongjoon-hyun]

Thank you for the update, @LuciferYang . Feel free to update the JIRA title, SPARK-40497

so I think we need wait until 2.13.11 to upgrade.