[Umbrella] Support Scala 2.13

[pan3793]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the proposal

Scala 2.13 has been adopted widely, e.g. Apache Spark, Apache Kafka, it's time to make Kyuubi support Scala 2.13.

We should make Kyuubi pass build Scala 2.13 first, and deliver scala 2.13 binary tarball in 1.8 or 1.9

Task list

• [Subtask] Refine deprecated config #3194
• [Subtask] Allow use dynamic Scala binary version in Maven module's artifactId suffix  #5177
• [DOC] Make MarkdownBuilder not extending Growable #5178
• Use Iterable instead of Seq in kyuubi-ctl commands #5179
• Replace deprecated classes by equivalent in scala.tools.nsc.interpreter package #5180
• [Improvement] Rename Kyuubi's StageInfo to SparkStageInfo to fix class mismatch #5184
• Introducing profile scala-2.13 for development use #5183
• Introduce change-scala-version.sh
• pass compile on scala-2.13 (maybe module by module)
  • Make server module and Spark sql engine module compilable on Scala 2.13 #5188
  • spark authz plugin [AUTHZ] Make spark authz module compilable on Scala 2.13 #5189
  • spark extension plugin Make Spark extension plugin compilable on Scala 2.13 #5191
  • spark lineage plugin Make Spark sql lineage plugin compilable on Scala 2.13 #5192
  • spark hive connector plugin Make Spark hive connector plugin compilable on Scala 2.13 #5193
  • spark tpc-ds / tpc-h connector plugin Make Spark TPC-H connector plugin compilable on Scala 2.13 #5195
• add CI in github action for scala-2.13 Make server module and Spark sql engine module compilable on Scala 2.13 #5188

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[SergWh]

Anyone planning to take this one? If not, I can try to make PR later this week. In our company we managed to make working 2.13 distribution (only modules required to run with Spark) but code needs a little cleanup

[pan3793]

Thanks @SergWh, feel free to open PRs, we plan to make the whole project support scala 2.13 eventually.

[AdarshRawat1]

Requesting permission to contribute to this feature. . Can you please advise on how I can get involved? Thanks

[pan3793]

@AdarshRawat1 thanks for interesting in this project.

I suppose you have basic knowledge about Maven, Scala, and How to send GitHub PR. (correct me if not).

You can start w/ one module, say kyuubi-common, making it pass compiling w/ Scala 2.13.

[AdarshRawat1]

@AdarshRawat1 thanks for interesting in this project.

I suppose you have basic knowledge about Maven, Scala, and How to send GitHub PR. (correct me if not).

You can start w/ one module, say kyuubi-common, making it pass compiling w/ Scala 2.13.

Yes, I have basic knowledge about Maven, Scala and sending GitHub PR.
I'll start with kyuubi-common. Could you kindly provide me with additional resources and guidance, please?

[pan3793]

Basically, add a scala-2.13 profile in root pom.xml

        <profile>
            <id>scala-2.13</id>
            <properties>
                <scala.version>2.13.10</scala.version>
                <scala.binary.version>2.13</scala.binary.version>
            </properties>
        </profile>

Change the kyuubi-common/pom.xml

-    <artifactId>kyuubi-common_2.12</artifactId>
+    <artifactId>kyuubi-common_${scala.binary.version}</artifactId>

And then, compile using build/mvn clean package -pl kyuubi-common -am -Pscala-2.13, there may be some compiling or testing errors caused by breaking changes between scala 2.12 and 2.13, find and fix them until pass the building and compiling.

Also, better to update the GitHub workflow to check your change by CI

[chrisfw]

Hello,

First, thank you for all of the great work on this project. It has made exposing Iceberg functionality through JDBC much easier. I just wanted to inquire about the status of completing the integration of Scala 2.13? Some of the most recent iceberg functionality seems to require 2.13 in order to successfully utilize the latest iceberg spark sql extensions (org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions). Attempting to use them in Spark with Scala 2.12 results in a class not found exception that is not an issue when using 2.13. This extension is necessary to call Iceberg Spark procedures, e.g.

CALL my_catalog.system.create_changelog_view(table => 'iceberg.mysample',
 identifier_columns => array('id'),
 compute_updates => true );

I am hoping to utilize this functionality through kyuubi hence my interest in this issue.

Regards,
Chris Whelan

[pan3793]

Attempting to use them in Spark with Scala 2.12 results in a class not found exception that is not an issue when using 2.13

Please provide more information, which artifacts you are using? any error stacks? scala 2.12 is the spark default scala version until the latest 3.4.0, it should be adopted more widely in production than scala 2.13