Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-37880][BUILD] Upgrade Scala to 2.13.8 #35181

Closed
wants to merge 1 commit into from
Closed

[SPARK-37880][BUILD] Upgrade Scala to 2.13.8 #35181

wants to merge 1 commit into from

Conversation

LuciferYang
Copy link
Contributor

@LuciferYang LuciferYang commented Jan 12, 2022 •
edited

What changes were proposed in this pull request?

This pr aims to update from Scala 2.13.7 to Scala 2.13.8 for Apache Spark 3.3.

Why are the changes needed?

Scala 2.13.8 is a maintenance release for 2.13 line and the release notes as follows:

Does this PR introduce any user-facing change?

No

How was this patch tested?

  • Pass the GitHub Action Scala 2.13 job

  • Manual test (Will add)

@LuciferYang LuciferYang marked this pull request as draft January 12, 2022 14:48
@github-actions github-actions bot added the BUILD label Jan 12, 2022
@LuciferYang
Copy link
Contributor Author

Maven compiled successfully locally, waiting GA

dongjoon-hyun
dongjoon-hyun requested changes Jan 12, 2022
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @LuciferYang .

Actually, the following is not an issue at Apache Spark because we use jline2. Could you remove that misleading description?

this version fixed REPL broken on M1 Macs

The following is the result of Apache Spark. FYI, the Apache Spark community is running a Jenkins CI on M1 machine.

$ java -version
openjdk version "17.0.1" 2021-10-19 LTS
OpenJDK Runtime Environment Zulu17.30+15-CA (build 17.0.1+12-LTS)
OpenJDK 64-Bit Server VM Zulu17.30+15-CA (build 17.0.1+12-LTS, mixed mode, sharing)

$ bin/spark-shell
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/01/12 08:21:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1642004498149).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.3.0-SNAPSHOT
      /_/

Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 17.0.1)
Type in expressions to have them evaluated.
Type :help for more information.

scala>

@LuciferYang
Copy link
Contributor Author

LuciferYang commented Jan 13, 2022 •
edited

ok~ some UT failed with Scala 2.13.8, I will check it and upgrade the pr later:

AdaptiveQueryExecSuite

- Logging plan changes for AQE *** FAILED ***
  java.util.ConcurrentModificationException: mutation occurred during iteration
  at scala.collection.mutable.MutationTracker$.checkMutations(MutationTracker.scala:43)
  at scala.collection.mutable.CheckedIndexedSeqView$CheckedIterator.hasNext(CheckedIndexedSeqView.scala:47)
  at scala.collection.StrictOptimizedIterableOps.filterImpl(StrictOptimizedIterableOps.scala:225)
  at scala.collection.StrictOptimizedIterableOps.filterImpl$(StrictOptimizedIterableOps.scala:222)
  at scala.collection.mutable.ArrayBuffer.filterImpl(ArrayBuffer.scala:43)
  at scala.collection.StrictOptimizedIterableOps.filterNot(StrictOptimizedIterableOps.scala:220)
  at scala.collection.StrictOptimizedIterableOps.filterNot$(StrictOptimizedIterableOps.scala:220)
  at scala.collection.mutable.ArrayBuffer.filterNot(ArrayBuffer.scala:43)
  at org.apache.spark.SparkFunSuite$LogAppender.loggingEvents(SparkFunSuite.scala:288)
  at org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite.$anonfun$new$152(AdaptiveQueryExecSuite.scala:1487)

@LuciferYang
Copy link
Contributor Author

LuciferYang commented Jan 13, 2022 •
edited

Actually, the following is not an issue at Apache Spark because we use jline2. Could you remove that misleading description?

I've deleted the relevant pr description. I'll verify it later

@LuciferYang
Copy link
Contributor Author

ok~ some UT failed with Scala 2.13.8, I will check it and upgrade the pr later:

AdaptiveQueryExecSuite

- Logging plan changes for AQE *** FAILED ***
  java.util.ConcurrentModificationException: mutation occurred during iteration
  at scala.collection.mutable.MutationTracker$.checkMutations(MutationTracker.scala:43)
  at scala.collection.mutable.CheckedIndexedSeqView$CheckedIterator.hasNext(CheckedIndexedSeqView.scala:47)
  at scala.collection.StrictOptimizedIterableOps.filterImpl(StrictOptimizedIterableOps.scala:225)
  at scala.collection.StrictOptimizedIterableOps.filterImpl$(StrictOptimizedIterableOps.scala:222)
  at scala.collection.mutable.ArrayBuffer.filterImpl(ArrayBuffer.scala:43)
  at scala.collection.StrictOptimizedIterableOps.filterNot(StrictOptimizedIterableOps.scala:220)
  at scala.collection.StrictOptimizedIterableOps.filterNot$(StrictOptimizedIterableOps.scala:220)
  at scala.collection.mutable.ArrayBuffer.filterNot(ArrayBuffer.scala:43)
  at org.apache.spark.SparkFunSuite$LogAppender.loggingEvents(SparkFunSuite.scala:288)
  at org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite.$anonfun$new$152(AdaptiveQueryExecSuite.scala:1487)

It seems a flaky test, I re-run it many times but never reproduce.At the same time, it does not seem to be related to scala-2.13.8 because the relevant files have not changed since scala-2.13.7, such as ArrayBuffer, StrictOptimizedIterableOps and CheckedIndexedSeqView, so I will file a new Jira to tracking this issue

dongjoon-hyun
dongjoon-hyun reviewed Jan 13, 2022
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's great. Could you switch to Ready for review status and remove [WIP], @LuciferYang ?

@LuciferYang LuciferYang marked this pull request as ready for review January 14, 2022 02:59
@LuciferYang LuciferYang changed the title [WIP][SPARK-37880][BUILD] Upgrade Scala to 2.13.8 [SPARK-37880][BUILD] Upgrade Scala to 2.13.8 Jan 14, 2022
dongjoon-hyun
dongjoon-hyun approved these changes Jan 14, 2022
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you so much, @LuciferYang .
Merged to master for Apache Spark 3.3.

dchvn pushed a commit to dchvn/spark that referenced this pull request Jan 19, 2022
@LuciferYang @dchvn
[SPARK-37880][BUILD] Upgrade Scala to 2.13.8
825f51d 
### What changes were proposed in this pull request?
This pr aims to update from Scala 2.13.7 to Scala 2.13.8 for Apache Spark 3.3.

### Why are the changes needed?
Scala 2.13.8 is a maintenance release for 2.13 line and the release notes as follows:
- https://github.com/scala/scala/releases/tag/v2.13.8

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?

- Pass the GitHub Action Scala 2.13 job

- Manual test (Will add)

Closes apache#35181 from LuciferYang/SPARK-37880.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
@LuciferYang LuciferYang deleted the SPARK-37880 branch October 22, 2023 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

Assignees
No one assigned
Labels
BUILD
Projects
None yet
Milestone
No milestone
2 participants
@LuciferYang @dongjoon-hyun