-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-31168][BUILD] Upgrade Scala to 2.12.14 #32697
Conversation
Kubernetes integration test unable to build dist. exiting with code: 1 |
Test build #139065 has finished for PR 32697 at commit
|
Looks good if it passes, but looks like some library is missing: sbt.librarymanagement.ResolveException: Error downloading com.github.ghik:silencer-lib_2.12.14:1.7.5 |
From https://github.com/ghik/silencer: NOTE: Scala 2.13.2 and 2.12.13 introduced configurable warnings. Maybe we can remove it and avoid this error. |
I see, just exclude it? possibly. Looks like it may just need some time to get cross-published as the artifact appears quite specific to the scala version: https://search.maven.org/search?q=silencer-lib |
Hi, @srowen and @iemejia . Yes, it does. As I wrote in the PR description, I made a PR to silencer and it's merged three hours ago and released.
This is the released one. I'll rebase this PR to the branch to retrigger it. However, the second item in the PR description,
And, @iemejia . I'm intentionally not going down that path. :) We can do that separately after this PR. |
Test build #139082 has finished for PR 32697 at commit
|
Kubernetes integration test unable to build dist. exiting with code: 1 |
Now, genjavadoc is ready. |
Kubernetes integration test unable to build dist. exiting with code: 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good if CI can pass.
Thank you, @viirya ! |
Test build #139085 has finished for PR 32697 at commit
|
@@ -162,7 +162,7 @@ | |||
<commons.math3.version>3.4.1</commons.math3.version> | |||
<!-- managed up from 3.2.1 for SPARK-11652 --> | |||
<commons.collections.version>3.2.2</commons.collections.version> | |||
<scala.version>2.12.10</scala.version> | |||
<scala.version>2.12.14</scala.version> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dongjoon-hyun please modify the scala version to 2.12.14 in profile scala-2.12
, otherwise there maybe a Error:scala: bad option: -P:silencer:globalFilters=.*deprecated.*
error when tick the profile scala-2.12
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, thanks. I missed there.
### What changes were proposed in this pull request? This PR is a follow-up of #32697 to update the missed part. After SPARK-34774, we have Scala 2.12 version in `scala-2.12` profile. ### Why are the changes needed? To be consistent. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs and manual. **BEFORE** ``` $ build/mvn help:evaluate -Pscala-2.12 -Dexpression=scala.version | grep "^2.12" Using `mvn` from path: /usr/local/bin/mvn 2.12.10 ``` **AFTER** ``` $ build/mvn help:evaluate -Pscala-2.12 -Dexpression=scala.version | grep "^2.12" Using `mvn` from path: /usr/local/bin/mvn 2.12.14 ``` Closes #32707 from dongjoon-hyun/SPARK-31168-2. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Excellent @dongjoon-hyun! My suggestion to remove silencer was to unblock the issue but this can be definitely be done afterwards. |
Sure, I agree with you that we need to remove it in Apache Spark 3.2.0 timeframe. Thanks, @iemejia .
|
* [SPARK-31168][BUILD] Upgrade Scala to 2.12.14 ### What changes were proposed in this pull request? This PR is the 4th try to upgrade Scala 2.12.x in order to see the feasibility. - #27929 (Upgrade Scala to 2.12.11, wangyum ) - #30940 (Upgrade Scala to 2.12.12, viirya ) - #31223 (Upgrade Scala to 2.12.13, dongjoon-hyun ) Note that Scala 2.12.14 has the following fix for Apache Spark community. - Fix cyclic error in runtime reflection (protobuf), a regression that prevented Spark upgrading to 2.12.13 REQUIREMENTS: - [x] `silencer` library is released via ghik/silencer#66 - [x] `genjavadoc` library is released via lightbend/genjavadoc#282 ### Why are the changes needed? Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11/2.12.12/2.12.13. This will bring all the bug fixes. - https://github.com/scala/scala/releases/tag/v2.12.14 - https://github.com/scala/scala/releases/tag/v2.12.13 - https://github.com/scala/scala/releases/tag/v2.12.12 - https://github.com/scala/scala/releases/tag/v2.12.11 ### Does this PR introduce _any_ user-facing change? Yes, but this is a bug-fixed version. ### How was this patch tested? Pass the CIs. Closes #32697 from dongjoon-hyun/SPARK-31168. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 6c4b60f) * [SPARK-31168][BUILD] Upgrade Scala to 2.12.14 ### What changes were proposed in this pull request? This PR is the 4th try to upgrade Scala 2.12.x in order to see the feasibility. - #27929 (Upgrade Scala to 2.12.11, wangyum ) - #30940 (Upgrade Scala to 2.12.12, viirya ) - #31223 (Upgrade Scala to 2.12.13, dongjoon-hyun ) Note that Scala 2.12.14 has the following fix for Apache Spark community. - Fix cyclic error in runtime reflection (protobuf), a regression that prevented Spark upgrading to 2.12.13 REQUIREMENTS: - [x] `silencer` library is released via ghik/silencer#66 - [x] `genjavadoc` library is released via lightbend/genjavadoc#282 ### Why are the changes needed? Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11/2.12.12/2.12.13. This will bring all the bug fixes. - https://github.com/scala/scala/releases/tag/v2.12.14 - https://github.com/scala/scala/releases/tag/v2.12.13 - https://github.com/scala/scala/releases/tag/v2.12.12 - https://github.com/scala/scala/releases/tag/v2.12.11 ### Does this PR introduce _any_ user-facing change? Yes, but this is a bug-fixed version. ### How was this patch tested? Pass the CIs. Closes #32697 from dongjoon-hyun/SPARK-31168. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 6c4b60f) * [SPARK-36759][BUILD] Upgrade Scala to 2.12.15 ### What changes were proposed in this pull request? This PR aims to upgrade Scala to 2.12.15 to support Java 17/18 better. ### Why are the changes needed? Scala 2.12.15 improves compatibility with JDK 17 and 18: https://github.com/scala/scala/releases/tag/v2.12.15 - Avoids IllegalArgumentException in JDK 17+ for lambda deserialization - Upgrades to ASM 9.2, for JDK 18 support in optimizer ### Does this PR introduce _any_ user-facing change? Yes, this is a Scala version change. ### How was this patch tested? Pass the CIs Closes #33999 from dongjoon-hyun/SPARK-36759. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 16f1f71) * [SPARK-36759][BUILD][FOLLOWUP] Update version in scala-2.12 profile and doc ### What changes were proposed in this pull request? This is a follow-up to fix the leftover during switching the Scala version. ### Why are the changes needed? This should be consistent. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? This is not tested by UT. We need to check manually. There is no more `2.12.14`. ``` $ git grep 2.12.14 R/pkg/tests/fulltests/test_sparkSQL.R: c(as.Date("2012-12-14"), as.Date("2013-12-15"), as.Date("2014-12-16"))) data/mllib/ridge-data/lpsa.data:3.5307626,0.987291634724086 -0.36279314978779 -0.922212414640967 0.232904453212813 -0.522940888712441 1.79270085261407 0.342627053981254 1.26288870310799 sql/hive/src/test/resources/data/files/over10k:-3|454|65705|4294967468|62.12|14.32|true|mike white|2013-03-01 09:11:58.703087|40.18|joggying ``` Closes #34020 from dongjoon-hyun/SPARK-36759-2. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit adbea25) * [SPARK-39414][BUILD] Upgrade Scala to 2.12.16 ### What changes were proposed in this pull request? This PR aims to upgrade Scala to 2.12.16 ### Why are the changes needed? This version bring some bug fix and start to try to support Java 19 scala/scala@v2.12.15...v2.12.16 - [Upgrade to asm 9.3, for JDK19 support](scala/scala#10000) - [Fix codegen for MH.invoke etc under JDK 17 -release](scala/scala#9930) - [Deprecation related SecurityManager on JDK 17 ](scala/scala#9775) ### Does this PR introduce _any_ user-facing change? Yes, this is a Scala version change. ### How was this patch tested? Pass Github Actions Closes #36807 from LuciferYang/SPARK-39414. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit ed875a8) * fix * fix * fix * fix �[0m[�[31merror�[0m] �[0m[warn] /home/jenkins/workspace/spark-sql-catalyst-3.0/core/src/main/scala/org/apache/spark/scheduler/SpillableTaskResultGetter.scala:36: non-variable type argument org.apache.spark.scheduler.DirectTaskResult[_] in type pattern scala.runtime.NonLocalReturnControl[org.apache.spark.scheduler.DirectTaskResult[_]] is unchecked since it is eliminated by erasure�[0m �[0m[�[31merror�[0m] �[0m[warn] private[spark] class SpillableTaskResultGetter(sparkEnv: SparkEnv, scheduler: TaskSchedulerImpl)�[0m �[0m[�[31merror�[0m] �[0m[warn] �[0m * fix �[0m[�[31merror�[0m] �[0m[warn] /home/jenkins/workspace/spark-sql-catalyst-3.0/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala:287: match may not be exhaustive.�[0m �[0m[�[31merror�[0m] �[0mIt would fail on the following input: ~(~(_, (x: String forSome x not in "^")), _)�[0m �[0m[�[31merror�[0m] �[0m[warn] private val pow: Parser[Term] = term ~ "^" ~ "^[1-9]\\d*".r ^^ {�[0m �[0m[�[31merror�[0m] �[0m[warn] �[0m �[0m[�[31merror�[0m] �[0m[warn] /home/jenkins/workspace/spark-sql-catalyst-3.0/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala:301: match may not be exhaustive.�[0m �[0m[�[31merror�[0m] �[0mIt would fail on the following input: ~(~(_, (x: String forSome x not in "~")), _)�[0m �[0m[�[31merror�[0m] �[0m[warn] (label ~ "~" ~ expr) ^^ { case r ~ "~" ~ t => ParsedRFormula(r, t.asTerms.terms) }�[0m �[0m[�[31merror�[0m] �[0m[warn] �[0m * fix Co-authored-by: Dongjoon Hyun <dhyun@apple.com> Co-authored-by: Dongjoon Hyun <dongjoon@apache.org> Co-authored-by: yangjie01 <yangjie01@baidu.com>
What changes were proposed in this pull request?
This PR is the 4th try to upgrade Scala 2.12.x in order to see the feasibility.
Note that Scala 2.12.14 has the following fix for Apache Spark community.
REQUIREMENTS:
silencer
library is released via Add 2.12.14 ghik/silencer#66genjavadoc
library is released via Release for Scala 2.12.14 lightbend/genjavadoc#282Why are the changes needed?
Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11/2.12.12/2.12.13. This will bring all the bug fixes.
Does this PR introduce any user-facing change?
Yes, but this is a bug-fixed version.
How was this patch tested?
Pass the CIs.