[SPARK-26144][BUILD] build/mvn should detect scala.version based on scala.binary.version
#23118
Conversation
…on `scala.binary.version`
|
cc @srowen , @shaneknapp , @dbtsai , @HyukjinKwon I noticed this during investigating Scala-2.11 Jenkins jobs. Sorry for pinging you guys at Thanksgiving holiday. As I'm the one who merged |
|
Haha today's not holiday here :D. |
|
Thank you, @HyukjinKwon . :D |
| @@ -116,7 +116,8 @@ install_zinc() { | |||
| # the build/ folder | |||
| install_scala() { | |||
| # determine the Scala version used in Spark | |||
| local scala_version=`grep "scala.version" "${_DIR}/../pom.xml" | head -n1 | awk -F '[<>]' '{print $3}'` | |||
| local scala_binary_version=`grep "scala.binary.version" "${_DIR}/../pom.xml" | head -n1 | awk -F '[<>]' '{print $3}'` | |||
| local scala_version=`grep "scala.version" "${_DIR}/../pom.xml" | grep ${scala_binary_version} | head -n1 | awk -F '[<>]' '{print $3}'` | |||
There was a problem hiding this comment.
So after doing dev/change-scala-version.sh 2.11, scala.version and scala.binary.version are expected to be not matched?
There was a problem hiding this comment.
Thank you for review, @viirya .
Previously, it doesn't matched; scala.binary.version=2.11 and scala.version=2.12.7.
Now, they will be matched; scala.binary.version=2.11 and scala.version=2.11.12.
By default (without change-scala-version.sh 2.11), it's scala.binary.version=2.12 and scala.version=2.12.7.
There was a problem hiding this comment.
Don't we need to update dev/change-scala-version.sh to let it update <scala.version>?
There was a problem hiding this comment.
I have above question because I think after current change, scala.version and scala.binary.version are still not matched in pom.xml, isn't?
Or we just need to do as current change to download & install correct Scala and this is enough?
There was a problem hiding this comment.
@viirya . I think you are confused about the meaning of scala.version.
In maven/sbt, scala.version is controlled by profile -Pscala-2.11.
This build/mvn doesn't know the profile -Pscala-2.11. That's the problem this PR want to fix.
There was a problem hiding this comment.
oh, I see. Thanks @dongjoon-hyun
|
Test build #99173 has finished for PR 23118 at commit
|
|
Retest this please. |
|
Test build #99193 has finished for PR 23118 at commit
|
|
Merged to master. |
|
Late to the party! Thanks @dongjoon-hyun for taking care of this. |
…on `scala.binary.version` ## What changes were proposed in this pull request? Currently, `build/mvn` downloads and uses **Scala 2.12.7** in `Scala-2.11` Jenkins job. The root cause is `build/mvn` got the first match from `pom.xml` blindly. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-ubuntu-scala-2.11/6/consoleFull ``` exec: curl -s -L https://downloads.lightbend.com/zinc/0.3.15/zinc-0.3.15.tgz exec: curl -s -L https://downloads.lightbend.com/scala/2.12.7/scala-2.12.7.tgz exec: curl -s -L https://www.apache.org/dyn/closer.lua?action=download&filename=/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz ``` ## How was this patch tested? Manual. ``` $ build/mvn clean exec: curl --progress-bar -L https://downloads.lightbend.com/scala/2.12.7/scala-2.12.7.tgz ... $ git clean -fdx $ dev/change-scala-version.sh 2.11 $ build/mvn clean exec: curl --progress-bar -L https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz ``` Closes apache#23118 from dongjoon-hyun/SPARK-26144. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
What changes were proposed in this pull request?
Currently,
build/mvndownloads and uses Scala 2.12.7 inScala-2.11Jenkins job. The root cause isbuild/mvngot the first match frompom.xmlblindly.How was this patch tested?
Manual.