[ZEPPELIN-3810] Support Spark 2.4 #3206

HyukjinKwon · 2018-10-17T14:58:49Z

What is this PR for?

Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418).

There are two problems for this upgrade at Zeppelin side:

1.. Some methods that are used in private by reflection, for instance, loopPostInit became inaccessible.

See:

To work around this, I manually ported loopPostInit at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new loopPostInit by reflection.

2.. Upgrade from 2.11.8 to 2.11.12 requires jline.version upgrade. Otherwise, we will hit:

Caused by: java.lang.NoSuchMethodError: 
jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
  at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)

To work around this, I tweaked this by upgrading jline from 2.12.1 to 2.14.3.

What type of PR is it?

[Improvement]

Todos

- Wait until Spark 2.4.0 is officially released.

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-3810

How should this be tested?

Verified manually against Spark 2.4.0 RC3

Questions:

Does the licenses files need update? Yes
Is there breaking changes for older versions? No
Does this needs documentation? No

HyukjinKwon · 2018-10-17T15:01:35Z

This is a WIP. We should wait for Spark 2.4.0.

cc @zjffdu and @felixcheung

zjffdu · 2018-10-17T15:02:33Z

Thanks @HyukjinKwon Have you checked this PR (#3034) for supporting scala 2.12

HyukjinKwon · 2018-10-17T15:06:33Z

oops. I haven't. Will check that too while I am here.

BTW, my understanding is that we need this one as well since Spark still can be compiled against Scala 2.11.x, am I in the right way?

zjffdu · 2018-10-18T06:11:08Z

Yes， we need to support scala 2.11 for spark 2.4 first.
And please also update travis.yml to build it with spark-2.4 profile

HyukjinKwon · 2018-10-18T06:46:39Z

spark/pom.xml

@@ -192,6 +192,15 @@

    <profiles>

+        <profile>
+            <id>spark-2.4</id>


Hm, I thought this profile is not meant to be used for building. I referred #2880.

BTW, I am verifying this in my personal Travis CI as well (https://travis-ci.org/HyukjinKwon/zeppelin/builds/442994923)

This is not necessary for building, but it is necessary for running unit test against spark 2.4 in travis

@zjffdu, do you mind what travis update I should do please? Should I build this against spark-2.2, make it download Spark 2.4.0 and test it after setting SPARK_HOME like I did?

@HyukjinKwon Could you update .travis.yml add test matrix for spark 2.4 ?

I tried but looks I don't know how to .. 2.3 looks not being tested as well..

felixcheung · 2018-10-20T06:40:56Z

is this going to break, say spark 2.3 with scala 2.11.8?

HyukjinKwon · 2018-10-20T09:50:26Z

Nope, it will work for both 2.11.8 and 2.11.12. I manually checked. This change only uses the methods existing in both 2.11.8 and 2.11.12 at Scala.

ajaygk95 · 2018-10-29T07:41:04Z

Hello,

I was trying to run zeppelin with spark 2.4 and I have pulled your code.
While building zeppelin I'm facing the below issue,

[ERROR] /home/cloud-user/ajay/code/csf-cc-zeppelin-k8szep/spark/scala-2.11/src/main/scala/org/apache/zeppelin/spark/SparkScala211Interpreter.scala:37: error: class SparkScala211Interpreter needs to be abstract, since method interpret in class BaseSparkScalaInterpreter of type (code: String, context: org.apache.zeppelin.interpreter.InterpreterContext)org.apache.zeppelin.interpreter.InterpreterResult is not defined
[ERROR] class SparkScala211Interpreter(override val conf: SparkConf,
[ERROR]       ^
[ERROR] one error found
[INFO] ------------------------------------------------------------------------

Can you please help.
Thanks

HyukjinKwon · 2018-10-29T10:01:20Z

Does that happen only with this code changes? The change here does not touch signature at class SparkScala211Interpreter( and the error message looks pretty unrelated. The whole change proposed here does not change any signature.

HyukjinKwon · 2018-10-29T10:05:24Z

The error message:

[ERROR] /home/cloud-user/ajay/code/csf-cc-zeppelin-k8szep/spark/scala-
2.11/src/main/scala/org/apache/zeppelin/spark/SparkScala211Interpreter.scala:37: error: class 
SparkScala211Interpreter needs to be abstract, since method interpret in class 
BaseSparkScalaInterpreter of type (code: String, context: 
org.apache.zeppelin.interpreter.InterpreterContext)org.apache.zeppelin.interpreter.InterpreterResult is 
not defined

complains there's no interpret method defined at BaseSparkScalaInterpreter.

ajaygk95 · 2018-10-29T10:07:55Z

Does that happen only with this code changes? The change here does not touch signature at class SparkScala211Interpreter( and the error message looks pretty unrelated. The whole change proposed here does not change any signature.

Yes. With the original 0.8.0 source code I'm able to build.

ajaygk95 · 2018-10-29T10:14:42Z

The error message:

[ERROR] /home/cloud-user/ajay/code/csf-cc-zeppelin-k8szep/spark/scala-
2.11/src/main/scala/org/apache/zeppelin/spark/SparkScala211Interpreter.scala:37: error: class 
SparkScala211Interpreter needs to be abstract, since method interpret in class 
BaseSparkScalaInterpreter of type (code: String, context: 
org.apache.zeppelin.interpreter.InterpreterContext)org.apache.zeppelin.interpreter.InterpreterResult is 
not defined

complains there's no interpret method defined at BaseSparkScalaInterpreter.

Thanks for the reply.
I found the issue. I was using zeppelin 0.8.0 source code where "interpret" method was overriden. I think in the current branch this method is not present.

HyukjinKwon · 2018-10-29T10:43:34Z

It should be usable if the changes is cherry-picked properly. This PR basically just replace one line:

https://github.com/apache/zeppelin/blob/v0.8.0/spark/scala-2.11/src/main/scala/org/apache/zeppelin/spark/SparkScala211Interpreter.scala#L84

to a private function.

HyukjinKwon · 2018-10-31T05:23:12Z

BTW, I tested this via my Travis CI - https://travis-ci.org/HyukjinKwon/zeppelin/builds/448215776. Tests seems got passed.

HyukjinKwon · 2018-10-31T05:23:28Z

I also locally tested this patch against Spark RC5.

zjffdu · 2018-10-31T05:33:26Z

Awesome @HyukjinKwon Let's wait for spark 2.4 release

ajaygk95 · 2018-10-31T09:41:48Z

picked

I was able to get this working by doing cherry-picking. We needed some changes in our environment not related to zeppelin, but for the usage of spark-images.
Thanks for the help.

HyukjinKwon · 2018-10-31T10:12:22Z

Thanks for confirmation.

HyukjinKwon · 2018-11-08T11:44:58Z

Hey all ~ could this get in by any chance maybe?

dongjoon-hyun · 2018-11-08T18:13:57Z

Hi, All. It's announced finally.
http://spark.apache.org/news/spark-2-4-0-released.html

zjffdu · 2018-11-09T01:49:16Z

Thanks for everyone, the only remaining thing is to update .travis.yml to make sure the test pass again spark 2.4

zjffdu · 2018-11-09T09:45:03Z

@HyukjinKwon I created one PR for you to add test for spark 2.4, would mind to merge that ? HyukjinKwon#1

spark/pom.xml

zjffdu · 2018-11-11T09:32:50Z

CI is passed, will merge it if no more comments

antonkulaga · 2018-11-12T21:44:56Z

Looking forward to seeing it merged and Zeppelin 0.9.0 released, Spark 2.4 fixes some very nasty bugs

dongjoon-hyun · 2018-11-13T01:59:23Z

Thank you, @HyukjinKwon, @zjffdu and ALL!

HyukjinKwon · 2018-11-13T02:05:59Z

Thank you all!!

Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418). There are two problems for this upgrade at Zeppelin side: 1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible. See: - https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala - https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection. 2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit: ``` Caused by: java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139) ``` To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`. [Improvement] * [x] - Wait until Spark 2.4.0 is officially released. * https://issues.apache.org/jira/browse/ZEPPELIN-3810 Verified manually against Spark 2.4.0 RC3 * Does the licenses files need update? Yes * Is there breaking changes for older versions? No * Does this needs documentation? No Author: hyukjinkwon <gurwls223@apache.org> Author: Hyukjin Kwon <gurwls223@apache.org> Author: Jeff Zhang <zjffdu@gmail.com> Closes apache#3206 from HyukjinKwon/ZEPPELIN-3810 and squashes the following commits: c2456c9 [Hyukjin Kwon] Py4J 0.10.6 to 0.10.7 573f07d [Jeff Zhang] add test for spark 2.4 (#1) 9ac1797 [hyukjinkwon] Support Spark 2.4 (cherry picked from commit 4f73272)

alonshoham · 2018-11-18T11:18:01Z

Hello,
Is this PR going to be officially released soon as part of zeppelin 0.9.0?
Thank you

zjffdu · 2018-11-18T13:10:08Z

It is merged to branch-0.8 and master branch. So yes it will be released in zeppelin 0.9.0, but I believe 0.8.1 will be released before zeppelin 0.9.0

alonshoham · 2018-11-18T13:12:22Z

thank you for the quick response! does 0.8.1 have a release date? thanks again בתאריך יום א׳, 18 בנוב׳ 2018, 15:10, מאת Jeff Zhang < notifications@github.com>:

…

It is merged to branch-0.8 and master branch. So yes it will be released in zeppelin 0.9.0, but I believe 0.8.1 will be released before zeppelin 0.9.0 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3206 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AP-5fNF0kNk3qGODsqniD9sa1TDGaFPvks5uwVxGgaJpZM4XkINL> .

zjffdu · 2018-11-18T13:17:48Z

I will try to do it by the end of 2018

alonshoham · 2018-11-18T13:19:03Z

Thank you very much! בתאריך יום א׳, 18 בנוב׳ 2018, 15:17, מאת Jeff Zhang < notifications@github.com>:

…

I will try to do it by the end of 2018 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3206 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AP-5fC-vkwaAJVD2G81Bq8JBKLaS1PQdks5uwV4GgaJpZM4XkINL> .

### What is this PR for? Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418). There are two problems for this upgrade at Zeppelin side: 1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible. See: - https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala - https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection. 2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit: ``` Caused by: java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139) ``` To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`. ### What type of PR is it? [Improvement] ### Todos * [x] - Wait until Spark 2.4.0 is officially released. ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3810 ### How should this be tested? Verified manually against Spark 2.4.0 RC3 ### Questions: * Does the licenses files need update? Yes * Is there breaking changes for older versions? No * Does this needs documentation? No Author: hyukjinkwon <gurwls223@apache.org> Author: Hyukjin Kwon <gurwls223@apache.org> Author: Jeff Zhang <zjffdu@gmail.com> Closes apache#3206 from HyukjinKwon/ZEPPELIN-3810 and squashes the following commits: c2456c9 [Hyukjin Kwon] Py4J 0.10.6 to 0.10.7 573f07d [Jeff Zhang] add test for spark 2.4 (#1) 9ac1797 [hyukjinkwon] Support Spark 2.4

NicolasDutronc · 2018-11-21T14:01:55Z

Hi all, I'm using Zeppelin with the official Docker image and I'm still getting java.lang.NoSuchMethodException: scala.tools.nsc.interpreter.ILoop.scala$tools$nsc$interpreter$ILoop$$loopPostInit() when using Spark 2.4.0.
I've seen that on docker store the image has been updated for the last time 5 months ago. Could you please update it ? Or should I build the docker branch of the repo ?
Thank you

HyukjinKwon · 2018-11-21T14:31:45Z

This fix is not released yet. This PR exactly fixes the problem you faced. This fix will be available in the next release of Zeppelin.

maziyarpanahi · 2019-01-07T16:20:12Z

Happy new year guys! I've waited for the release of 0.8.1 but I made a mistake and upgraded my Cloudera to 6.1 which it comes with Hadoop 3.0 and Spark 2.4! I am trying to compile it myself, but I have a question. I have cloned the repo and checkout to branch-0.8 which I can see it has this already merged into. I have built it as follow but still tells me Spark 2.4 is not supported:

./dev/change_scala_version.sh 2.11
 mvn package -DskipTests -Pspark-2.0 -Phadoop-2.4 -Pscala-2.11

Shall I use -Pspark-2.4 or -Pspark-2.0? Or this is not related to my issue?
Thank you!

maziyarpanahi · 2019-01-07T17:15:07Z

To answer my own basic question! Yes, it has to be -Pspark-2.4! But it didn't work on my macOS:

[WARNING] The requested profile "spark-2.4" could not be activated because it does not exist.
[WARNING] The requested profile "hadoop-3.0" could not be activated because it does not exist.

I did it again on my Ubuntu which is part of Cloudera cluster and it went perfectly. (obviously, Hadoop 3.0 still not available in the profiles, but it didn't change anything)
Now I have:

spark.version
res0: String = 2.4.0-cdh6.1.0

Can't wait for this release and 0.9.0! Great work and thank you.

maziyarpanahi · 2019-01-08T14:35:21Z

I have some bad news related to Spark 2.4 and commons-lang3 compatibility:

https://issues.apache.org/jira/browse/ZEPPELIN-3939

Unfortunately, it is not possible to test everything in a new Spark/Hadoop/CDH at once. I successfully ran an ML pipeline, read from parquet, but after a day I realized I can't read JSON and CSV files.

maziyarpanahi · 2019-01-18T17:03:54Z

@HyukjinKwon @zjffdu @Leemoonsoo is it possible to shade Spark and Hadoop related dependencies when there is already Spark and Hadoop exist? I was thinking to avoid situations like this, to have a type of build where it shades all the deps for users with Spark and Hadoop.
Do you think this could work?

Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418). There are two problems for this upgrade at Zeppelin side: 1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible. See: - https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala - https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection. 2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit: ``` Caused by: java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139) ``` To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`. [Improvement] * [x] - Wait until Spark 2.4.0 is officially released. * https://issues.apache.org/jira/browse/ZEPPELIN-3810 Verified manually against Spark 2.4.0 RC3 * Does the licenses files need update? Yes * Is there breaking changes for older versions? No * Does this needs documentation? No Author: hyukjinkwon <gurwls223@apache.org> Author: Hyukjin Kwon <gurwls223@apache.org> Author: Jeff Zhang <zjffdu@gmail.com> Closes apache#3206 from HyukjinKwon/ZEPPELIN-3810 and squashes the following commits: c2456c9 [Hyukjin Kwon] Py4J 0.10.6 to 0.10.7 573f07d [Jeff Zhang] add test for spark 2.4 (#1) 9ac1797 [hyukjinkwon] Support Spark 2.4 (cherry picked from commit 4f73272) Change-Id: I05583aff76758936ccd84fa3820fa1e733d4416f

Nourimane · 2019-06-01T16:20:03Z

Hello everyone
I have a problem with zeppelin; the notebook does not run and displays the following message
java.lang.NoSuchMethodException: scala.tools.nsc.interpreter.ILoop.scala$tools$nsc$interpreter$ILoop$$loopPostInit...

and in log zeppelin-interpreter-spark-hduser-ubuntu.log

INFO [2019-06-01 14:17:09,785] ({pool-2-thread-2} NewSparkInterpreter.java[open]:83) - Using Scala Version: 2.11
ERROR [2019-06-01 14:17:33,290] ({pool-2-thread-2} NewSparkInterpreter.java[open]:124) - Fail to open SparkInterpreter
ERROR [2019-06-01 14:17:33,290] ({pool-2-thread-2} Job.java[run]:190) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: Fail to open SparkInterpreter
at org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:125)
at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)

version scala 2.11.12
version zeppelin 0.8.0
version spark 2.4.1

can someone help me plz ?

HyukjinKwon closed this Oct 18, 2018

HyukjinKwon reopened this Oct 18, 2018

HyukjinKwon closed this Oct 18, 2018

HyukjinKwon reopened this Oct 18, 2018

HyukjinKwon closed this Oct 18, 2018

HyukjinKwon reopened this Oct 18, 2018

HyukjinKwon commented Oct 18, 2018

View reviewed changes

HyukjinKwon force-pushed the ZEPPELIN-3810 branch from e2d224a to 4b9ca42 Compare October 18, 2018 08:19

Support Spark 2.4

9ac1797

HyukjinKwon force-pushed the ZEPPELIN-3810 branch from 4b9ca42 to 9ac1797 Compare October 30, 2018 08:47

HyukjinKwon closed this Oct 31, 2018

HyukjinKwon reopened this Oct 31, 2018

HyukjinKwon mentioned this pull request Nov 1, 2018

[SPARK-25906][SHELL] Documents '-I' option (from Scala REPL) in spark-shell apache/spark#22919

Closed

add test for spark 2.4 (#1)

573f07d

zjffdu reviewed Nov 9, 2018

View reviewed changes

spark/pom.xml Outdated Show resolved Hide resolved

Py4J 0.10.6 to 0.10.7

c2456c9

asfgit closed this in 4f73272 Nov 13, 2018

HyukjinKwon mentioned this pull request Dec 5, 2018

[ZEPPELIN-3551] Upgrade Scala to 2.11.12 #3253

Open

1 task

[ZEPPELIN-3810] Support Spark 2.4 #3206

[ZEPPELIN-3810] Support Spark 2.4 #3206

Conversation

HyukjinKwon commented Oct 17, 2018 • edited Loading

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Questions:

HyukjinKwon commented Oct 17, 2018

zjffdu commented Oct 17, 2018

HyukjinKwon commented Oct 17, 2018

zjffdu commented Oct 18, 2018

HyukjinKwon Oct 18, 2018

Choose a reason for hiding this comment

HyukjinKwon Oct 18, 2018 • edited Loading

Choose a reason for hiding this comment

zjffdu Oct 18, 2018

Choose a reason for hiding this comment

HyukjinKwon Oct 19, 2018

Choose a reason for hiding this comment

zjffdu Nov 8, 2018 • edited Loading

Choose a reason for hiding this comment

HyukjinKwon Nov 8, 2018 • edited Loading

Choose a reason for hiding this comment

felixcheung commented Oct 20, 2018

HyukjinKwon commented Oct 20, 2018

ajaygk95 commented Oct 29, 2018 • edited Loading

HyukjinKwon commented Oct 29, 2018

HyukjinKwon commented Oct 29, 2018 • edited Loading

ajaygk95 commented Oct 29, 2018

ajaygk95 commented Oct 29, 2018

HyukjinKwon commented Oct 29, 2018

HyukjinKwon commented Oct 31, 2018

HyukjinKwon commented Oct 31, 2018 • edited Loading

zjffdu commented Oct 31, 2018

ajaygk95 commented Oct 31, 2018

HyukjinKwon commented Oct 31, 2018

HyukjinKwon commented Nov 8, 2018 • edited Loading

dongjoon-hyun commented Nov 8, 2018

zjffdu commented Nov 9, 2018

zjffdu commented Nov 9, 2018

zjffdu commented Nov 11, 2018

antonkulaga commented Nov 12, 2018

dongjoon-hyun commented Nov 13, 2018 • edited Loading

HyukjinKwon commented Nov 13, 2018

alonshoham commented Nov 18, 2018

zjffdu commented Nov 18, 2018

alonshoham commented Nov 18, 2018 via email

zjffdu commented Nov 18, 2018

alonshoham commented Nov 18, 2018 via email

NicolasDutronc commented Nov 21, 2018

HyukjinKwon commented Nov 21, 2018

maziyarpanahi commented Jan 7, 2019

maziyarpanahi commented Jan 7, 2019

maziyarpanahi commented Jan 8, 2019

maziyarpanahi commented Jan 18, 2019 • edited Loading

Nourimane commented Jun 1, 2019

HyukjinKwon commented Oct 17, 2018 •

edited

Loading

HyukjinKwon Oct 18, 2018 •

edited

Loading

zjffdu Nov 8, 2018 •

edited

Loading

HyukjinKwon Nov 8, 2018 •

edited

Loading

ajaygk95 commented Oct 29, 2018 •

edited

Loading

HyukjinKwon commented Oct 29, 2018 •

edited

Loading

HyukjinKwon commented Oct 31, 2018 •

edited

Loading

HyukjinKwon commented Nov 8, 2018 •

edited

Loading

dongjoon-hyun commented Nov 13, 2018 •

edited

Loading

maziyarpanahi commented Jan 18, 2019 •

edited

Loading