Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ZEPPELIN-3810] Support Spark 2.4 #3206

Closed
wants to merge 3 commits into from

Conversation

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Oct 17, 2018

What is this PR for?

Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418).

There are two problems for this upgrade at Zeppelin side:

1.. Some methods that are used in private by reflection, for instance, loopPostInit became inaccessible.

See:

To work around this, I manually ported loopPostInit at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new loopPostInit by reflection.

2.. Upgrade from 2.11.8 to 2.11.12 requires jline.version upgrade. Otherwise, we will hit:

Caused by: java.lang.NoSuchMethodError: 
jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
  at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)

To work around this, I tweaked this by upgrading jline from 2.12.1 to 2.14.3.

What type of PR is it?

[Improvement]

Todos

  • - Wait until Spark 2.4.0 is officially released.

What is the Jira issue?

How should this be tested?

Verified manually against Spark 2.4.0 RC3

Questions:

  • Does the licenses files need update? Yes
  • Is there breaking changes for older versions? No
  • Does this needs documentation? No
@HyukjinKwon

This comment has been minimized.

Copy link
Member Author

HyukjinKwon commented Oct 17, 2018

This is a WIP. We should wait for Spark 2.4.0.

cc @zjffdu and @felixcheung

@zjffdu

This comment has been minimized.

Copy link
Contributor

zjffdu commented Oct 17, 2018

Thanks @HyukjinKwon Have you checked this PR (#3034) for supporting scala 2.12

@HyukjinKwon

This comment has been minimized.

Copy link
Member Author

HyukjinKwon commented Oct 17, 2018

oops. I haven't. Will check that too while I am here.

BTW, my understanding is that we need this one as well since Spark still can be compiled against Scala 2.11.x, am I in the right way?

@zjffdu

This comment has been minimized.

Copy link
Contributor

zjffdu commented Oct 18, 2018

Yes, we need to support scala 2.11 for spark 2.4 first.
And please also update travis.yml to build it with spark-2.4 profile

@@ -192,6 +192,15 @@

<profiles>

<profile>
<id>spark-2.4</id>

This comment has been minimized.

Copy link
@HyukjinKwon

HyukjinKwon Oct 18, 2018

Author Member

Hm, I thought this profile is not meant to be used for building. I referred #2880.

This comment has been minimized.

Copy link
@HyukjinKwon

HyukjinKwon Oct 18, 2018

Author Member

BTW, I am verifying this in my personal Travis CI as well (https://travis-ci.org/HyukjinKwon/zeppelin/builds/442994923)

This comment has been minimized.

Copy link
@zjffdu

zjffdu Oct 18, 2018

Contributor

This is not necessary for building, but it is necessary for running unit test against spark 2.4 in travis

This comment has been minimized.

Copy link
@HyukjinKwon

HyukjinKwon Oct 19, 2018

Author Member

@zjffdu, do you mind what travis update I should do please? Should I build this against spark-2.2, make it download Spark 2.4.0 and test it after setting SPARK_HOME like I did?

This comment has been minimized.

Copy link
@zjffdu

zjffdu Nov 8, 2018

Contributor

@HyukjinKwon Could you update .travis.yml add test matrix for spark 2.4 ?

This comment has been minimized.

Copy link
@HyukjinKwon

HyukjinKwon Nov 8, 2018

Author Member

I tried but looks I don't know how to .. 2.3 looks not being tested as well..

@HyukjinKwon HyukjinKwon force-pushed the HyukjinKwon:ZEPPELIN-3810 branch from e2d224a to 4b9ca42 Oct 18, 2018
@felixcheung

This comment has been minimized.

Copy link
Member

felixcheung commented Oct 20, 2018

is this going to break, say spark 2.3 with scala 2.11.8?

@HyukjinKwon

This comment has been minimized.

Copy link
Member Author

HyukjinKwon commented Oct 20, 2018

Nope, it will work for both 2.11.8 and 2.11.12. I manually checked. This change only uses the methods existing in both 2.11.8 and 2.11.12 at Scala.

@ajaygk95

This comment has been minimized.

Copy link

ajaygk95 commented Oct 29, 2018

Hello,

I was trying to run zeppelin with spark 2.4 and I have pulled your code.
While building zeppelin I'm facing the below issue,

[ERROR] /home/cloud-user/ajay/code/csf-cc-zeppelin-k8szep/spark/scala-2.11/src/main/scala/org/apache/zeppelin/spark/SparkScala211Interpreter.scala:37: error: class SparkScala211Interpreter needs to be abstract, since method interpret in class BaseSparkScalaInterpreter of type (code: String, context: org.apache.zeppelin.interpreter.InterpreterContext)org.apache.zeppelin.interpreter.InterpreterResult is not defined
[ERROR] class SparkScala211Interpreter(override val conf: SparkConf,
[ERROR]       ^
[ERROR] one error found
[INFO] ------------------------------------------------------------------------

Can you please help.
Thanks

@HyukjinKwon

This comment has been minimized.

Copy link
Member Author

HyukjinKwon commented Oct 29, 2018

Does that happen only with this code changes? The change here does not touch signature at class SparkScala211Interpreter( and the error message looks pretty unrelated. The whole change proposed here does not change any signature.

@HyukjinKwon

This comment has been minimized.

Copy link
Member Author

HyukjinKwon commented Oct 29, 2018

The error message:

[ERROR] /home/cloud-user/ajay/code/csf-cc-zeppelin-k8szep/spark/scala-
2.11/src/main/scala/org/apache/zeppelin/spark/SparkScala211Interpreter.scala:37: error: class 
SparkScala211Interpreter needs to be abstract, since method interpret in class 
BaseSparkScalaInterpreter of type (code: String, context: 
org.apache.zeppelin.interpreter.InterpreterContext)org.apache.zeppelin.interpreter.InterpreterResult is 
not defined

complains there's no interpret method defined at BaseSparkScalaInterpreter.

@ajaygk95

This comment has been minimized.

Copy link

ajaygk95 commented Oct 29, 2018

Does that happen only with this code changes? The change here does not touch signature at class SparkScala211Interpreter( and the error message looks pretty unrelated. The whole change proposed here does not change any signature.

Yes. With the original 0.8.0 source code I'm able to build.

@ajaygk95

This comment has been minimized.

Copy link

ajaygk95 commented Oct 29, 2018

The error message:

[ERROR] /home/cloud-user/ajay/code/csf-cc-zeppelin-k8szep/spark/scala-
2.11/src/main/scala/org/apache/zeppelin/spark/SparkScala211Interpreter.scala:37: error: class 
SparkScala211Interpreter needs to be abstract, since method interpret in class 
BaseSparkScalaInterpreter of type (code: String, context: 
org.apache.zeppelin.interpreter.InterpreterContext)org.apache.zeppelin.interpreter.InterpreterResult is 
not defined

complains there's no interpret method defined at BaseSparkScalaInterpreter.

Thanks for the reply.
I found the issue. I was using zeppelin 0.8.0 source code where "interpret" method was overriden. I think in the current branch this method is not present.

@HyukjinKwon

This comment has been minimized.

Copy link
Member Author

HyukjinKwon commented Oct 29, 2018

It should be usable if the changes is cherry-picked properly. This PR basically just replace one line:

https://github.com/apache/zeppelin/blob/v0.8.0/spark/scala-2.11/src/main/scala/org/apache/zeppelin/spark/SparkScala211Interpreter.scala#L84

to a private function.

@HyukjinKwon HyukjinKwon force-pushed the HyukjinKwon:ZEPPELIN-3810 branch from 4b9ca42 to 9ac1797 Oct 30, 2018
@HyukjinKwon HyukjinKwon reopened this Oct 31, 2018
@HyukjinKwon

This comment has been minimized.

Copy link
Member Author

HyukjinKwon commented Oct 31, 2018

BTW, I tested this via my Travis CI - https://travis-ci.org/HyukjinKwon/zeppelin/builds/448215776. Tests seems got passed.

@HyukjinKwon

This comment has been minimized.

Copy link
Member Author

HyukjinKwon commented Oct 31, 2018

I also locally tested this patch against Spark RC5.

@zjffdu

This comment has been minimized.

Copy link
Contributor

zjffdu commented Oct 31, 2018

Awesome @HyukjinKwon Let's wait for spark 2.4 release

@ajaygk95

This comment has been minimized.

Copy link

ajaygk95 commented Oct 31, 2018

picked

I was able to get this working by doing cherry-picking. We needed some changes in our environment not related to zeppelin, but for the usage of spark-images.
Thanks for the help.

@HyukjinKwon

This comment has been minimized.

Copy link
Member Author

HyukjinKwon commented Oct 31, 2018

Thanks for confirmation.

@dongjoon-hyun

This comment has been minimized.

Copy link
Member

dongjoon-hyun commented Nov 8, 2018

@zjffdu

This comment has been minimized.

Copy link
Contributor

zjffdu commented Nov 9, 2018

Thanks for everyone, the only remaining thing is to update .travis.yml to make sure the test pass again spark 2.4

@zjffdu

This comment has been minimized.

Copy link
Contributor

zjffdu commented Nov 9, 2018

@HyukjinKwon I created one PR for you to add test for spark 2.4, would mind to merge that ? HyukjinKwon#1

spark/pom.xml Outdated Show resolved Hide resolved
@zjffdu

This comment has been minimized.

Copy link
Contributor

zjffdu commented Nov 11, 2018

CI is passed, will merge it if no more comments

@antonkulaga

This comment has been minimized.

Copy link

antonkulaga commented Nov 12, 2018

Looking forward to seeing it merged and Zeppelin 0.9.0 released, Spark 2.4 fixes some very nasty bugs

@asfgit asfgit closed this in 4f73272 Nov 13, 2018
@dongjoon-hyun

This comment has been minimized.

Copy link
Member

dongjoon-hyun commented Nov 13, 2018

Thank you, @HyukjinKwon, @zjffdu and ALL!

@HyukjinKwon

This comment has been minimized.

Copy link
Member Author

HyukjinKwon commented Nov 13, 2018

Thank you all!!

zjffdu added a commit to zjffdu/zeppelin that referenced this pull request Nov 13, 2018
Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418).

There are two problems for this upgrade at Zeppelin side:

1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible.

See:
 - https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
 - https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala

To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.

2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError:
jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
  at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```

To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`.

[Improvement]

* [x] - Wait until Spark 2.4.0 is officially released.

* https://issues.apache.org/jira/browse/ZEPPELIN-3810

Verified manually against Spark 2.4.0 RC3

* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: hyukjinkwon <gurwls223@apache.org>
Author: Hyukjin Kwon <gurwls223@apache.org>
Author: Jeff Zhang <zjffdu@gmail.com>

Closes apache#3206 from HyukjinKwon/ZEPPELIN-3810 and squashes the following commits:

c2456c9 [Hyukjin Kwon] Py4J 0.10.6 to 0.10.7
573f07d [Jeff Zhang] add test for spark 2.4 (#1)
9ac1797 [hyukjinkwon] Support Spark 2.4

(cherry picked from commit 4f73272)
zjffdu added a commit to zjffdu/zeppelin that referenced this pull request Nov 13, 2018
Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418).

There are two problems for this upgrade at Zeppelin side:

1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible.

See:
 - https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
 - https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala

To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.

2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError:
jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
  at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```

To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`.

[Improvement]

* [x] - Wait until Spark 2.4.0 is officially released.

* https://issues.apache.org/jira/browse/ZEPPELIN-3810

Verified manually against Spark 2.4.0 RC3

* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: hyukjinkwon <gurwls223@apache.org>
Author: Hyukjin Kwon <gurwls223@apache.org>
Author: Jeff Zhang <zjffdu@gmail.com>

Closes apache#3206 from HyukjinKwon/ZEPPELIN-3810 and squashes the following commits:

c2456c9 [Hyukjin Kwon] Py4J 0.10.6 to 0.10.7
573f07d [Jeff Zhang] add test for spark 2.4 (#1)
9ac1797 [hyukjinkwon] Support Spark 2.4

(cherry picked from commit 4f73272)
zjffdu added a commit to zjffdu/zeppelin that referenced this pull request Nov 13, 2018
Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418).

There are two problems for this upgrade at Zeppelin side:

1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible.

See:
 - https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
 - https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala

To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.

2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError:
jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
  at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```

To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`.

[Improvement]

* [x] - Wait until Spark 2.4.0 is officially released.

* https://issues.apache.org/jira/browse/ZEPPELIN-3810

Verified manually against Spark 2.4.0 RC3

* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: hyukjinkwon <gurwls223@apache.org>
Author: Hyukjin Kwon <gurwls223@apache.org>
Author: Jeff Zhang <zjffdu@gmail.com>

Closes apache#3206 from HyukjinKwon/ZEPPELIN-3810 and squashes the following commits:

c2456c9 [Hyukjin Kwon] Py4J 0.10.6 to 0.10.7
573f07d [Jeff Zhang] add test for spark 2.4 (#1)
9ac1797 [hyukjinkwon] Support Spark 2.4

(cherry picked from commit 4f73272)
@alonshoham

This comment has been minimized.

Copy link

alonshoham commented Nov 18, 2018

Hello,
Is this PR going to be officially released soon as part of zeppelin 0.9.0?
Thank you

@zjffdu

This comment has been minimized.

Copy link
Contributor

zjffdu commented Nov 18, 2018

It is merged to branch-0.8 and master branch. So yes it will be released in zeppelin 0.9.0, but I believe 0.8.1 will be released before zeppelin 0.9.0

@alonshoham

This comment has been minimized.

Copy link

alonshoham commented Nov 18, 2018

@zjffdu

This comment has been minimized.

Copy link
Contributor

zjffdu commented Nov 18, 2018

I will try to do it by the end of 2018

@alonshoham

This comment has been minimized.

Copy link

alonshoham commented Nov 18, 2018

Leemoonsoo added a commit to Leemoonsoo/zeppelin that referenced this pull request Nov 18, 2018
### What is this PR for?

Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418).

There are two problems for this upgrade at Zeppelin side:

1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible.

See:
 - https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
 - https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala

To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.

2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError:
jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
  at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```

To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`.

### What type of PR is it?
[Improvement]

### Todos
* [x] - Wait until Spark 2.4.0 is officially released.

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3810

### How should this be tested?

Verified manually against Spark 2.4.0 RC3

### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: hyukjinkwon <gurwls223@apache.org>
Author: Hyukjin Kwon <gurwls223@apache.org>
Author: Jeff Zhang <zjffdu@gmail.com>

Closes apache#3206 from HyukjinKwon/ZEPPELIN-3810 and squashes the following commits:

c2456c9 [Hyukjin Kwon] Py4J 0.10.6 to 0.10.7
573f07d [Jeff Zhang] add test for spark 2.4 (#1)
9ac1797 [hyukjinkwon] Support Spark 2.4
@NicolasDutronc

This comment has been minimized.

Copy link

NicolasDutronc commented Nov 21, 2018

Hi all, I'm using Zeppelin with the official Docker image and I'm still getting java.lang.NoSuchMethodException: scala.tools.nsc.interpreter.ILoop.scala$tools$nsc$interpreter$ILoop$$loopPostInit() when using Spark 2.4.0.
I've seen that on docker store the image has been updated for the last time 5 months ago. Could you please update it ? Or should I build the docker branch of the repo ?
Thank you

@HyukjinKwon

This comment has been minimized.

Copy link
Member Author

HyukjinKwon commented Nov 21, 2018

This fix is not released yet. This PR exactly fixes the problem you faced. This fix will be available in the next release of Zeppelin.

baluchicken added a commit to banzaicloud/zeppelin that referenced this pull request Nov 27, 2018
Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418).

There are two problems for this upgrade at Zeppelin side:

1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible.

See:
 - https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
 - https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala

To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.

2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError:
jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
  at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```

To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`.

[Improvement]

* [x] - Wait until Spark 2.4.0 is officially released.

* https://issues.apache.org/jira/browse/ZEPPELIN-3810

Verified manually against Spark 2.4.0 RC3

* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: hyukjinkwon <gurwls223@apache.org>
Author: Hyukjin Kwon <gurwls223@apache.org>
Author: Jeff Zhang <zjffdu@gmail.com>

Closes apache#3206 from HyukjinKwon/ZEPPELIN-3810 and squashes the following commits:

c2456c9 [Hyukjin Kwon] Py4J 0.10.6 to 0.10.7
573f07d [Jeff Zhang] add test for spark 2.4 (apache#1)
9ac1797 [hyukjinkwon] Support Spark 2.4

(cherry picked from commit 4f73272)
@HyukjinKwon HyukjinKwon mentioned this pull request Dec 5, 2018
1 of 1 task complete
@maziyarpanahi

This comment has been minimized.

Copy link
Contributor

maziyarpanahi commented Jan 7, 2019

Happy new year guys! I've waited for the release of 0.8.1 but I made a mistake and upgraded my Cloudera to 6.1 which it comes with Hadoop 3.0 and Spark 2.4! I am trying to compile it myself, but I have a question. I have cloned the repo and checkout to branch-0.8 which I can see it has this already merged into. I have built it as follow but still tells me Spark 2.4 is not supported:

./dev/change_scala_version.sh 2.11
 mvn package -DskipTests -Pspark-2.0 -Phadoop-2.4 -Pscala-2.11

Shall I use -Pspark-2.4 or -Pspark-2.0? Or this is not related to my issue?
Thank you!

@maziyarpanahi

This comment has been minimized.

Copy link
Contributor

maziyarpanahi commented Jan 7, 2019

To answer my own basic question! Yes, it has to be -Pspark-2.4! But it didn't work on my macOS:

[WARNING] The requested profile "spark-2.4" could not be activated because it does not exist.
[WARNING] The requested profile "hadoop-3.0" could not be activated because it does not exist.

I did it again on my Ubuntu which is part of Cloudera cluster and it went perfectly. (obviously, Hadoop 3.0 still not available in the profiles, but it didn't change anything)
Now I have:

spark.version
res0: String = 2.4.0-cdh6.1.0

Can't wait for this release and 0.9.0! Great work and thank you.

@maziyarpanahi

This comment has been minimized.

Copy link
Contributor

maziyarpanahi commented Jan 8, 2019

I have some bad news related to Spark 2.4 and commons-lang3 compatibility:

https://issues.apache.org/jira/browse/ZEPPELIN-3939

Unfortunately, it is not possible to test everything in a new Spark/Hadoop/CDH at once. I successfully ran an ML pipeline, read from parquet, but after a day I realized I can't read JSON and CSV files.

@maziyarpanahi

This comment has been minimized.

Copy link
Contributor

maziyarpanahi commented Jan 18, 2019

@HyukjinKwon @zjffdu @Leemoonsoo is it possible to shade Spark and Hadoop related dependencies when there is already Spark and Hadoop exist? I was thinking to avoid situations like this, to have a type of build where it shades all the deps for users with Spark and Hadoop.
Do you think this could work?

prabhjyotsingh pushed a commit to prabhjyotsingh/zeppelin that referenced this pull request Apr 29, 2019
Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418).

There are two problems for this upgrade at Zeppelin side:

1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible.

See:
 - https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
 - https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala

To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.

2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError:
jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
  at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```

To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`.

[Improvement]

* [x] - Wait until Spark 2.4.0 is officially released.

* https://issues.apache.org/jira/browse/ZEPPELIN-3810

Verified manually against Spark 2.4.0 RC3

* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: hyukjinkwon <gurwls223@apache.org>
Author: Hyukjin Kwon <gurwls223@apache.org>
Author: Jeff Zhang <zjffdu@gmail.com>

Closes apache#3206 from HyukjinKwon/ZEPPELIN-3810 and squashes the following commits:

c2456c9 [Hyukjin Kwon] Py4J 0.10.6 to 0.10.7
573f07d [Jeff Zhang] add test for spark 2.4 (#1)
9ac1797 [hyukjinkwon] Support Spark 2.4

(cherry picked from commit 4f73272)

Change-Id: I05583aff76758936ccd84fa3820fa1e733d4416f
@Nourimane

This comment has been minimized.

Copy link

Nourimane commented Jun 1, 2019

Hello everyone
I have a problem with zeppelin; the notebook does not run and displays the following message
java.lang.NoSuchMethodException: scala.tools.nsc.interpreter.ILoop.scala$tools$nsc$interpreter$ILoop$$loopPostInit...

and in log zeppelin-interpreter-spark-hduser-ubuntu.log

INFO [2019-06-01 14:17:09,785] ({pool-2-thread-2} NewSparkInterpreter.java[open]:83) - Using Scala Version: 2.11
ERROR [2019-06-01 14:17:33,290] ({pool-2-thread-2} NewSparkInterpreter.java[open]:124) - Fail to open SparkInterpreter
ERROR [2019-06-01 14:17:33,290] ({pool-2-thread-2} Job.java[run]:190) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: Fail to open SparkInterpreter
at org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:125)
at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)

version scala 2.11.12
version zeppelin 0.8.0
version spark 2.4.1

can someone help me plz ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.