ZEPPELIN-3111. Refactor SparkInterpreter

[zjffdu]

What is this PR for?

This is for the refactoring of SparkInterpreter. See design doc. https://docs.google.com/document/d/1AfGg3aGXonDyri1jrP4MMFT4Y4j3wpN1t8kL-GAKSUc/edit?usp=sharing

What type of PR is it?

[Refactoring]

Todos

• - Task

What is the Jira issue?

• https://issues.apache.org/jira/browse/ZEPPELIN-3111

How should this be tested?

• Unit test is added.

Screenshots (if appropriate)

Questions:

• Does the licenses files need update? No
• Is there breaking changes for older versions? No
• Does this needs documentation? No

[zjffdu]

@Leemoonsoo @felixcheung @prabhjyotsingh @jongyoul Could you help review it ? Thanks

[Leemoonsoo]

in the binary package, how spark/scala-2.11 and spark/scala-2.10 modules will be packaged?

[zjffdu]

@Leemoonsoo binary package is the same as before, spark/scala-2.11 and spark/scala-2.10 will be packaged together into spark-interpreter-0.8.0-SNAPSHOT.jar which doesn't contain scala library. scala library is in the spark distribution. So SparkInterpreter will load the correct scala version of SparkInterpreter (SparkScala211Interpreter or SparkScala210Interpreter) based the scala version in your spark distribution (depends on SPARK_HOME you set).

[Leemoonsoo]

Thanks @zjffdu for explain. Looks good to me.

[zjffdu]

Thanks @Leemoonsoo , will merge it if no more comments

[alonshoham]

In this PR a change was made to bin/interpreter.sh script. In line 124 SPARK_APP_JAR="$(ls ${ZEPPELIN_HOME}/interpreter/spark/spark-interpreter*.jar)" used to be SPARK_APP_JAR="$(ls ${ZEPPELIN_HOME}/interpreter/spark/zeppelin-spark*.jar)"
This change was not updated in bin/interpreter.cmd, meaning that spark interpreter fails in windows.

[zjffdu]

Thanks @alonshoham feel free to create a PR for that.

[alonshoham]

Hi I have created a JIRA iisue and corrected this locally. I cannot push my local branch to github so I cannot create the PR. How do i complete this process?

…

On Thu, Sep 27, 2018 at 5:02 AM Jeff Zhang ***@***.***> wrote: Thanks @alonshoham <https://github.com/alonshoham> feel free to create a PR for that. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2709 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AP-5fK35EuONRQ5-yOJn0AzYecD8OxNqks5ufDGzgaJpZM4RG0hm> .

[zjffdu]

@alonshoham Sorry I didn't notice you created the ticket, I have fixed via this commit, 8b2f4a7

Could you help verify it ?

[alonshoham]

Your commit looks OK. Notice that I mentioned another problem in the ticket with interpreter.cmd

…

On Wed, Oct 3, 2018 at 9:55 AM Jeff Zhang ***@***.***> wrote: @alonshoham <https://github.com/alonshoham> Sorry I didn't notice you created the ticket, I have fixed via this commit, 8b2f4a7 <8b2f4a7> Could you help verify it ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2709 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AP-5fM7nTPBvjF0z939KUeSGV9T73Ui3ks5uhF90gaJpZM4RG0hm> .

[zjffdu]

Thanks @alonshoham Feel free to create a PR for that issue