[ZEPPELIN-4323] Kotlin support for Spark interpreter #3440

dkaznacheev · 2019-09-06T12:18:32Z

What is this PR for?

This PR aims to add Kotlin language support to Apache Zeppelin as a standalone interpreter as well as a part of Spark interpreter group.
For added features and details, see Jira issue: https://issues.apache.org/jira/browse/ZEPPELIN-4323

Feel free to ask any questions, I will be happy to answer!

What type of PR is it?

Feature

Todos

- Add support for Spark versions older than 2.4

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-4323

How should this be tested?

Unit tests are added to standalone Kotlin interpreter and to Spark interpreter.
Manually test by building Zepppelin under profile -Pspark2.4 and using %kotlin in standalone interpreter or %spark.kotlin in Spark interpreter

Screenshots (if appropriate)

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? Yes, and documentation is updated for Kotlin and Spark

…avoid confusion with ZeppelinContext

ileasile · 2019-11-08T14:03:30Z

spark/interpreter/src/main/java/org/apache/zeppelin/spark/KotlinSparkInterpreter.java

+
+  private List<String> sparkClasspath() {
+    String sparkJars = System.getProperty("spark.jars");
+    Pattern isKotlinJar = Pattern.compile("/kotlin-(runtime|stdlib|compiler|reflect)(-.*)?\\.jar");


May be this pattern simplified to /kotlin-[a-z]*(-.*)?\\.jar? I suppose we don't want to change this regex every time new jar is added.

ileasile · 2019-11-08T14:07:03Z

spark/interpreter/src/main/java/org/apache/zeppelin/spark/KotlinSparkInterpreter.java

+  }
+
+  private List<String> sparkClasspath() {
+    String sparkJars = System.getProperty("spark.jars");


Shouldn't this method be static?

ileasile · 2019-11-08T14:11:47Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/storage/LocalConfigStorage.java

@@ -128,7 +128,8 @@ static void atomicWriteToFile(String content, File file) throws IOException {
    }
    try {
      file.getParentFile().mkdirs();
-      Files.move(tempFile.toPath(), destinationFilePath,  StandardCopyOption.ATOMIC_MOVE);
+      Files.move(tempFile.toPath(), destinationFilePath,
+              StandardCopyOption.REPLACE_EXISTING); //StandardCopyOption.ATOMIC_MOVE);


Why has this changed?

Atomic move caused errors on Linux, I suggest asking khud about that

# Conflicts: # .travis.yml # LICENSE

zjffdu · 2019-11-13T01:51:35Z

@dkaznacheev Could you resolve the conflict ?

# Conflicts: # .travis.yml

ileasile · 2019-11-13T08:14:55Z

@zjffdu Conflict was resolved

zjffdu · 2019-11-13T10:03:10Z

@ileasile @dkaznacheev Could any of you paste the travis CI link ?

…terpreter

ileasile · 2019-11-13T14:07:30Z

@zjffdu Here is the link: https://travis-ci.org/ileasile/zeppelin/builds/611340288

zjffdu · 2019-11-14T05:30:33Z

Thanks @ileasile Could you rerun the failed test ?

zjffdu · 2019-11-18T12:47:09Z

Thanks @ileasile the PR LGTM except on minor thing.

Kotlin unit test is ignored for now, you need to add org.apache.zeppelin.kotlin.* in one of the build, such as this one for spark24 https://github.com/apache/zeppelin/blob/master/.travis.yml#L115

We do the same thing for python interpreter, https://github.com/apache/zeppelin/blob/master/.travis.yml#L123

ileasile · 2019-11-18T16:10:42Z

@zjffdu This issue is fixed, new build is here: https://travis-ci.org/ileasile/zeppelin

zjffdu · 2019-11-19T05:15:11Z

LGTM, will merge if no more comments

ileasile · 2019-11-19T06:31:26Z

@zjffdu Yes, this may be already merged

pan3793 · 2023-09-11T16:43:18Z

@dkaznacheev @ileasile the Koltin version is pretty old and depends on unstable API, do you guys have a plan to upgrade it to keep things up-to-date?

dkaznacheev added 30 commits September 6, 2019 15:05

Added kotlin interpreter stub

881cb41

Loading compiler configuration

61ff6d3

Added kotlin REPL interpreter with IO stubs

bf6825e

Fixed kotlin interpreter, path to java and kotlin runtimes are hardcoded

2cca5c9

Turned on Kotlin syntax highlighting

4016576

Added output and cancellation

422c76d

Implemented kotlin repl diagnostic message holder

d79df4d

Added tests

bc018c7

Minor cleanups

6add47d

Added bind method to Kotlin interpreter

b29c448

Changed KotlinInterpreter to use new scripting API

6c01762

Added object binding to KotlinInterpreter

634b757

Added Kotlin Spark interpreter stub

f135a06

Fixed maven-enforcer package version error

eed0d94

Updated KotlinInterpreter to use new API

01cd61e

Attempt to add spark context to ExecutionContext

d196552

Reordered imports for checkstyle

2eeb2f6

Passing compiler options to KotlinInterpreter

eb00b5c

Fixed REPLInterpreter not using builder

b9a3927

Added class writing to classoutputdir

04e86d1

Shading jpountz module

d89dbe6

Switched to local kotlin JARs with bugfixes

d478dd1

Added ZeppelinContext to ExecutionContext

ea024b5

Fixed show in ZeppelinContext

62d87c1

Fixed interpreter output error

fb5a64b

Refactored Kotlin execution context

b9e9f7e

Added kotlin interpreter to SparkZeppelinContext list of interpreters

4aaf0bb

Added ZeppelinContext to default Kotlin Interpreter

019a7c5

Refactoring: renamed KotlinContext and children to KotlinReceiver to …

de1cd14

…avoid confusion with ZeppelinContext

Added max result output limit for Kotlin REPL

060a822

ileasile reviewed Nov 8, 2019

View reviewed changes

dkaznacheev and others added 7 commits November 8, 2019 20:15

Minor PR fixes

2c25a7d

Minor PR fixes

171a35b

Error fixes in classpath and type printing

2984577

Merge branch 'refs/heads/master' into kotlin-spark-interpreter

6e10ed7

# Conflicts: # .travis.yml # LICENSE

Added a comment about repl history records

eacb964

Fix comment

92f00af

Fix comment indentation

7a5252d

Merge branch 'refs/heads/master' into kotlin-spark-interpreter

29a4dba

# Conflicts: # .travis.yml

ileasile added 2 commits November 13, 2019 13:18

[ZEPPELIN-4323] Merge branch 'refs/heads/master' into kotlin-spark-in…

c8ebf79

…terpreter

Fix KotlinSpark interpreter test for old versions of spark.

347c9eb

ileasile added 3 commits November 14, 2019 12:00

Include kotlin interpreter in zeppelin_server builds

4358084

Merge branch 'master' into kotlin-spark-interpreter

c8e68e3

Increased memory limits

ad2c1a9

ileasile force-pushed the kotlin-spark-interpreter branch from e6c4721 to ad2c1a9 Compare November 15, 2019 13:47

Temporary fix for hadoop_lzo

9d5b6a3

Changed hadoop-lzo repo and added Kotlin test

ad0380f

asfgit closed this in ee2613f Nov 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ZEPPELIN-4323] Kotlin support for Spark interpreter #3440

[ZEPPELIN-4323] Kotlin support for Spark interpreter #3440

dkaznacheev commented Sep 6, 2019 •

edited

ileasile Nov 8, 2019

dkaznacheev Nov 8, 2019

ileasile Nov 8, 2019

dkaznacheev Nov 8, 2019

ileasile Nov 8, 2019 •

edited

dkaznacheev Nov 8, 2019

zjffdu commented Nov 13, 2019

ileasile commented Nov 13, 2019

zjffdu commented Nov 13, 2019

ileasile commented Nov 13, 2019

zjffdu commented Nov 14, 2019

zjffdu commented Nov 18, 2019

ileasile commented Nov 18, 2019

zjffdu commented Nov 19, 2019

ileasile commented Nov 19, 2019

pan3793 commented Sep 11, 2023

[ZEPPELIN-4323] Kotlin support for Spark interpreter #3440

[ZEPPELIN-4323] Kotlin support for Spark interpreter #3440

Conversation

dkaznacheev commented Sep 6, 2019 • edited

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

ileasile Nov 8, 2019

Choose a reason for hiding this comment

dkaznacheev Nov 8, 2019

Choose a reason for hiding this comment

ileasile Nov 8, 2019

Choose a reason for hiding this comment

dkaznacheev Nov 8, 2019

Choose a reason for hiding this comment

ileasile Nov 8, 2019 • edited

Choose a reason for hiding this comment

dkaznacheev Nov 8, 2019

Choose a reason for hiding this comment

zjffdu commented Nov 13, 2019

ileasile commented Nov 13, 2019

zjffdu commented Nov 13, 2019

ileasile commented Nov 13, 2019

zjffdu commented Nov 14, 2019

zjffdu commented Nov 18, 2019

ileasile commented Nov 18, 2019

zjffdu commented Nov 19, 2019

ileasile commented Nov 19, 2019

pan3793 commented Sep 11, 2023

dkaznacheev commented Sep 6, 2019 •

edited

ileasile Nov 8, 2019 •

edited