ZEPPELIN-3085 Introduce generic ConfInterpreter for more fine-grained control of interpreter setting #2692

zjffdu · 2017-11-30T06:01:29Z

What is this PR for?

Zeppelin's interpreter setting is shared by all the users and notes, if you want to have different setting you have to create new interpreter, e.g. you can create spark_jar1 for running spark with dependency jar1 and spark_jar2 for running spark with dependency jar2.
This approach works, but not so convenient. ConfInterpreter can provide more fine-grained control on interpreter setting and more flexibility.

ConfInterpreter is a generic interpreter that could be used by any interpreters. The input format should be property file format.

In the first paragraph, we use ConfInterpreter to make custom configuration of spark interpreter (set app name, yarn-client mode & add spark-csv dependencies). Then you can run the second paragraph which use spark-csv.

It can be used to make custom setting for any interpreter. ConfInterpreter would run in the zeppelin server side, it would update the interpreter properties before you launch the interpreter process, so it needs to run before interpreter process launched. And when interpreter process is launched is determined by interpreter mode setting. So users needs to understand the interpreter mode setting of zeppelin and be aware when interpreter process is launched. E.g. If we set spark interpreter setting as isolated per note. Under this setting, each note will launch one interpreter process. In this scenario, user need to put ConfInterpreter as the first paragraph as the above example. Otherwise the customized setting can not be applied (The paragraph using ConfInterpreter will report ERROR).

What type of PR is it?

[Feature | Documentation]

Todos

- Task

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-3085

How should this be tested?

Unit test, System test is added, also manually verified it.

Screenshots (if appropriate)

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

zjffdu · 2017-12-04T01:39:38Z

@Leemoonsoo @jongyoul @felixcheung @prabhjyotsingh Could you help review it ? Thanks

felixcheung

interesting idea.
can I have multiple ConfInterpreter at the same time?

felixcheung · 2017-12-05T06:43:12Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java

@@ -716,6 +719,11 @@ private String getInterpreterClassFromInterpreterSetting(String replName) {
        return info.getClassName();
      }
    }
+    //TODO(zjffdu) It requires user can not create interpreter with name `conf`,
+    // conf is a reserved word of interpreter name
+    if (replName.equals("conf")) {


why do we have to hardcode this?

I make conf as a reserved interpreter name for this generic ConfInterpreter so that all the interpreters can use it. e.g. %md.conf, %sh.conf, %spark.conf

felixcheung · 2017-12-05T06:44:49Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java

+        if (!intp.getProperties().equals(properties) &&
+            interpreterGroup.getRemoteInterpreterProcess() != null &&
+            interpreterGroup.getRemoteInterpreterProcess().isRunning()) {
+          throw new IOException("Can not change interpreter properties when interpreter process " +


I'm a bit worry about this? I think this is the way why %dep is so hard to use - the implicit order of interpreter is not very visible to an user, especially with different interpreter isolation mode etc.

Right, this is what I mention in the documentation. This generic ConfInterpreter is for advanced users that they needs to understand the interpreter mode setting of zeppelin and be aware when interpreter process is launched.

zjffdu · 2017-12-05T07:08:02Z

can I have multiple ConfInterpreter at the same time?

I don't quite get it. For now each interpreter process have one ConfInterpreter which is used for customize this interpreter process setting.

felixcheung · 2017-12-05T08:39:06Z

right, that's what I'm wondering

zjffdu · 2017-12-12T23:59:54Z

@Leemoonsoo @jongyoul @felixcheung @prabhjyotsingh any more comments ? I will merge it if no more comments

…d control of interpreter setting

… control of interpreter setting ### What is this PR for? Zeppelin's interpreter setting is shared by all the users and notes, if you want to have different setting you have to create new interpreter, e.g. you can create `spark_jar1` for running spark with dependency jar1 and `spark_jar2` for running spark with dependency jar2. This approach works, but not so convenient. `ConfInterpreter` can provide more fine-grained control on interpreter setting and more flexibility. `ConfInterpreter` is a generic interpreter that could be used by any interpreters. The input format should be property file format. In the first paragraph, we use ConfInterpreter to make custom configuration of spark interpreter (set app name, yarn-client mode & add spark-csv dependencies). Then you can run the second paragraph which use spark-csv. ![conf_interpreter](https://user-images.githubusercontent.com/164491/33419465-74a3fae8-d5e5-11e7-8b25-76407804d979.png) It can be used to make custom setting for any interpreter. `ConfInterpreter` would run in the zeppelin server side, it would update the interpreter properties before you launch the interpreter process, so it needs to run before interpreter process launched. And when interpreter process is launched is determined by interpreter mode setting. So users needs to understand the interpreter mode setting of zeppelin and be aware when interpreter process is launched. E.g. If we set spark interpreter setting as isolated per note. Under this setting, each note will launch one interpreter process. In this scenario, user need to put `ConfInterpreter` as the first paragraph as the above example. Otherwise the customized setting can not be applied (The paragraph using ConfInterpreter will report ERROR). ### What type of PR is it? [Feature | Documentation] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3085 ### How should this be tested? * Unit test, System test is added, also manually verified it. ### Screenshots (if appropriate) ![conf_interpreter](https://user-images.githubusercontent.com/164491/33419465-74a3fae8-d5e5-11e7-8b25-76407804d979.png) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes apache#2692 from zjffdu/ZEPPELIN-3085 and squashes the following commits: 87ce20f [Jeff Zhang] ZEPPELIN-3085. Introduce generic ConfInterpreter for more fine-grained control of interpreter setting

* [ZEPPELIN-3062] Removes ctrl+s default behavior from notebook ### What is this PR for? As a programmer I habitually press CTRL + S out of fear for my sanity. Other web text editing tools (google docs, jupyter notebooks) ignore the base functionality of CTRL + S of saving the HTML page, and do nothing instead. This is a small change that will makes the user experience of Zeppelin notebooks much much better. ### What type of PR is it? Improvement ### Todos ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3062 ### How should this be tested? * I tested manually: 1. Ran `yarn run dev` 2. Went to `localhost:9000` 3. Created a new note, now anytime a user presses 'CTRL+S' in the scope of the note nothing happens, and the user can continue coding uninterrupted. ### Screenshots (if appropriate) ### Questions: Author: Tess <tesshbianchi@gmail.com> Closes apache#2677 from tessbianchi/ZEPPELIN-3062 and squashes the following commits: db80533 [Tess] Removes ctrl+s default behavior from notebook * ZEPPELIN-3072: Zeppelin UI becomes slow/unresponsive if there are too many notebooks Zeppelin UI becomes slow/unresponsive if there are too many notebooks Have attached a notebook directory in [JIRA](https://issues.apache.org/jira/secure/attachment/12898650/notebook.zip) with 500+ notebooks, now with these notebooks, every time user goes to homepage Zeppelin UI becomes unresponsive for few seconds. [Bug Fix | Improvement] * [x] - Fix search box * [x] - Order of notebook * [ZEPPELIN-3072](https://issues.apache.org/jira/browse/ZEPPELIN-3072) Create 500+ notebook, or import it from [JIRA](https://issues.apache.org/jira/secure/attachment/12898650/notebook.zip), now observe UI it becomes slow/laggy while homepage is rendering. Before: ![before](https://user-images.githubusercontent.com/674497/33070354-c36acdfa-cedd-11e7-81f9-ff0b526622f3.gif) After: ![after](https://user-images.githubusercontent.com/674497/33070353-c3317988-cedd-11e7-9431-fcf596928c3b.gif) * Does the licenses files need update? N/A * Is there breaking changes for older versions? N/A * Does this needs documentation? N/A Author: prabhjyotsingh <prabhjyotsingh@gmail.com> Closes apache#2683 from prabhjyotsingh/ZEPPELIN-3072 and squashes the following commits: 06b8ef8 [prabhjyotsingh] add license file 483a3ff [prabhjyotsingh] navbar dropdown bug eb506ba [prabhjyotsingh] fix test 00ec295 [prabhjyotsingh] fix sort and serch b4cbba8 [prabhjyotsingh] ZEPPELIN-3072: Zeppelin UI becomes slow/unresponsive if there are too many notebooks Change-Id: Ibc157312b726b9704cab088192a39e942d8da43d * Add missing build dependencies. Add `apt-get install` lines for R dependencies that are needed to build successfully. ### What is this PR for? This pull request adds installation instructions for building which relate to missing dependencies resulting in failing to build (if using R). ### What type of PR is it? [Documentation] Author: Jens Grassel <jens@wegtam.com> Closes apache#2674 from jan0sch/docs-missing-build-deps and squashes the following commits: aa581c1 [Jens Grassel] Add missing build dependencies. * ZEPPELIN-3037 Configure Http Request Header Size Limit for Jetty ### What is this PR for? In some deployment scenarios it is necessary to increase jetty.request.header.size, which default value is 8192. This will reduce the chance of HTTP Error 413 Request entity too large. There should be a mechanism to configure this setting. ### What type of PR is it? [Feature] ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3037 ### How should this be tested? * There is an integration test (automated unit test included) for testing this feature. * To test manually, after increasing setting, make any http request to zeppelin with a request header bigger than 8K , you should not get an HTTP Error 413 Request entity too large. ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? Yes. Author: byamthev <byamthev@gmail.com> Closes apache#2663 from byamthev/zeppelin3037 and squashes the following commits: 8ff2620 [byamthev] [ZEPPELIN-3037] Configure Http Request Header Size Limit for Jetty * ZEPPELIN-2984: allow enter from username ### What is this PR for? Allow an "Enter" keypress in the username field during login ### What type of PR is it? Improvement ### Todos * N/A ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2984 ### How should this be tested? * Pull up login screen. Fill in username and password. Click in username field. Press enter. We would expect the login handler to be called. * Pull up login screen. Fill in username and password. Press enter (while still in password field). We would expect the login handler to be called just as before. * Alternatively, use a password manager (e.g. LastPass). Save the password. Pull up the login screen. Press Enter. We would expect the login handler to be called. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Gabe Fernando <gef756@gmail.com> Closes apache#2616 from gef756/dev-login-enter and squashes the following commits: 301d398 [Gabe Fernando] MRG: Merge remote-tracking branch 'origin/master' into dev-login-enter 2a1865a [Gabe Fernando] (ZEPPELIN-2984) ENH: allow enter from username * [ZEPPELIN-3042] updating jgit to support post-commit hooks ### What is this PR for? Git functionality in Zeppelin is provided by `jgit`, but it's fixed to an old version. A newer version of said library supports new things, primarily post-commit hooks. ### What type of PR is it? Improvement ### Todos * [ x ] - CI ([two failed, but due to yarn](https://travis-ci.org/kokes/zeppelin)) ### What is the Jira issue? [ZEPPELIN-3042](https://issues.apache.org/jira/browse/ZEPPELIN-3042) ### Questions: * Does the licenses files need update? -- Yes, only because the license file notes the exact version of included libraries. `jgit` hasn't changed, license-wise, so it's only a matter of updating the version information * Is there breaking changes for older versions? -- nope * Does this needs documentation? -- possibly, to let people know they get use post-commit hooks now? Author: Ondrej Kokes <ondrej.kokes@gmail.com> Closes apache#2658 from kokes/jgit-update and squashes the following commits: a2cde16 [Ondrej Kokes] [ZEPPELIN-3042] newer jgit supports post-commit hooks a4fc816 [Ondrej Kokes] [ZEPPELIN-3042] newer jgit supports post-commit hooks * [ZEPPELIN-3076]Chart field is also draggable and sortable in the 'keys', 'groups' and 'values' ### What is this PR for? Current the `keys`, `groups` and `values` is only droppable. It is a little inconvenient if I want to drag it to other place or sort it. This feature let `keys`, `groups` and `values` not only `droppable`, but also `draggable` and `sorttable`. ### What type of PR is it? [Feature] ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3076 [ZEPPELIN-3076] ### How should this be tested? I have Add the Screenshots for the test. Just as the Screenshots showing, all the element in `keys`, `groups` and `values` is draggable and sorttable. ![untitled project4](https://user-images.githubusercontent.com/5969176/33252922-ed1ab0b4-d37b-11e7-8a5c-b3dbb6765d18.gif) ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? NO Author: liguohui <liguohui@cmss.chinamobile.com> Closes apache#2686 from liguohuicmss/draggable-field-keys-groups-values2 and squashes the following commits: 08b86b8 [liguohui] add a new line ebb8436 [liguohui] Chart field is also draggable and sortable in the 'keys', 'groups' and 'values' * [hotfix] Downgrade JGit from 4.9.0 to 4.5.4 ### What is this PR for? The version of JGit was updated to 4.9.0 at apache#2658. However, this version does not support Java 7 and Travis CI test always fails now: https://travis-ci.org/kjmrknsn/zeppelin/builds/310104872 To fix this issue, downgrade JGit from 4.9.0 to 4.5.4 which is the latest JGit version which supports Java 7: https://projects.eclipse.org/projects/technology.jgit. I confirmed that Travis CI test was passed with JGit 4.5.4: https://travis-ci.org/kjmrknsn/zeppelin/builds/310107611 Now, all PRs to Zeppelin don't pass Travis CI tests, so it's preferable that this PR is merged soon. ### What type of PR is it? [Bug Fix] ### Todos ### What is the Jira issue? ### How should this be tested? I confirmed that Travis CI test passed with JGit 4.5.4: https://travis-ci.org/kjmrknsn/zeppelin/builds/310107611 ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? Yes. zeppelin-distribution/src/bin_license/LICENSE was updated at this PR. * Is there breaking changes for older versions? No. * Does this needs documentation? No. Author: Keiji Yoshida <kjmrknsn@gmail.com> Closes apache#2693 from kjmrknsn/ci-test-against-jgit-4.5.4 and squashes the following commits: 27b08c9 [Keiji Yoshida] CI Test against JGit 4.5.4 * [Zeppelin-2571] & [Zeppelin-465] Run paragraphs: from first/current to current/last ### What is this PR for? This pr add the ability to run all paragraphs from the first to the current and from the current to the last. This makes it easier to update the data if changes are made in one of the related paragraphs. ### What type of PR is it? Feature ### What is the Jira issue? [ZEPPELIN-2571](https://issues.apache.org/jira/browse/ZEPPELIN-2571) - (Add a "Run to here" option on paragraphs) [ZEPPELIN-465](https://issues.apache.org/jira/browse/ZEPPELIN-465) - (Capability to run all cells below the current cell) ### How should this be tested? 1. Click on the "Run: from first to this" or "Run: from this to last" in the dropdown menu. Paragraphs from the first/current to the current/last will be started in order. 2. Press the key combination: CTRL + SHIFT + ENTER. A window will appear with the choice of the desired action. ![capture2](https://user-images.githubusercontent.com/25951039/33269914-58eff828-d393-11e7-9ebf-6437ec11c8f2.PNG) Choose one of two actions. The selected action will be performed. ### Screenshots (if appropriate) ![capture1_edit](https://user-images.githubusercontent.com/25951039/33269915-5951b19e-d393-11e7-831b-42d4523908ae.png) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: tinkoff-dwh <tinkoff.dwh@gmail.com> Closes apache#2688 from tinkoff-dwh/ZEPPELIN-2571&465 and squashes the following commits: 304c196 [tinkoff-dwh] [ZEPPELIN-2517]&[ZEPPELIN-465] fix shortcut description 3d57b30 [tinkoff-dwh] [ZEPPELIN-2517]&[ZEPPELIN-465] save paragraph's focus 70f130c [tinkoff-dwh] [Zeppelin-2517]&[Zeppelin-465] some text change 8ca1df2 [tinkoff-dwh] [Zeppelin-2517]&[Zeppelin-465] line reduction a7a4158 [tinkoff-dwh] [ZEPPELIN-2517]&[ZEPPELIN-465] logic change 9fbcdb6 [tinkoff-dwh] [ZEPPELIN-2517]&[ZEPPELIN-465] add keyboard shortcut bfc3891 [tinkoff-dwh] [ZEPPELIN-2517]&[ZEPPELIN-465] Run paragraphs from 1st to this. From this to last * [ZEPPELIN-3075]Fix unqiue algo for the web side in pivot.js file. ### What is this PR for? unique() algorithm is not correct in pivot.js file. If the input is `[2, 3, 3, 3, 4, 5]` and the output will be `[2, 3, 3, 4, 5]`. The number `3` is still duplicated. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3075 [ZEPPELIN-3075] ### How should this be tested? This is very easy and no need test. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: liguohui <liguohui@cmss.chinamobile.com> Closes apache#2685 from liguohuicmss/pivot-unqiue-algo and squashes the following commits: 2063175 [liguohui] delete the multi empty line ea582d9 [liguohui] delete some spaces at the end of the line 79c763a [liguohui] add a empty line 99cf93d [liguohui] Revert "Chart field is also draggable and sortable in the 'keys', 'groups' and 'values'" fdde39f [liguohui] add unit test for unique algo in pivot.js f996747 [liguohui] Chart field is also draggable and sortable in the 'keys', 'groups' and 'values' 943e80a [liguohui] Fix unqiue algo for the web side in pivot.js file. * [FIX] fix autocomplete ### What is this PR for? After refactoring of Interpreter autocomplete (from server side) not works without first Run of interpreter. This PR fix it. ### What type of PR is it? [Fix] ### How should this be tested? * Create new Note (JDBC interpreter), try to use autocomplete (schema, tables) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: tinkoff-dwh <tinkoff.dwh@gmail.com> Closes apache#2691 from tinkoff-dwh/fix_autocomplete and squashes the following commits: e9bad01 [tinkoff-dwh] remove trim from completion 141dff5 [tinkoff-dwh] [FIX] fix autocomplete * [HOTFIX]: Fix IPythonInterpreter unit test ### What is this PR for? This is for hotfix of `IPythonInterpreter` unit test failure. Just specify the version of ipython in `install_external_dependencies.sh`, otherwise latest ipython version will be installed, and the behavior may change. ### What type of PR is it? [Hot Fix] ### Todos * [ ] - Task ### What is the Jira issue? * ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes apache#2703 from zjffdu/ipython_version and squashes the following commits: b4c7b42 [Jeff Zhang] HotFix: Fix IPythonInterpreter unit test * [ZEPPELIN-3014] NPE bug fix and Error message enhancement with Kylin Interpreter ### What is this PR for? A few sentences describing the overall goals of the pull request's commits. First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html ### What type of PR is it? Bug Fix ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3014 ### How should this be tested? * Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration * Use existing unit tests in kylin module. ### Screenshots (if appropriate) #### before: NPE when result set is empty ![image](https://user-images.githubusercontent.com/18542573/32154048-f1b8ba58-bcfb-11e7-98cc-98cdf484f2d5.png) #### after: no NPE when result set is empty, just an empty table ![image](https://user-images.githubusercontent.com/18542573/32154069-110215d0-bcfc-11e7-87e9-cc049001f1c7.png) #### before: when query fails, only error code is returned, no error message ![image](https://user-images.githubusercontent.com/18542573/32154088-29651938-bcfc-11e7-9e66-cd2cfccba054.png) #### after: when query fails, both error code and error message are displayed to users ![image](https://user-images.githubusercontent.com/18542573/32154096-3d3ab01c-bcfc-11e7-8cf3-d710d96b8c5a.png) ### Questions: * Does the licenses files need update? No. * Is there breaking changes for older versions? No. * Does this needs documentation? No. Author: Liu <jinxliu@ebay.com> Closes apache#2645 from jinxliu/kylin-intp-new and squashes the following commits: d5692bf [Liu] refactor 85b6424 [Liu] add test for empty result set 4596470 [Liu] ZEPPELIN-3014: NPE bug fix and Error message enhancement with Kylin Interpreter * ZEPPELIN-3085 Introduce generic ConfInterpreter for more fine-grained control of interpreter setting ### What is this PR for? Zeppelin's interpreter setting is shared by all the users and notes, if you want to have different setting you have to create new interpreter, e.g. you can create `spark_jar1` for running spark with dependency jar1 and `spark_jar2` for running spark with dependency jar2. This approach works, but not so convenient. `ConfInterpreter` can provide more fine-grained control on interpreter setting and more flexibility. `ConfInterpreter` is a generic interpreter that could be used by any interpreters. The input format should be property file format. In the first paragraph, we use ConfInterpreter to make custom configuration of spark interpreter (set app name, yarn-client mode & add spark-csv dependencies). Then you can run the second paragraph which use spark-csv. ![conf_interpreter](https://user-images.githubusercontent.com/164491/33419465-74a3fae8-d5e5-11e7-8b25-76407804d979.png) It can be used to make custom setting for any interpreter. `ConfInterpreter` would run in the zeppelin server side, it would update the interpreter properties before you launch the interpreter process, so it needs to run before interpreter process launched. And when interpreter process is launched is determined by interpreter mode setting. So users needs to understand the interpreter mode setting of zeppelin and be aware when interpreter process is launched. E.g. If we set spark interpreter setting as isolated per note. Under this setting, each note will launch one interpreter process. In this scenario, user need to put `ConfInterpreter` as the first paragraph as the above example. Otherwise the customized setting can not be applied (The paragraph using ConfInterpreter will report ERROR). ### What type of PR is it? [Feature | Documentation] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3085 ### How should this be tested? * Unit test, System test is added, also manually verified it. ### Screenshots (if appropriate) ![conf_interpreter](https://user-images.githubusercontent.com/164491/33419465-74a3fae8-d5e5-11e7-8b25-76407804d979.png) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes apache#2692 from zjffdu/ZEPPELIN-3085 and squashes the following commits: 87ce20f [Jeff Zhang] ZEPPELIN-3085. Introduce generic ConfInterpreter for more fine-grained control of interpreter setting * ZEPPELIN-3051. Support Interpreter Process Recovery ### What is this PR for? This PR is for the purpose of recover running interpreter process when zeppelin server is restarted. This would be useful when restarting zeppelin without interrupt current running interpreter processes, should be useful when admin do maintenance or upgrading. Interface `RecoveryStorage` is used for storing the information of running interpreter process. Currently it only has one implementation `FileSystemRecoveryStorage`, other implementation could be done later (such as zookeeper based). `InterpreterLauncher` is the component where to recover the running interpreter process. Test: * RecoveryTest.java * FileSystemRecoveryStorageTest.java Design Doc: https://docs.google.com/document/d/1Plm3Hd40aGdNaXmjdsoY4ek3f-gTijTMGMkNjAZN39Y/edit?usp=sharing ### What type of PR is it? [Feature] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3051 ### How should this be tested? Unit test & Integration Test is added. Also manually verified. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes apache#2668 from zjffdu/ZEPPELIN-3051 and squashes the following commits: a4c9b9c [Jeff Zhang] address comments 575b7b9 [Jeff Zhang] fix the pid of interpreter process id 02b118f [Jeff Zhang] address comments da7cbb9 [Jeff Zhang] ZEPPELIN-3051. Support Interpreter Process Recovery * [MINOR] Fix notebook title bar margin ### What is this PR for? Notebook title bar location is little bit shifted to left. This minor fix notebook title bar margin. See screenshots below. ### What type of PR is it? Bug Fix ### Todos * [x] - fix margin ### Screenshots (if appropriate) Before ![image](https://user-images.githubusercontent.com/1540981/33742650-3469c2fe-db5e-11e7-8aa6-936d0a28de3a.png) After ![image](https://user-images.githubusercontent.com/1540981/33742640-250c3dc8-db5e-11e7-8af3-e8a9d7105963.png) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <moon@apache.org> Closes apache#2699 from Leemoonsoo/minor_noteaction_margin and squashes the following commits: 703f458 [Lee moon soo] fix noteAction magin * [ZEPPELIN-3091] Correct aggregation functionality in charts ### What is this PR for? The aggregation functions interpret NaN columns as 1 which leads to incorrect output being shown in charts. This PR fixes this by correcting the sum, min, max and average aggregation method. ### What type of PR is it? Bug Fix ### Todos ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3091 ### How should this be tested? * Update the data field of a paragraph results with %table to contain null values, e.g. `"data": "age\tvalue\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n20\t5\n20\tnull\n"` The "null" values should be ignored for sum, min, max but included for count (and hence average). ### Screenshots (if appropriate) ![correct_sum](https://user-images.githubusercontent.com/6438072/33609178-14e05988-d9ed-11e7-9f1b-99e0141c5153.png) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Naman Mishra <namanmishra1991@gmail.com> Closes apache#2696 from namanmishra91/ZEPPELIN-3091 and squashes the following commits: d8a57c2 [Naman Mishra] Add test 38ad39c [Naman Mishra] Merge branch 'master' into ZEPPELIN-3091 568ae3f [Naman Mishra] Correct aggregation functionality in charts * [ZEPPELIN-3101] updated network label, added link to network display in index.md ### What is this PR for? The docs index must show the reference to the network visualization as for the the other types ### What type of PR is it? [Improvement] ### Todos * [x] - Add missing link ### What is the Jira issue? [ZEPPELIN-3101](https://issues.apache.org/jira/projects/ZEPPELIN/issues/ZEPPELIN-3101) ### How should this be tested? 1. cd `docs/` 2. build: `bundle exec jekyll build --safe` 3. check the link is present ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Andrea Santurbano <santand@gmail.com> Closes apache#2702 from conker84/graph-docs and squashes the following commits: 58d58cc [Andrea Santurbano] updated network label, added link to network display in index.md

… control of interpreter setting ### What is this PR for? Zeppelin's interpreter setting is shared by all the users and notes, if you want to have different setting you have to create new interpreter, e.g. you can create `spark_jar1` for running spark with dependency jar1 and `spark_jar2` for running spark with dependency jar2. This approach works, but not so convenient. `ConfInterpreter` can provide more fine-grained control on interpreter setting and more flexibility. `ConfInterpreter` is a generic interpreter that could be used by any interpreters. The input format should be property file format. In the first paragraph, we use ConfInterpreter to make custom configuration of spark interpreter (set app name, yarn-client mode & add spark-csv dependencies). Then you can run the second paragraph which use spark-csv. ![conf_interpreter](https://user-images.githubusercontent.com/164491/33419465-74a3fae8-d5e5-11e7-8b25-76407804d979.png) It can be used to make custom setting for any interpreter. `ConfInterpreter` would run in the zeppelin server side, it would update the interpreter properties before you launch the interpreter process, so it needs to run before interpreter process launched. And when interpreter process is launched is determined by interpreter mode setting. So users needs to understand the interpreter mode setting of zeppelin and be aware when interpreter process is launched. E.g. If we set spark interpreter setting as isolated per note. Under this setting, each note will launch one interpreter process. In this scenario, user need to put `ConfInterpreter` as the first paragraph as the above example. Otherwise the customized setting can not be applied (The paragraph using ConfInterpreter will report ERROR). ### What type of PR is it? [Feature | Documentation] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3085 ### How should this be tested? * Unit test, System test is added, also manually verified it. ### Screenshots (if appropriate) ![conf_interpreter](https://user-images.githubusercontent.com/164491/33419465-74a3fae8-d5e5-11e7-8b25-76407804d979.png) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes apache#2692 from zjffdu/ZEPPELIN-3085 and squashes the following commits: 87ce20f [Jeff Zhang] ZEPPELIN-3085. Introduce generic ConfInterpreter for more fine-grained control of interpreter setting

zjffdu force-pushed the ZEPPELIN-3085 branch 4 times, most recently from af004ab to b986234 Compare December 4, 2017 01:39

zjffdu force-pushed the ZEPPELIN-3085 branch 2 times, most recently from e1dea86 to 29fc384 Compare December 4, 2017 05:56

felixcheung reviewed Dec 5, 2017

View reviewed changes

zjffdu force-pushed the ZEPPELIN-3085 branch from 29fc384 to fb34533 Compare December 14, 2017 01:08

ZEPPELIN-3085. Introduce generic ConfInterpreter for more fine-graine…

87ce20f

…d control of interpreter setting

zjffdu force-pushed the ZEPPELIN-3085 branch from fb34533 to 87ce20f Compare December 14, 2017 02:45

asfgit closed this in 13f8e6c Dec 14, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ZEPPELIN-3085 Introduce generic ConfInterpreter for more fine-grained control of interpreter setting #2692

ZEPPELIN-3085 Introduce generic ConfInterpreter for more fine-grained control of interpreter setting #2692

zjffdu commented Nov 30, 2017 •

edited

Loading

zjffdu commented Dec 4, 2017

felixcheung left a comment

felixcheung Dec 5, 2017

zjffdu Dec 5, 2017

felixcheung Dec 5, 2017

zjffdu Dec 5, 2017

zjffdu commented Dec 5, 2017

felixcheung commented Dec 5, 2017

zjffdu commented Dec 12, 2017 •

edited

Loading

ZEPPELIN-3085 Introduce generic ConfInterpreter for more fine-grained control of interpreter setting #2692

ZEPPELIN-3085 Introduce generic ConfInterpreter for more fine-grained control of interpreter setting #2692

Conversation

zjffdu commented Nov 30, 2017 • edited Loading

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

zjffdu commented Dec 4, 2017

felixcheung left a comment

Choose a reason for hiding this comment

felixcheung Dec 5, 2017

Choose a reason for hiding this comment

zjffdu Dec 5, 2017

Choose a reason for hiding this comment

felixcheung Dec 5, 2017

Choose a reason for hiding this comment

zjffdu Dec 5, 2017

Choose a reason for hiding this comment

zjffdu commented Dec 5, 2017

felixcheung commented Dec 5, 2017

zjffdu commented Dec 12, 2017 • edited Loading

zjffdu commented Nov 30, 2017 •

edited

Loading

zjffdu commented Dec 12, 2017 •

edited

Loading