Conversation
### What is this PR for? A tutorial notebook filename, which includes `|`, is not compatible with windows filesystem. This PR replace `|` with `,`. ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3839 ### How should this be tested? CI pass ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <moon@apache.org> Closes apache#3213 from Leemoonsoo/ZEPPELIN-3839 and squashes the following commits: 0c1a5dd [Lee moon soo] Replace | to , in the filename
### What is this PR for? After apache#3211, some link in README.md need to be updated. ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3836 ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <leemoonsoo@gmail.com> Closes apache#3214 from Leemoonsoo/ZEPPELIN-3836-followup and squashes the following commits: ba2fe1c [Lee moon soo] http -> https 7257366 [Lee moon soo] update install, source build link
### What is this PR for? how_to_build.md includes some build profiles that has removed/changed. This PR makes how_to_build.md up to dated. ### What type of PR is it? Documentation ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3837 ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <leemoonsoo@gmail.com> Author: Lee moon soo <moon@apache.org> Closes apache#3212 from Leemoonsoo/ZEPPELIN-3837 and squashes the following commits: 704b8c8 [Lee moon soo] spark 2.1 -> 2.3 profile for binary release creation 9c35656 [Lee moon soo] update how_to_build.md
### What is this PR for? The current implementation uses the default google applications service account to establish a connection with GCS. We cannot specify a custom service account for the connection with the existing implementation. This PR enables the same. * Added ZEPPELIN_NOTEBOOK_GCS_CREDENTIALS_FILE to ConfVars * Added functionality in GCSNotebookRepo and OldGCSNotebookRepo to get credentials from CREDENTIALS_FILE * Updated doc string and documentation ### What type of PR is it? [Improvement] ### Todos * None ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3825 ### How should this be tested? * Create a new service account that has access to write to google cloud storage. * Disable Google Storage access to the default application service account. * Build and deploy zeppelin after updating zeppelin.notebook.gcs.credentialsJsonFilePath in zeppelin-site.xml * Validate that new notes are being written to GCS bucket ### Screenshots (if appropriate) * None ### Questions: * Does the licenses files need update? -No * Is there breaking changes for older versions? - No * Does this needs documentation? - Yes. Existing documentation has been updated to reflect the same. Author: sanjaykumar <sanjay.kumar@shopkick.com> Closes apache#3207 from 88sanjay/ZEPPELIN-3825 and squashes the following commits: 15c8aa0 [sanjaykumar] Merge branch 'master' into ZEPPELIN-3825 84b8da4 [sanjaykumar] ZEPPELIN-3825. Allow custom service account for GCSNotebookRepo
### What is this PR for? Master branch displays note id instead of note name. I think it's side effect after change apache#3163. ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3849 ### How should this be tested? See if note name is displayed instead of note id ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <moon@apache.org> Closes apache#3218 from Leemoonsoo/ZEPPELIN-3849 and squashes the following commits: 3e91b99 [Lee moon soo] display note name
### What is this PR for? "Run all paragraphs" button runs only the first paragraph of the note. This PR fixes the problem. ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3850 ### How should this be tested? Unittest included ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <moon@apache.org> Closes apache#3219 from Leemoonsoo/ZEPPELIN-3850 and squashes the following commits: 6cf5dd8 [Lee moon soo] add test runAllParagraphs 1ea145c [Lee moon soo] check condition correctly to run all paragraphs
### What is this PR for? Code completion is broken due to the shaded jar in zeppelin-interpreter-api, this PR fix this issue and also add unit test for it. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://jira.apache.org/jira/browse/ZEPPELIN-3846 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes apache#3220 from zjffdu/ZEPPELIN-3846 and squashes the following commits: 5cc385e [Jeff Zhang] ZEPPELIN-3846. Code Completion is broken
### What is this PR for? zeppelin-web development mode is broken. `npm run dev` throws error ``` Error: Cannot find module 'webpack/bin/config-yargs' ``` This PR fixes error by upgrading webpack from version 1.x to 4.x ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3848 ### How should this be tested? check if `npm run dev` works under zeppelin-web dir. And able to browse localhost:9000 ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <moon@apache.org> Closes apache#3217 from Leemoonsoo/upgrade_webpack and squashes the following commits: 5b9bc04 [Lee moon soo] http->https ff7e10f [Lee moon soo] update package-lock.json with sha512 instead of sha1 1fc1fff [Lee moon soo] update webpack config and package-lock b66768e [Lee moon soo] upgrade webpack version
This PR change logic of add new event listener on frontend. Now then add new event listener, also added destroy listener. Destroy listener attach to some DOM element and then this element destroyed listener drop event listener. Since the old listeners were not deleted when changing the size of the paragraph / window, an error occurred. Because of this error, updating the text in the paragraph did not happen until the end. <p align="center">before (77 lis.)</p>  <p align="center">after (16 lis.)</p>  * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Savalek <def113@mail.ru> Closes apache#3113 from Savalek/ZEPPELIN-3616 and squashes the following commits: 259703e [Savalek] [ZEPPELIN-3616] remove event if dom's element destroy Change-Id: I35cadc40f152973d6a2d1e9b8f1a797f7ea08445
…n cursor is in the middle of code ### What is this PR for? It is a trivial fix for the code completion of spark interpreter when the cursor is in the middle of code. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://jira.apache.org/jira/browse/ZEPPELIN-3858 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes apache#3221 from zjffdu/ZEPPELIN-3858 and squashes the following commits: bb56b72 [Jeff Zhang] ZEPPELIN-3858. Code completion doesn't work for spark interpreter when cursor is in the middle of code
### What is this PR for? Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418). There are two problems for this upgrade at Zeppelin side: 1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible. See: - https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala - https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection. 2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit: ``` Caused by: java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139) ``` To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`. ### What type of PR is it? [Improvement] ### Todos * [x] - Wait until Spark 2.4.0 is officially released. ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3810 ### How should this be tested? Verified manually against Spark 2.4.0 RC3 ### Questions: * Does the licenses files need update? Yes * Is there breaking changes for older versions? No * Does this needs documentation? No Author: hyukjinkwon <gurwls223@apache.org> Author: Hyukjin Kwon <gurwls223@apache.org> Author: Jeff Zhang <zjffdu@gmail.com> Closes apache#3206 from HyukjinKwon/ZEPPELIN-3810 and squashes the following commits: c2456c9 [Hyukjin Kwon] Py4J 0.10.6 to 0.10.7 573f07d [Jeff Zhang] add test for spark 2.4 (#1) 9ac1797 [hyukjinkwon] Support Spark 2.4
…e deleted when maven clean is invoked ### What is this PR for? This PR is to fix the bug that `ZEPPELIN_HOME`/interpreter folder is always deleted when maven clean is invoked for module `zeppelin`. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://jira.apache.org/jira/browse/ZEPPELIN-3822 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? NO Author: Jeff Zhang <zjffdu@apache.org> Closes apache#3223 from zjffdu/ZEPPELIN-3822 and squashes the following commits: da4773d [Jeff Zhang] ZEPPELIN-3822. All interpreter folders (ZEPPELIN_HOME/interpreter) are deleted when maven clean is invoked
### What is this PR for? Trivial PR for fixing group name missing in spark interpreter log file ### What type of PR is it? [ Hot Fix ] ### Todos * [ ] - Task ### What is the Jira issue? * https://jira.apache.org/jira/browse/ZEPPELIN-3815 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? NO * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes apache#3222 from zjffdu/ZEPPELIN-3815 and squashes the following commits: c1fee03 [Jeff Zhang] ZEPPELIN-3815. No interpreter group name in spark interpreter log file
### What is this PR for? Since v0.8.0, REST API JSON response `status` attribute has changed from `CREATED` to `OK`. Related source code: * https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java * https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/rest/InterpreterRestApi.java ### What type of PR is it? Documentation ### Questions: * Shall I add tests to check the returned `status` in: * https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/test/java/org/apache/zeppelin/rest/InterpreterRestApiTest.java * https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/test/java/org/apache/zeppelin/rest/NotebookRestApiTest.java Author: John Lau <jolks@outlook.com> Closes apache#3224 from jolks/doc_rest_api_fix and squashes the following commits: 9037b55 [John Lau] REST API JSON response status fix
### What is this PR for? Duplicate results in notebooks after ws reconnect. The problem occurs via duplicated 'setConnectedStatus' event handler in notebook.controller.js. The same function exists at https://github.com/Leemoonsoo/zeppelin/blob/dbbf0436490941d117fd4c06b9da5b6cb47697ff/zeppelin-web/src/app/notebook/notebook.controller.js#L99-L104. ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3847 ### How should this be tested? Disconnect and reconnect websocket connection to browser, and check if result is duplicated. ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <leemoonsoo@gmail.com> Closes apache#3230 from Leemoonsoo/ZEPPELIN-3847 and squashes the following commits: dbbf043 [Lee moon soo] remove duplicated setConnectedStatus event handler
### What is this PR for? Currently actual CI build occurs in individual contributor's travis-ci account, and apache Jenkins poll the result from travis-ci and print build status on it's console output, like ``` Build https://travis-ci.org/test/zeppelin/builds/451036679 [1] OK https://travis-ci.org/test/zeppelin/jobs/451036680 [2] Running ... https://travis-ci.org/test/zeppelin/jobs/451036681 [3] Running ... https://travis-ci.org/test/zeppelin/jobs/451036682 [4] Not completed https://travis-ci.org/test/zeppelin/jobs/451036683 [5] Running ... https://travis-ci.org/test/zeppelin/jobs/451036684 [6] Running ... https://travis-ci.org/test/zeppelin/jobs/451036685 [7] Running ... https://travis-ci.org/test/zeppelin/jobs/451036686 [8] Not started https://travis-ci.org/test/zeppelin/jobs/451036687 [9] Not started https://travis-ci.org/test/zeppelin/jobs/451036688 [10] Not started https://travis-ci.org/test/zeppelin/jobs/451036689 [11] Not started https://travis-ci.org/test/zeppelin/jobs/451036690 1 job(s) failed, 9 job(s) running/pending ``` When there're failed job, developer has to open the travis-ci link and download the log and manually look through to see what was wrong. This PR make Jenkins not only print build status but also short summary of failure, so developer quickly identify what went wrong. After this patch, Jenkins console output will look something like ``` [2018-11-18 09:11:20] Author: test, commit: ee6d89b [2018-11-18 09:11:20] -------------------------------- [2018-11-18 09:11:25] Get build status ... Build https://travis-ci.org/test/zeppelin/builds/451036679 [1] OK https://travis-ci.org/test/zeppelin/jobs/451036680 [2] OK https://travis-ci.org/test/zeppelin/jobs/451036681 [3] Error 1 https://travis-ci.org/test/zeppelin/jobs/451036682 Tests run: 38, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 153.857 sec <<< FAILURE! - in org.apache.zeppelin.notebook.NotebookTest Please check full log at https://api.travis-ci.org/v3/job/451036682/log.txt [4] Not completed https://travis-ci.org/test/zeppelin/jobs/451036683 Your build has been stopped. Please check full log at https://api.travis-ci.org/v3/job/451036683/log.txt [5] OK https://travis-ci.org/test/zeppelin/jobs/451036684 [6] Error 1 https://travis-ci.org/test/zeppelin/jobs/451036685 Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 14.374 sec <<< FAILURE! - in org.apache.zeppelin.spark.SparkRInterpreterTest Please check full log at https://api.travis-ci.org/v3/job/451036685/log.txt [7] Error 1 https://travis-ci.org/test/zeppelin/jobs/451036686 Tests run: 15, Failures: 15, Errors: 0, Skipped: 0, Time elapsed: 216.32 sec <<< FAILURE! - in org.apache.zeppelin.interpreter.SparkIntegrationTest Please check full log at https://api.travis-ci.org/v3/job/451036686/log.txt [8] Error 1 https://travis-ci.org/test/zeppelin/jobs/451036687 Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.021 sec <<< FAILURE! - in org.apache.zeppelin.spark.SparkRInterpreterTest Please check full log at https://api.travis-ci.org/v3/job/451036687/log.txt [9] Error 1 https://travis-ci.org/test/zeppelin/jobs/451036688 Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 18.169 sec <<< FAILURE! - in org.apache.zeppelin.spark.SparkRInterpreterTest Please check full log at https://api.travis-ci.org/v3/job/451036688/log.txt [10] OK https://travis-ci.org/test/zeppelin/jobs/451036689 [11] Error 1 https://travis-ci.org/test/zeppelin/jobs/451036690 Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 353.585 sec <<< FAILURE! - in org.apache.zeppelin.livy.LivyInterpreterIT Please check full log at https://api.travis-ci.org/v3/job/451036690/log.txt 7 job(s) failed, 0 job(s) running/pending ``` ### What type of PR is it? Improvement ### How should this be tested? * try script manually against some build e.g. ``` python ./travis_check.py [github username] [commit hash] python ./travis_check.py NicolasRouquette ee6d89b ``` Author: Lee moon soo <leemoonsoo@gmail.com> Closes apache#3231 from Leemoonsoo/ci_summary and squashes the following commits: 9f42b32 [Lee moon soo] print build log summary
### What is this PR for? Paragraph cancel is broken due to previous code refactoring, This PR fix it in straightforward way and add system test as well. ### What type of PR is it? [Bug Fix ] ### Todos * [ ] - Task ### What is the Jira issue? * https://jira.apache.org/jira/browse/ZEPPELIN-3865 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes apache#3228 from zjffdu/ZEPPELIN-3865 and squashes the following commits: 574a1b9 [Jeff Zhang] ZEPPELIN-3865. Paragraph cancel is broken
### What is this PR for? This is due to another version of kryo in zeppelin-interpreter-api, this PR just exclude this and also add test to avoid regression. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://jira.apache.org/jira/browse/ZEPPELIN-3871 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes apache#3237 from zjffdu/ZEPPELIN-3871 and squashes the following commits: fac84d4 [Jeff Zhang] ZEPPELIN-3871. Spark interpreter doesn't work with KryoSerializer
### What is this PR for? By default, spark.executor.memory is not set in spark's interpreter setting, if I set it in spark-defaults.xml, then it won't take effect. Because spark.executor.memory in spark's interpreter setting will override that value in spark-defaults.xml. This doesn't make sense, we should only override property in spark-default.xml when you have non-empty value. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://jira.apache.org/jira/browse/ZEPPELIN-3870 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes apache#3238 from zjffdu/ZEPPELIN-3870 and squashes the following commits: b6c0017 [Jeff Zhang] ZEPPELIN-3870. Configuration in spark-defaults.conf doesn't take effect when it is empty in interpreter setting
### What is this PR for? This PR provides multithreading realization of interpreters closing. Benchmark based on [JMH](http://openjdk.java.net/projects/code-tools/jmh/) - `ZeppelinServerBenchmarkTest`: * Creates `InterpreterSettingManager` and `InterpreterFactory`; * Runs 6 interpreters (`python`, `sh`, `md`, `groovy`, `angular`, `spark`) for each user in isolated mode; * Measures InterpreterSettingManager closing. Result with usersCnt - 7, 5, 3, 1 (42, 30, 18, 6 interpreter processes): ``` Benchmark (usersCnt) Mode Cnt Score Error Units ZeppelinServerBenchmarkTest.testDefaultServerShutdown 7 avgt 10 16,690 ± 0,191 s/op ZeppelinServerBenchmarkTest.testDefaultServerShutdown 5 avgt 10 11,532 ± 0,064 s/op ZeppelinServerBenchmarkTest.testDefaultServerShutdown 3 avgt 10 6,942 ± 0,020 s/op ZeppelinServerBenchmarkTest.testDefaultServerShutdown 1 avgt 10 2,348 ± 0,018 s/op ZeppelinServerBenchmarkTest.testServerShutdownInParallel 7 avgt 10 3,612 ± 0,317 s/op ZeppelinServerBenchmarkTest.testServerShutdownInParallel 5 avgt 10 2,635 ± 0,075 s/op ZeppelinServerBenchmarkTest.testServerShutdownInParallel 3 avgt 10 2,473 ± 0,072 s/op ZeppelinServerBenchmarkTest.testServerShutdownInParallel 1 avgt 10 2,378 ± 0,029 s/op ``` [benchmark.zip](https://github.com/TinkoffCreditSystems/zeppelin/files/2591694/benchmark.zip) contains: * `Benchmark.patch` - patch with `ZeppelinServerBenchmarkTest`; * `benchmark.csv` - benchmark result;  * `benchmark-result.json` - results (you could visualize it using [JMH Visualizer](http://jmh.morethan.io/))  ### How to run benchmark 1. Apply patch from `benchmark.zip` 2. Add [IntelliJ IDEA JMH Plugin](https://plugins.jetbrains.com/plugin/7529-jmh-plugin) 3. Build Zeppelin 4. Run ZeppelinServerBenchmarkTest in IntelliJ IDEA ### What type of PR is it? Refactoring ### What is the Jira issue? [[ZEPPELIN-3869]](https://issues.apache.org/jira/browse/ZEPPELIN-3869) ### How should this be tested? * CI pass ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: George Klimov <klim.electronicmail@gmail.com> Closes apache#3229 from egorklimov/ZP-71 and squashes the following commits: 1cad17a [George Klimov] Remove watchdog check e7abdd3 [George Klimov] Fix issues 4bab30f [George Klimov] Add exceptions 380f539 [George Klimov] Refactor code efacb77 [George Klimov] Fix code style issues becfdf1 [George Klimov] Parallel close
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is this PR for?
Get all upstream changes up to 04b726a
What type of PR is it?
Synchronization
Todos
What is the Jira issue?
How should this be tested?
Screenshots (if appropriate)
Questions: