ZEPPELIN-143: Git as a versioned notebook storage #497

bzz · 2015-12-01T08:48:47Z

This is very basic implementation of the ZEPPELIN-143 at the backend.

It makes a local git repository our of your /notebook dir and commits a new revision for each save/update.

It does not:

add any remotes to the git repo. It is totally possible to do that manually though. It would be interesting to add this later, to be able to push the notebook to hostings like GH
have any GUI modifications. It is left as further work, to add the ability for a user to switch "versions" of the notebook, navigating between previous runs.

Feedback is very welcome!

bzz · 2015-12-01T08:55:26Z

\cc @Leemoonsoo @jongyoul @felixcheung @khalidhuseynov for a review

bzz · 2015-12-01T09:36:49Z

CI fails with something, that looks not very relevant to the changes in this PR. Can not reproduce on my local env too.

Results :

Failed tests: 
  NotebookTest.testAbortParagraphStatusOnInterpreterRestart:347 expected:<ABORT> but was:<RUNNING>

Tests run: 34, Failures: 1, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Zeppelin ........................................... SUCCESS [ 12.289 s]
[INFO] Zeppelin: Interpreter .............................. SUCCESS [01:18 min]
[INFO] Zeppelin: Zengine .................................. FAILURE [  8.411 s]

.....

Tests run: 10, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.662 sec <<< FAILURE! - in org.apache.zeppelin.notebook.NotebookTest
testAbortParagraphStatusOnInterpreterRestart(org.apache.zeppelin.notebook.NotebookTest)  Time elapsed: 0.108 sec  <<< FAILURE!
java.lang.AssertionError: expected:<ABORT> but was:<RUNNING>
    at org.junit.Assert.fail(Assert.java:88)
    at org.junit.Assert.failNotEquals(Assert.java:743)
    at org.junit.Assert.assertEquals(Assert.java:118)
    at org.junit.Assert.assertEquals(Assert.java:144)
    at org.apache.zeppelin.notebook.NotebookTest.testAbortParagraphStatusOnInterpreterRestart(NotebookTest.java:347)

khalidhuseynov · 2015-12-01T15:16:47Z

This CI problem should be solved by #468, may need to rebase for that

felixcheung · 2015-12-01T20:13:56Z

docs/storage/storage.md

@@ -19,10 +19,27 @@ limitations under the License.
 -->
 ### Notebook Storage

-In Zeppelin there are two option for storage Notebook, by default the notebook is storage in the notebook folder in your local File System and the second option is S3.
+Zeppelin a pluggable notebook storage mechanism with multiple implementations.
+There are few Notebook storage options avaialble for a use,


Is it possible to have multiple storage simultaneously? Could we clarify in this doc?

... multiple storage enabled simultaneously...

felixcheung · 2015-12-01T20:32:01Z

looks good! thanks for working on this. looking forward to git push ;)

jeffsteinmetz · 2015-12-02T05:06:51Z

Is there a document (or could one be added) that describes setting up the Git repo integration (git ssh or https path, connecting Zeppelin to a specific git repo, passwords, ssh keys, etc.)?

bzz · 2015-12-02T08:48:39Z

Guys, thanks for reviews, appreciate that a lot!

Will address each of them in next couple of days.

felixcheung · 2015-12-02T19:51:19Z

@jeffsteinmetz I think this is committing to local only, no remote repo support yet

bzz · 2015-12-03T02:46:11Z

@jeffsteinmetz @felixcheung well noted, this PR introduces automation to keep notebook dir inside the git repository in your local filesystem.

Right now, this repository does not have any 'remotes' setup, and Zeppelin does not push it anywhere, so there are no Zeppelin configuration needed (except documented property change).

It is possible and quite simple to sync this local repo manually by cd notebooks; git add remote ...; git push, as well as I can add this ability to Zeppelin itself, if people find it useful (in the separate PR).

bzz · 2015-12-03T03:32:11Z

@khalidhuseynov thanks for looking into CI problem, that's a bummer!

This CI problem should be solved by #468, may need to rebase for that

It looks like it was merged 6 days ago, so current branch should be on top of it already, but just in case - I have synced with the latest master and it seems to pass! If that repeats - will file the jira issue with label 'flaky-test'.

bzz · 2015-12-03T04:58:46Z

Have addressed all the reviews, please let me know if something is missing.

In 5d7ffea .close() lifecycle method was introduced for the NotebookRero to cleanup the resources that storage implementation might have occupied.
As this is an API change \cc @khalidhuseynov and @Leemoonsoo who worked on this before, as I recall, to take a quick look.

bzz · 2015-12-03T10:44:13Z

New CI failure - cassandra interpreter tests (one more candidate for flaky-test label in JIRA)

[INFO] Zeppelin: Lens interpreter ......................... SUCCESS [  3.570 s]
[INFO] Zeppelin: Cassandra ................................ FAILURE [ 33.880 s]

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 21.414 sec <<< FAILURE! - in org.apache.zeppelin.cassandra.CassandraInterpreterTest
org.apache.zeppelin.cassandra.CassandraInterpreterTest  Time elapsed: 21.414 sec  <<< ERROR!
java.lang.ExceptionInInitializerError: null
    at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
    at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:269)
    at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:183)
    at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)
    at info.archinnov.achilles.script.ScriptExecutor.executeScript(ScriptExecutor.java:58)
    at info.archinnov.achilles.embedded.AchillesInitializer.executeStartupScripts(AchillesInitializer.java:194)
    at info.archinnov.achilles.embedded.AchillesInitializer.initialize(AchillesInitializer.java:97)
    at info.archinnov.achilles.embedded.AchillesInitializer.initializeFromParameters(AchillesInitializer.java:60)
    at info.archinnov.achilles.embedded.CassandraEmbeddedServer.<init>(CassandraEmbeddedServer.java:76)
    at info.archinnov.achilles.junit.AchillesResource.initResource(AchillesResource.java:59)
    at info.archinnov.achilles.junit.AchillesResource.<init>(AchillesResource.java:52)
    at info.archinnov.achilles.junit.AchillesResourceBuilder.build(AchillesResourceBuilder.java:205)
    at org.apache.zeppelin.cassandra.CassandraInterpreterTest.<clinit>(CassandraInterpreterTest.java:62)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9267 (com.datastax.driver.core.OperationTimedOutException: [localhost/127.0.0.1:9267] Operation timed out))
    at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:216)
    at com.datastax.driver.core.RequestHandler.access$900(RequestHandler.java:45)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.sendRequest(RequestHandler.java:276)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution$1.run(RequestHandler.java:374)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

bzz · 2015-12-03T10:44:26Z

re-triggering CI build

bzz · 2015-12-03T15:43:10Z

CI passes now, ready to be merged.

khalidhuseynov · 2015-12-03T17:05:48Z

@bzz thanks for starting this feature! .close() is required in this case and is fine to add to API. LGTM

Leemoonsoo · 2015-12-04T07:57:56Z

@bzz Tested and working really well. And the API looks fine.

Could you also take care zeppelin-distribution/src/bin_license/LICENSE for newly added dependency, jgit? jgit and it's transitive dependency looks like

+- org.eclipse.jgit:org.eclipse.jgit:jar:4.1.1.201511131810-r:compile
  +- com.jcraft:jsch:jar:0.1.53:compile
  +- com.googlecode.javaewah:JavaEWAH:jar:0.7.9:compile
  +- org.apache.httpcomponents:httpclient:jar:4.3.6:compile
  \- org.eclipse.jdt:org.eclipse.jdt.annotation:jar:1.1.0:compile

bzz · 2015-12-07T02:06:54Z

@Leemoonsoo thank you for kind reminder! Will do it and ping back

bzz · 2015-12-07T06:22:04Z

Thank all participants for kind reviews!

@Leemoonsoo could you check 468a858 and let me know if you think that is enough?

Leemoonsoo · 2015-12-07T21:08:38Z

Thanks @bzz Looks good to me!

bzz · 2015-12-08T01:34:08Z

Thank you!
Merging if there is no more discussion

p0wl · 2017-07-26T15:26:57Z

Thanks for the feature!

Any news on automatic github integration? Or a guidline how to setup github sync?

herval · 2017-07-26T17:29:06Z

@p0wl I started remote git support here: https://github.com/herval/zeppelin/tree/remote-git-support - it's a WIP, if someone wants to take over (it would work w/ github or any other git repo)

Gauravshah · 2017-11-09T07:11:06Z

@herval were you able to finish it ?

herval · 2017-11-23T19:37:29Z

Haven't had the time to look into that further. TBH, I don't think storing notebooks as text files is a good strategy - not in the environment we use Zeppelin, anyway. We're likely moving towards some sort of centralized storage (maybe a KV storage), in order to be able to scale horizontally

bzz added 8 commits December 1, 2015 17:22

Small cleanup: JavaDoc, annotations, warning for NotebookRepos

cf0b4bc

ZEPPELIN-143: WIP - very basic sketch of GitNotebookRepo

c21b3e6

ZEPPELIN-143: adding JGit as initial impl

20a93a4

ZEPPELIN-143: simple case save() implemented

bdee039

ZEPPELIN-143: initial repo creation + tests

4ef879a

ZEPPELIN-143: .history() implemented

274830f

ZEPPELIN-143: docs and conf updated

8d66c51

ZEPPELIN-143: last-minute cleanup

85de737

felixcheung reviewed Dec 1, 2015
View reviewed changes

Merge branch 'master' into add-git-notebook-repo

94720d9

bzz added 3 commits December 3, 2015 12:45

ZEPPELIN-143: general Notebook storage doc update

fca2300

ZEPPELIN-143: muting down the logs

978e62c

ZEPPELIN-143: introduce NotebookRepo.close()

5d7ffea

bzz closed this Dec 3, 2015

bzz reopened this Dec 3, 2015

ZEPPELIN-143: add all new dependencies to LICENCE

468a858

asfgit closed this in b5e2e62 Dec 8, 2015

bzz deleted the add-git-notebook-repo branch December 8, 2015 02:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ZEPPELIN-143: Git as a versioned notebook storage #497

ZEPPELIN-143: Git as a versioned notebook storage #497

bzz commented Dec 1, 2015

bzz commented Dec 1, 2015

bzz commented Dec 1, 2015

khalidhuseynov commented Dec 1, 2015

felixcheung Dec 1, 2015

felixcheung Dec 3, 2015

felixcheung commented Dec 1, 2015

jeffsteinmetz commented Dec 2, 2015

bzz commented Dec 2, 2015

felixcheung commented Dec 2, 2015

bzz commented Dec 3, 2015

bzz commented Dec 3, 2015

bzz commented Dec 3, 2015

bzz commented Dec 3, 2015

bzz commented Dec 3, 2015

bzz commented Dec 3, 2015

khalidhuseynov commented Dec 3, 2015

Leemoonsoo commented Dec 4, 2015

bzz commented Dec 7, 2015

bzz commented Dec 7, 2015

Leemoonsoo commented Dec 7, 2015

bzz commented Dec 8, 2015

p0wl commented Jul 26, 2017

herval commented Jul 26, 2017

Gauravshah commented Nov 9, 2017

herval commented Nov 23, 2017

ZEPPELIN-143: Git as a versioned notebook storage #497

ZEPPELIN-143: Git as a versioned notebook storage #497

Conversation

bzz commented Dec 1, 2015

bzz commented Dec 1, 2015

bzz commented Dec 1, 2015

khalidhuseynov commented Dec 1, 2015

felixcheung Dec 1, 2015

Choose a reason for hiding this comment

felixcheung Dec 3, 2015

Choose a reason for hiding this comment

felixcheung commented Dec 1, 2015

jeffsteinmetz commented Dec 2, 2015

bzz commented Dec 2, 2015

felixcheung commented Dec 2, 2015

bzz commented Dec 3, 2015

bzz commented Dec 3, 2015

bzz commented Dec 3, 2015

bzz commented Dec 3, 2015

bzz commented Dec 3, 2015

bzz commented Dec 3, 2015

khalidhuseynov commented Dec 3, 2015

Leemoonsoo commented Dec 4, 2015

bzz commented Dec 7, 2015

bzz commented Dec 7, 2015

Leemoonsoo commented Dec 7, 2015

bzz commented Dec 8, 2015

p0wl commented Jul 26, 2017

herval commented Jul 26, 2017

Gauravshah commented Nov 9, 2017

herval commented Nov 23, 2017