[ZEPPELIN-2582][DOCS] docs for interpreter binding modes #2437

1ambda · 2017-06-26T06:47:57Z

What is this PR for?

Updated interpreter_binding_mode.md since users are sometimes confused what this mode means and there is already opened JIRA issue. This documentation will be helpful to Zeppelin users.

disclaimer: content was copied from here with author's consent.

What type of PR is it?

[Documentation]

Todos

DONE

What is the Jira issue?

ZEPPELIN-2582

How should this be tested?

setup rvm 2.1.0+ and install required bundles for docs/
bundle exec jekyll serve --watch
http://localhost:4000/usage/interpreter/interpreter_binding_mode.html

Screenshots (if appropriate)

Questions:

Does the licenses files need update? - NO
Is there breaking changes for older versions? - NO
Does this needs documentation? - This PR is about docs.

1ambda · 2017-06-26T10:26:49Z

CI failed but irrelevant.

https://travis-ci.org/1ambda/zeppelin/jobs/246983015

cacti77 · 2017-06-26T14:36:55Z

Thanks for doing this. I looked at the screenshots above. The diagrams are very helpful! Are you also going to add the table I created at https://issues.apache.org/jira/browse/ZEPPELIN-2582?focusedCommentId=16024796&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16024796 or do you think it doesn't add anything useful?

1ambda · 2017-06-26T15:25:57Z

@cacti77 Yes, it would be a good summary. Let me add it.

1ambda · 2017-06-27T01:16:40Z

I added your summary table.

1ambda · 2017-06-27T02:14:01Z

CI failed but irrelevant.

zjffdu · 2017-06-27T02:50:00Z

It is better to mention in what dimension of these modes. (per user or per note).
And from the code perspective, there's only one InterpreterGroup for scoped mode, but multiple sessions in one InterpreterGroup.

1ambda · 2017-06-27T04:28:56Z

@zjffdu Thanks. Let me add the description into the doc.

1ambda · 2017-06-27T09:03:02Z

@zjffdu enhanced description according to your suggestion. could you check that?

zjffdu · 2017-06-27T09:19:02Z

docs/usage/interpreter/interpreter_binding_mode.md

+</div>
+<br/>
+
+In Scoped mode, Zeppelin still runs single interpreter JVM process but multiple Interpreter Group serve each Note. 


Zeppelin still runs single interpreter JVM process but multiple Interpreter Group serve each Note.

This is only correct when set scoped per note. As in another comment, we should mention in which dimension the scope is. And what does it mean for Per User and Per Note. The same for isolated mode.

Here is the description for that. If we need to repeat again there, I will add. Do we need?

1ambda · 2017-06-27T09:45:23Z

@zjffdu Thanks for the review.

I emphasized per note in the description and added code perspective to the isolated mode as well.

Now it looks like

1ambda · 2017-06-28T00:37:56Z

CI failed but irrelevant.

zjffdu · 2017-06-28T01:56:55Z

So, each Note have their own dedicated session but still it’s possible to share objects between different Interpreter Groups while they’re in the same JVM process.

Is this correct ? AFAIK, only sparkcontext is shared in scoped mode. Variable defined in session 1 can not be used in session 2

From the code perspective, there is only one InterpreterGroup for the scoped mode, but multiple sessions in one InterpreterGroup.

Sorry, I think we can remove this before we find a more proper way to explain. This might confuse users too.

1ambda · 2017-06-28T02:20:12Z

@zjffdu regarding that, I have a question as well.

In isolated mode / shared mode, can't we share variables via z.put? and z.get?

zjffdu · 2017-06-28T02:23:37Z

We can, but this is via ResourcePool. We might need to say that we could share object via resource pool. Otherwise users might be confused.

1ambda · 2017-06-28T02:47:58Z

@zjffdu yes, it would be more clear.

@jongyoul. As zjffdu said, variables (e.g scala variables, ...) in the scope mode can't be shared directly. is it correct?

jongyoul · 2017-06-28T02:49:46Z

Yes, correct. Except Shared, user cannot share anything directly.

1ambda · 2017-06-28T02:57:49Z

@jongyoul Thanks!

I fixed the incorrect description. Now it looks like

1ambda · 2017-07-03T13:50:59Z

Thanks for the review.

@felixcheung Resolved all comments.
@zjffdu @cacti77 Update imgs and description. Please check the last 2 commits.

cacti77 · 2017-07-03T15:27:37Z

docs/usage/interpreter/interpreter_binding_mode.md

+Mode | Each notebook...	| Benefits | Disadvantages | Sharing objects
+--- | --- | --- | --- | ---
+**shared** | Shares a single sessions in a single interpreter process (JVM) | Low resource utilization and Easy to share data between notebooks | All notebooks are affected if Interpreter Process dies | can share directly
+**scoped** | Has its own note sessions in the same Interpreter Process (JVM) | Less resource utilization than isolated mode | All notebooks are affected if Interpreter Process dies | can't share directly, but possible to share objets via [ResourcePool](../../interpreter/spark.html#object-exchange)) 


scoped | Has its own session in the same interpreter process (JVM) | Less resource utilization than isolated mode | All notebooks are affected if the interpreter process dies | Can't share directly, but it's possible to share objects via ResourcePool

cacti77 · 2017-07-03T15:27:55Z

docs/usage/interpreter/interpreter_binding_mode.md

+--- | --- | --- | --- | ---
+**shared** | Shares a single sessions in a single interpreter process (JVM) | Low resource utilization and Easy to share data between notebooks | All notebooks are affected if Interpreter Process dies | can share directly
+**scoped** | Has its own note sessions in the same Interpreter Process (JVM) | Less resource utilization than isolated mode | All notebooks are affected if Interpreter Process dies | can't share directly, but possible to share objets via [ResourcePool](../../interpreter/spark.html#object-exchange)) 
+**isolated** | Has its own Interpreter Process | One notebook not affected directly by other notebooks (**per note**) | Can't share data between notebooks easily (**per note**) | can't share directly, but possible to share objets via [ResourcePool](../../interpreter/spark.html#object-exchange)) 


isolated | Has its own Interpreter Process | One notebook is not affected directly by other notebooks (per note) | Can't share data between notebooks easily (per note) | Can't share directly, but it's possible to share objects via ResourcePool

cacti77 · 2017-07-03T15:45:30Z

docs/usage/interpreter/interpreter_binding_mode.md

+<br/>
+
+Interpreter is a JVM process that communicates with Zeppelin daemon using thrift. 
+Each Interpreter process can have a interpreter group, and each interpreter instance belongs to this interpreter group.


Each interpreter process has a single interpreter group, and this interpreter group can have one or more instances of an interpreter.

cacti77 · 2017-07-03T15:46:55Z

docs/usage/interpreter/interpreter_binding_mode.md

+(See [here](../../development/writing_zeppelin_interpreter.html) to understand more about its internal structure.) 
+
+Zeppelin provides 3 different modes to run interpreter process: **shared**, **scoped** and **isolated**.   
+Also, user can specify the scope of these mode as well: **per user** or **per note**.  


Also, the user can specify the scope of these modes: per user or per note.

cacti77 · 2017-07-03T15:48:25Z

docs/usage/interpreter/interpreter_binding_mode.md

+Also, user can specify the scope of these mode as well: **per user** or **per note**.  
+These 3 modes give flexibility to fit Zeppelin into any type of use cases.
+
+In this documentation, we mainly discuss the combination of **per note** mode with **shared**, **scoped** and **isolated** modes for explanation. 


In this documentation, we mainly discuss the per note scope in combination with the shared, scoped and isolated modes.

cacti77 · 2017-07-03T15:54:03Z

docs/usage/interpreter/interpreter_binding_mode.md

+<br/>
+
+In Scoped mode, Zeppelin still runs single interpreter JVM process but multiple sessions serve each note. (in case of **per note**) 
+So, each note have their own dedicated session. (but still possible to share objects via [ResourcePool](../../interpreter/spark.html#object-exchange)) 


In Scoped mode, Zeppelin still runs a single interpreter JVM process but, in the case of per note scope, each note runs in its own dedicated session. (Note it is still possible to share objects between these notes via ResourcePool.)

cacti77 · 2017-07-03T15:55:27Z

docs/usage/interpreter/interpreter_binding_mode.md

+<br/>
+
+**Isolated** mode runs separate interpreter process for each note. (in case of **per note**) 
+So, each note have absolutely isolated session. (but still possible to share objects via [ResourcePool](../../interpreter/spark.html#object-exchange)) 


Isolated mode runs a separate interpreter process for each note in the case of per note scope.
So, each note has an absolutely isolated session. (But it is still possible to share objects via ResourcePool.)

cacti77 · 2017-07-03T15:57:14Z

docs/usage/interpreter/interpreter_binding_mode.md

+
+Mode | Each notebook...	| Benefits | Disadvantages | Sharing objects
+--- | --- | --- | --- | ---
+**shared** | Shares a single sessions in a single interpreter process (JVM) | Low resource utilization and Easy to share data between notebooks | All notebooks are affected if Interpreter Process dies | can share directly


shared | Shares a single session in a single interpreter process (JVM) | Low resource utilization and it's easy to share data between notebooks | All notebooks are affected if the interpreter process dies | Can share directly

cacti77 · 2017-07-03T16:01:36Z

docs/usage/interpreter/interpreter_binding_mode.md

+**scoped** | Has its own note sessions in the same Interpreter Process (JVM) | Less resource utilization than isolated mode | All notebooks are affected if Interpreter Process dies | can't share directly, but possible to share objets via [ResourcePool](../../interpreter/spark.html#object-exchange)) 
+**isolated** | Has its own Interpreter Process | One notebook not affected directly by other notebooks (**per note**) | Can't share data between notebooks easily (**per note**) | can't share directly, but possible to share objets via [ResourcePool](../../interpreter/spark.html#object-exchange)) 
+
+In case of **per user** (available on multi-user environment), Zeppelin manages interpreter sessions per user. For example,


In the case of the per user scope (available in a multi-user environment), Zeppelin manages interpreter sessions on a per user basis rather than a per note basis. For example:

cacti77 · 2017-07-03T16:03:07Z

docs/usage/interpreter/interpreter_binding_mode.md

+<br/>
+
+Each Interpreter implementation may have different characteristics depending on the back end system that they integrate. And 3 interpreter modes can be used differently.
+Let’s take a look how Spark Interpreter implementation uses these 3 interpreter modes with **per note** mdoe, as an example. 


Let’s take a look how Spark Interpreter implementation uses these 3 interpreter modes with per note mode, as an example.

cacti77 · 2017-07-03T16:04:55Z

docs/usage/interpreter/interpreter_binding_mode.md

+
+In Scoped mode, each note has its own Scala REPL. 
+So variable defined in a note can not be read or overridden in another note. 
+However, still single SparkContext serves all the sessions. 


However, a single SparkContext still serves all the sessions.

cacti77 · 2017-07-03T16:05:28Z

docs/usage/interpreter/interpreter_binding_mode.md

+In Scoped mode, each note has its own Scala REPL. 
+So variable defined in a note can not be read or overridden in another note. 
+However, still single SparkContext serves all the sessions. 
+And all the jobs are submitted to this SparkContext and fair scheduler schedules the job. 


And all the jobs are submitted to this SparkContext and the fair scheduler schedules the jobs.

cacti77 · 2017-07-03T16:10:52Z

@1ambda Thanks for finishing the diagrams. I would probably put the SparkContext inside the interpreter process but outside the InterpreterGroup; but if @zjffdu is happy with the diagrams I guess it doesn't matter. I've gone through interpreter_binding_mode.md, tidying up the English some more. I'm done with my edits now, thanks.

cacti77 · 2017-07-04T08:27:48Z

Apologies @1ambda for not interacting with this PR properly; I may have overlooked some of your recent edits in the version of interpreter_binding_mode.md I commented on yesterday.

1ambda · 2017-07-05T01:10:32Z

resolved @cacti77's all comments

zjffdu · 2017-07-05T05:20:01Z

docs/usage/interpreter/interpreter_binding_mode.md


+<br/>
+
+Interpreter is a JVM process that communicates with Zeppelin daemon using thrift. 


Interpreter -> Interpreter Process

1ambda · 2017-07-05T06:06:56Z

resolved @zjffdu's comment

1ambda · 2017-07-05T12:48:06Z

Please let me know if we need to add / improve / modify something.

zjffdu · 2017-07-05T13:28:30Z

LGTM

cacti77 · 2017-07-05T14:58:28Z

@1ambda And it looks good to me too; thanks for being patient and making all those changes!

1ambda · 2017-07-05T15:02:17Z

it was an honor to make more correct, concise documentation with your help.

Merge if no more discussion.

cacti77 · 2017-07-06T08:55:54Z

@1ambda One question please: Do these docs constitute a new web page in Zeppelin's docs? I'm just wondering how a user would navigate to this new information on binding modes from the Docs home page. Will the new page be linked to from under http://zeppelin.apache.org/docs/0.7.2/manual/interpreters.html#interpreter-binding-mode for example? Or will it become part of the interpreters.html page itself (i.e., Interpreter > Overview)?

1ambda · 2017-07-06T09:29:41Z

@cacti77 it's provided as of 0.8.0-SNAPSHOT.

http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/usage/interpreter/interpreter_binding_mode.html

here is the background

we re-organized contents structure of docs for 0.8.0-SNAPSHOT
in [ZEPPELIN-2596] Improving documentation page #2371
the interpreter binding page was created at that time (so it's currently empty)

cacti77 · 2017-07-06T09:41:59Z

Thanks @1ambda . So, looking at #2371, it looks like a user would simply click on the dropdown and then Interpreter > Interpreter Binding Mode to access the new web page. Is that correct? Basically, as long as all the info about interpreter binding modes is in one place and easy to find, that's all I care about:).

1ambda · 2017-07-07T07:22:08Z

Sure, users can easily access the doc.

1ambda · 2017-07-07T07:29:13Z

Deployed

http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/usage/interpreter/interpreter_binding_mode.html

1ambda added 2 commits June 26, 2017 15:42

feat: Add images

5683620

docs: Update interpreter binding mode page

b43a4da

docs: Add summary table of @cacti77

43e8b02

1ambda closed this Jun 27, 2017

1ambda reopened this Jun 27, 2017

1ambda added 2 commits June 27, 2017 18:00

docs: Add descroptoin for per user, per note

1eafb64

docs: enhance desc for scoped mode

83eb2e5

zjffdu reviewed Jun 27, 2017

View reviewed changes

1ambda added 2 commits June 27, 2017 18:42

docs: Enhance description for ioslated mode

66d02ba

docs: emphasize per note mode in sopced, iolsated

9ae9552

1ambda added 3 commits June 28, 2017 11:51

docs: Use lowercase for word note

af44226

fix: Remove code perspective

729b56a

docs: Fix incorrect info for scoped mode, isolated mode

316c563

fix: typo

33b0736

cacti77 reviewed Jul 3, 2017

View reviewed changes

fix: Correct sentences

a60fdbf

zjffdu reviewed Jul 5, 2017

View reviewed changes

fix: Docs

e0bcf26

asfgit closed this in 86bc933 Jul 7, 2017


		<br/>

		Interpreter is a JVM process that communicates with Zeppelin daemon using thrift.

[ZEPPELIN-2582][DOCS] docs for interpreter binding modes #2437

[ZEPPELIN-2582][DOCS] docs for interpreter binding modes #2437

Conversation

1ambda commented Jun 26, 2017

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

1ambda commented Jun 26, 2017

cacti77 commented Jun 26, 2017

1ambda commented Jun 26, 2017

1ambda commented Jun 27, 2017

1ambda commented Jun 27, 2017

zjffdu commented Jun 27, 2017

1ambda commented Jun 27, 2017

1ambda commented Jun 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

1ambda commented Jun 27, 2017 • edited

1ambda commented Jun 28, 2017

zjffdu commented Jun 28, 2017

1ambda commented Jun 28, 2017 • edited

zjffdu commented Jun 28, 2017

1ambda commented Jun 28, 2017

jongyoul commented Jun 28, 2017

1ambda commented Jun 28, 2017 • edited

1ambda commented Jul 3, 2017 • edited

cacti77 Jul 3, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cacti77 Jul 3, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cacti77 Jul 3, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cacti77 Jul 3, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cacti77 Jul 3, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cacti77 commented Jul 3, 2017

cacti77 commented Jul 4, 2017

1ambda commented Jul 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

1ambda commented Jul 5, 2017

1ambda commented Jul 5, 2017

zjffdu commented Jul 5, 2017

cacti77 commented Jul 5, 2017

1ambda commented Jul 5, 2017

cacti77 commented Jul 6, 2017 • edited

1ambda commented Jul 6, 2017 • edited

cacti77 commented Jul 6, 2017

1ambda commented Jul 7, 2017

1ambda commented Jul 7, 2017

1ambda commented Jun 27, 2017 •

edited

1ambda commented Jun 28, 2017 •

edited

1ambda commented Jun 28, 2017 •

edited

1ambda commented Jul 3, 2017 •

edited

cacti77 Jul 3, 2017 •

edited

cacti77 Jul 3, 2017 •

edited

cacti77 Jul 3, 2017 •

edited

cacti77 Jul 3, 2017 •

edited

cacti77 Jul 3, 2017 •

edited

cacti77 commented Jul 6, 2017 •

edited

1ambda commented Jul 6, 2017 •

edited