Allow the spark app name to be configurable

[craigching]

Allows spark.app.name to be configurable. This is useful when you have multiple zeppelin instances connected to the same spark cluster. It allows you to differentiate the instances in the spark ui.

[felixcheung]

Why not set spark.app.name from the interpreter config?

[craigching]

Hi Felix,

I started a discussion on this on the user list. Basically I am creating a zeppelin instance per user and having all instances, by default, use a spark cluster. When those zeppelin instances all talk to the cluster, it's hard to differentiate which is which. So I want to start them up with unique names so I can differentiate them. Asking users to make that change before they start using their instances is error-prone, especially considering I'm trying to introduce them to spark via zeppelin and want them to focus on spark.

Cheers,
Craig

On Nov 8, 2015, at 14:28, Felix Cheung notifications@github.com wrote:

In spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java:

@@ -90,7 +90,9 @@
"spark",
SparkInterpreter.class.getName(),
new InterpreterPropertyBuilder()

•        .add("spark.app.name", "Zeppelin", "The name of spark application.")
  

•        .add("spark.app.name",
  

  Why not set spark.app.name from the interpreter config?

—
Reply to this email directly or view it on GitHub.

[felixcheung]

Sure. If you are to set SPARK_APP_NAME in zeppelin-env.sh, you could also set spark.app.name in interpreter.json also in /conf?

[craigching]

Hi Felix,

But I also set MASTER (spark master) that way too. It seems you guys have a lot of different ways to set things and I was just trying to do what was accepted. The way I start zeppelin (I run zeppelin in a docker container), it's much easier to create the env file on the fly than to have to parse a large-ish json file. But whatever, I asked on the user list and moon thought it might be a good idea, I just want to be consistent. This seems like an area that you could tighten up, which is the correct way to specify these settings, zeppelin-env.sh, zeppelin-site.xml, or interpreter.json. Certainly interpreter.json is not the documented way to make these settings, at least it wasn't obvious to me.

But whatever, I'll retract the PR then.

Cheers,
Craig

Sent from my iPad

On Nov 8, 2015, at 16:14, Felix Cheung notifications@github.com wrote:

In spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java:

@@ -90,7 +90,9 @@
"spark",
SparkInterpreter.class.getName(),
new InterpreterPropertyBuilder()

•        .add("spark.app.name", "Zeppelin", "The name of spark application.")
  

•        .add("spark.app.name",
  

  Sure. If you are to set SPARK_APP_NAME in zeppelin-env.sh, you could also set spark.app.name in interpreter.json also in /conf?

—
Reply to this email directly or view it on GitHub.

[craigching]

Thinking about it just a bit more, it would be nice to have the env setting be a way of changing the default, but still allow the user to override it themselves. If I set it in interpreters.json (as well as spark master that way), I'll have to keep track of the first time I run vs. subsequent runs so I don't interfere with user settings. I can imagine that users may want to, eventually, start their own spark instance/cluster to run against and override that setting.

Cheers,
Craig

On Nov 8, 2015, at 16:29, Craig Ching craigching@gmail.com wrote:

Hi Felix,

But I also set MASTER (spark master) that way too. It seems you guys have a lot of different ways to set things and I was just trying to do what was accepted. The way I start zeppelin (I run zeppelin in a docker container), it's much easier to create the env file on the fly than to have to parse a large-ish json file. But whatever, I asked on the user list and moon thought it might be a good idea, I just want to be consistent. This seems like an area that you could tighten up, which is the correct way to specify these settings, zeppelin-env.sh, zeppelin-site.xml, or interpreter.json. Certainly interpreter.json is not the documented way to make these settings, at least it wasn't obvious to me.

But whatever, I'll retract the PR then.

Cheers,
Craig

Sent from my iPad

On Nov 8, 2015, at 16:14, Felix Cheung notifications@github.com wrote:

In spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java:

@@ -90,7 +90,9 @@
"spark",
SparkInterpreter.class.getName(),
new InterpreterPropertyBuilder()

•        .add("spark.app.name", "Zeppelin", "The name of spark application.")
  

•        .add("spark.app.name",
  

  Sure. If you are to set SPARK_APP_NAME in zeppelin-env.sh, you could also set spark.app.name in interpreter.json also in /conf?

—
Reply to this email directly or view it on GitHub.

[felixcheung]

ok.
I think the different mechanics facilitate different approaches to config management.
It will good if you could share and discuss with the community with which works well.

[craigching]

yeah, I think there are two different users here. One is a site admin and they need to be able to set reasonable defaults for their environment (spark master, spark app name, etc.). Then there are regular users who may want to override the defaults once they have a better understanding of zeppelin, spark and other interpreters.

So I did start a conversation on users, that's what lead to this PR. I guess I can take a different tact, but ultimately I need to solve my problem now. I'll have to maintain a fork until there is some consensus I guess.

[corneadoug]

@craigching Is this PR still needed?

[craigching]

It's not. Someone else added this exact feature as part of another PR, so this has been implemented.