-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ZEPPELIN-1258] Add Spark packages support to Livy interpreter #1255
Conversation
update from original
merge master
I would not recommend user to do that to add extra library. Because most of time zeppelin user don't know the cluster details like which/where extra library are located in the cluster machines. I would suggest livy to support dynamic library loading as native spark interpreter. |
@zjffdu - Livy doesn't support dynamic library loading right now. So, I don't think there's a ZeppelinContext available in Livy. Would this be acceptable until dynamic library loading is added to Livy and a Zeppelin Context becomes available? As of now, Livy is not that much use if you cannot add libraries to your Spark app. |
@jimdowling Right, there's no ZeppelinContext in livy right now. Would allowing user to specify jars/packages when creating livy session more general ? (like --jars/--packages in spark-submit) |
Do you mean by modifying the UI on notebook creation or using a param to selecting the interpreter like |
No, I mean adding interpreter property to allow user to specify jars/packages. |
I think that the livy itself doesn't support add (--jars/--packages property in spark-submit) |
You can use |
@zjffdu I added |
@zjffdu do you have any further comments ? |
<table class="table-configuration"> | ||
<tr> | ||
<th>Property</th> | ||
<th>Default</th> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this wouldn't be the "Default" right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
True, it's just example.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's change it to Example?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@felixcheung done.
could you update the title of this PR and JIRA ZEPPELIN-1258, if we are changing how we would approach this? |
@felixcheung I suggests this title "Adding packages to livy interpreter", Is it Ok ? |
how about "Add Spark packages support to Livy interpreter"? |
@felixcheung done. |
@felixcheung do you have any further comments ? |
There is a test failure? |
@felixcheung : This PR does not change the code. It is just a change in the configurations. So, it should not affect the tests |
That's a fair point. I'll merge if there is no more discussion. |
What is this PR for?
Adding extra libraries to livy interpreter which isn't exist by default.
What type of PR is it?
[ Improvement ]
Todos
What is the Jira issue?
How should this be tested?
livy.spark.jars.packages
to list of maven coordinates of jars. The format for the coordinates should be groupId:artifactId:version.Screenshots (if appropriate)
Questions: