[LIVY-414] Support environment variables for interactive and batch sessions#80
[LIVY-414] Support environment variables for interactive and batch sessions#80risdenk wants to merge 1 commit into
Conversation
|
Test failures: Will fix in a few minutes once all tests finish running locally. |
Codecov Report
@@ Coverage Diff @@
## master #80 +/- ##
===========================================
+ Coverage 71.47% 71.6% +0.12%
- Complexity 793 800 +7
===========================================
Files 97 97
Lines 5396 5413 +17
Branches 798 804 +6
===========================================
+ Hits 3857 3876 +19
+ Misses 1015 1011 -4
- Partials 524 526 +2
Continue to review full report at Codecov.
|
| (if (queue.isDefined) s"queue: ${queue.get}, " else "") + | ||
| (if (name.isDefined) s"name: ${name.get}, " else "") + | ||
| (if (conf.nonEmpty) s"conf: ${conf.mkString(",")}]" else "]") | ||
| (if (conf.nonEmpty) s"conf: ${conf.mkString(",")}]" else "]") + |
There was a problem hiding this comment.
I'll review more later, but quick nit: the ]s here needs to be removed since the line below now adds them
There was a problem hiding this comment.
Ah thanks good catch. I'll fix in a minute.
|
Some improvements that might be necessary:
|
|
@risdenk Why do we only support batch session, I think it is also needed for interactive session, right? |
|
Manually tested the current PR with changing SPARK_HOME during the request. Setup
Simple Python test file from pyspark import SparkContext
sc = SparkContext()
print('Spark Version: ' + sc.version)Testing different curl -i -XPOST http://$(hostname -f):8998/batches -H 'Content-Type: application/json' -d '{"file": "/home/USER/test_livy/livy.py"}'
# Spark Version: 1.6.3SPARK_HOME set to Spark 1.6.3 curl -i -XPOST http://$(hostname -f):8998/batches -H 'Content-Type: application/json' -d '{"file": "/home/USER/test_livy/livy.py", "env" : {"SPARK_HOME": "/home/USER/sparks/spark-1.6.3-bin-hadoop2.6"}}'
# Spark Version: 1.6.3SPARK_HOME set to Spark 2.2.1 curl -i -XPOST http://$(hostname -f):8998/batches -H 'Content-Type: application/json' -d '{"file": "/home/USERNAME/test_livy/livy.py", "env" : {"SPARK_HOME": "/home/USER/sparks/spark-2.2.1-bin-hadoop2.7"}}'
# Spark Version: 2.2.1SPARK_HOME set to Spark 2.3.0 curl -i -XPOST http://$(hostname -f):8998/batches -H 'Content-Type: application/json' -d '{"file": "/home/USER/test_livy/livy.py", "env" : {"SPARK_HOME": "/home/USER/sparks/spark-2.3.0-bin-hadoop2.7"}}'
# Spark Version: 2.3.0 |
|
@jerryshao - I only needed it for batch sessions currently so started with that. If you or someone has pointers on where to start adding to interactive session I can look at that too. |
|
For the completeness of PR, I think adding support of interactive session is also required. |
risdenk
changed the title
|
@jerryshao - I took a stab at adding environment variable support for interactive sessions as well. |
…ssions Signed-off-by: Kevin Risden <krisden@apache.org>
|
Rebased on current master. |
|
Test failures are unrelated to this change: |
|
This pull request has been automatically marked as stale because it has had no activity for at least 3 months. If you are still working on this change or plan to move it forward, please leave a comment or push a new commit so we know to keep it open. Otherwise, this PR will be closed automatically in about one month. Thank you for your contribution to Apache Livy! |
|
Closing this pull request due to at least 4 months of inactivity. If you would like to continue the work, please feel free to reopen this pull request or open a new one. |
4 participants
https://issues.apache.org/jira/browse/LIVY-414
What changes were proposed in this pull request?
spark.driver.extraClasspathSPARK_CLASSPATHto pickup delegation tokensSPARK_HOMEon each request to choose between different Spark versions like Spark 1 and Spark 2.envfor batch requestHow was this patch tested?
SparkProcessBuildergets environment variables before launchingspark-submitSparkLaunchergets environment variables before creating session