[ZEPPELIN-18] Running pyspark without deploying python libraries to every yarn node #118

jongyoul · 2015-06-24T06:13:44Z

Spark supports pyspark on yarn cluster without deploying python libraries from Spark 1.4
- https://issues.apache.org/jira/browse/SPARK-6869
- [SPARK-6869][PySpark] Add pyspark archives path to PYTHONPATH spark#5580, [SPARK-6869][PySpark] Add pyspark archives path to PYTHONPATH spark#5478

jongyoul · 2015-06-24T06:18:10Z

bin/interpreter.sh

+  export PYTHONPATH="${ZEPPELIN_HOME}/python/lib/pyspark.zip:${ZEPPELIN_HOME}/python/lib/py4j-0.8.2.1-src.zip"
+else
+  export PYTHONPATH="$PYTHONPATH${ZEPPELIN_HOME}/lib/pyspark.zip:${ZEPPELIN_HOME}/python/lib/py4j-0.8.2.1-src.zip"
+fi


@Leemoonsoo In case of PYTHONPATH on PysparkInterpreter, It's not affected by python driver. In the past, Zeppelin loads pyspark from SPARK_HOME/python

bzz · 2015-06-24T09:04:59Z

Rising a question on necessity of downloading spark on every dev machine because recently heard a joke from @anthonycorbacho (somebody on the Strata conference),

"Want do download whole internet? Build Zeppelin!"

I.e .m2 dir on my machine is about 4 Gb already.

Is there a reason we can't have it under some some kind of profile?
So users who need it will do like -Pyarn-python

jongyoul · 2015-06-24T09:11:20Z

Do you think it's good to make a profile? There is no problem to make it. I'll handle it to make that joke as just joke.

jongyoul · 2015-06-24T18:21:57Z

Can anyone help me know the reason why last build fails?

Leemoonsoo · 2015-06-24T18:27:36Z

You can download the log file and search for "BUILD FAILURE"

jongyoul · 2015-06-24T18:32:46Z

@Leemoonsoo thanks. I will check it tomorrow. And what do you think of making a profile for yarn-pyspark?

Leemoonsoo · 2015-06-24T18:39:35Z

Making profile for pyspark is not a bad idea. However, pyspark can work with not only yarn but also with mesos and standalone cluster. So i think it would be better if profile looks like -Ppyspark than -Pyarn-pyspark

jongyoul · 2015-06-25T05:38:13Z

I've rebased

jongyoul · 2015-06-25T11:43:23Z

@bzz @Leemoonsoo Review this again, please.

Leemoonsoo · 2015-06-25T22:03:14Z

@jongyoul
Thanks for great contribution. It really helps not only yarn but also local mode and standalone cluster.
LGTM!

bzz · 2015-06-26T00:53:57Z

Thank you very much for contributing this!

It would be great to have a high level summary of the changes, so please correct me in case I miss-understand something:

This PR allows users of pyspark skip setting pythonpath env var, copy Python modules on every node of he cluster and have spark installed(in case of pyspark in local mode on 1 machine) by adding new artefact to the Zeppelin build, a python, hidden behind optional build profile, that brings py4j as well as Python code of pyspark by downloading (and caching) actual spark distribution and re-packing those to a zip file, available in Z class path at runtime.

Is that correct?
If it is - that sounds great to me, pspark not working in Z with local interpreter without having spark installed was very frustrating.

One question: a Python dir is not a maven submodule now but may be it should be, one day, what do you guys think?

jongyoul · 2015-06-26T07:20:59Z

@bzz It's correct for your understanding. I'm worried about mesos cluster mode because I don't test my PR on these cluster yet, but standalone cluster will be ok because that cluster already has python libraries. I've only tested it from local mode and yarn cluster mode. Concerning a directory name of python, I think python interpreter will be added somedays, can you recommend the directory name?

Leemoonsoo · 2015-06-27T01:18:03Z

bin/interpreter.sh

+if [[ x"" == x${PYTHONPATH} ]]; then
+  export PYTHONPATH="${ZEPPELIN_HOME}/python/lib/pyspark.zip:${ZEPPELIN_HOME}/python/lib/py4j-0.8.2.1-src.zip"
+else
+  export PYTHONPATH="$PYTHONPATH${ZEPPELIN_HOME}/lib/pyspark.zip:${ZEPPELIN_HOME}/python/lib/py4j-0.8.2.1-src.zip"


How about add colon (:)?
from ....="$PYTHONPATH${ZEPPELIN_HOME}/lib/py.... to ....="$PYTHONPATH:${ZEPPELIN_HOME}/lib/py....

right, I'm a bit confused by this. is it not suppose to have a colon?

@Leemoonsoo @felixcheung Yes, you guys are right. I can't find this error because I don't set extra PYTHONPATH at all and don't test it. I'll fix it.

jongyoul · 2015-06-30T02:10:04Z

I'm trying to remove some unused settings. I hope not to set SPARK_HOME for using pyspark eventually.

jongyoul · 2015-07-03T03:33:09Z

I've heard close and reopen issue is the easiest way to trigger a travis, but In my case, I have a merge conflict.

…very yarn node - rebasing

- Removed redundant dependency setting

…very yarn node - Followed spark's way to support pyspark - https://issues.apache.org/jira/browse/SPARK-6869 - apache/spark#5580 - https://github.com/apache/spark/pull/5478/files

…very yarn node - Removed verbose setting

- Excludes python/** from apache-rat

…very yarn node - rebasing

…very yarn node - Fixed checkstyle

…very yarn node - rebasing

…very yarn node - Dummy for trigger

- Changed the location of pyspark's directory into interpreter/spark

jongyoul · 2015-07-04T14:14:16Z

@Leemoonsoo I've rebased it again because all of travis tests passed.

Leemoonsoo · 2015-07-04T18:36:35Z

Thanks @jongyoul. Great work!
Looks good to me.

…very yarn node - Spark supports pyspark on yarn cluster without deploying python libraries from Spark 1.4 - https://issues.apache.org/jira/browse/SPARK-6869 - apache/spark#5580, apache/spark#5478 Author: Jongyoul Lee <jongyoul@gmail.com> Closes #118 from jongyoul/ZEPPELIN-18 and squashes the following commits: a47e27c [Jongyoul Lee] - Fixed test script for spark 1.4.0 72a65fd [Jongyoul Lee] - Fixed test script for spark 1.4.0 ee6d100 [Jongyoul Lee] - Cleanup codes 47fd9c9 [Jongyoul Lee] - Cleanup codes 248e330 [Jongyoul Lee] - Cleanup codes 4cd10b5 [Jongyoul Lee] - Removed meaningless codes comments c9cda29 [Jongyoul Lee] - Removed setting SPARK_HOME - Changed the location of pyspark's directory into interpreter/spark ef240f5 [Jongyoul Lee] - Fixed typo 06002fd [Jongyoul Lee] - Fixed typo 4b35c8d [Jongyoul Lee] [ZEPPELIN-18] Running pyspark without deploying python libraries to every yarn node - Dummy for trigger 682986e [Jongyoul Lee] rebased 8a7bf47 [Jongyoul Lee] [ZEPPELIN-18] Running pyspark without deploying python libraries to every yarn node - rebasing ad610fb [Jongyoul Lee] rebased 94bdf30 [Jongyoul Lee] [ZEPPELIN-18] Running pyspark without deploying python libraries to every yarn node - Fixed checkstyle 929333d [Jongyoul Lee] rebased 64b8195 [Jongyoul Lee] [ZEPPELIN-18] Running pyspark without deploying python libraries to every yarn node - rebasing 0a2d90e [Jongyoul Lee] rebased b05ae6e [Jongyoul Lee] [ZEPPELIN-18] Remove setting SPARK_HOME for PySpark - Excludes python/** from apache-rat 71e2a92 [Jongyoul Lee] [ZEPPELIN-18] Running pyspark without deploying python libraries to every yarn node - Removed verbose setting 0ddb436 [Jongyoul Lee] [ZEPPELIN-18] Running pyspark without deploying python libraries to every yarn node - Followed spark's way to support pyspark - https://issues.apache.org/jira/browse/SPARK-6869 - apache/spark#5580 - https://github.com/apache/spark/pull/5478/files 1b192f6 [Jongyoul Lee] [ZEPPELIN-18] Remove setting SPARK_HOME for PySpark - Removed redundant dependency setting 32fd9e1 [Jongyoul Lee] [ZEPPELIN-18] Running pyspark without deploying python libraries to every yarn node - rebasing (cherry picked from commit 3bd2b21) Signed-off-by: Lee moon soo <moon@apache.org>

jongyoul reviewed Jun 24, 2015
View reviewed changes

jongyoul closed this Jun 25, 2015

jongyoul reopened this Jun 25, 2015

jongyoul force-pushed the ZEPPELIN-18 branch from 27e7dbc to 0ed0e8b Compare June 25, 2015 05:37

Leemoonsoo reviewed Jun 27, 2015
View reviewed changes

jongyoul closed this Jul 3, 2015

jongyoul reopened this Jul 3, 2015

jongyoul force-pushed the ZEPPELIN-18 branch from 914741b to 8224e1a Compare July 3, 2015 03:41

jongyoul closed this Jul 3, 2015

jongyoul reopened this Jul 3, 2015

jongyoul force-pushed the ZEPPELIN-18 branch from de709e6 to a47e27c Compare July 4, 2015 14:12

jongyoul added 22 commits July 4, 2015 23:13

[ZEPPELIN-18] Running pyspark without deploying python libraries to e…

32fd9e1

…very yarn node - rebasing

[ZEPPELIN-18] Remove setting SPARK_HOME for PySpark

1b192f6

- Removed redundant dependency setting

[ZEPPELIN-18] Running pyspark without deploying python libraries to e…

0ddb436

…very yarn node - Followed spark's way to support pyspark - https://issues.apache.org/jira/browse/SPARK-6869 - apache/spark#5580 - https://github.com/apache/spark/pull/5478/files

[ZEPPELIN-18] Running pyspark without deploying python libraries to e…

71e2a92

…very yarn node - Removed verbose setting

[ZEPPELIN-18] Remove setting SPARK_HOME for PySpark

b05ae6e

- Excludes python/** from apache-rat

rebased

0a2d90e

[ZEPPELIN-18] Running pyspark without deploying python libraries to e…

64b8195

…very yarn node - rebasing

rebased

929333d

[ZEPPELIN-18] Running pyspark without deploying python libraries to e…

94bdf30

…very yarn node - Fixed checkstyle

rebased

ad610fb

[ZEPPELIN-18] Running pyspark without deploying python libraries to e…

8a7bf47

…very yarn node - rebasing

rebased

682986e

[ZEPPELIN-18] Running pyspark without deploying python libraries to e…

4b35c8d

…very yarn node - Dummy for trigger

- Fixed typo

06002fd

- Fixed typo

ef240f5

- Removed setting SPARK_HOME

c9cda29

- Changed the location of pyspark's directory into interpreter/spark

- Removed meaningless codes comments

4cd10b5

- Cleanup codes

248e330

- Cleanup codes

47fd9c9

- Cleanup codes

ee6d100

- Fixed test script for spark 1.4.0

72a65fd

- Fixed test script for spark 1.4.0

a47e27c

asfgit closed this in 3bd2b21 Jul 5, 2015

felixcheung mentioned this pull request Aug 9, 2015

pom typo fix for pyspark profile #196

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ZEPPELIN-18] Running pyspark without deploying python libraries to every yarn node #118

[ZEPPELIN-18] Running pyspark without deploying python libraries to every yarn node #118

jongyoul commented Jun 24, 2015

jongyoul Jun 24, 2015

bzz commented Jun 24, 2015

jongyoul commented Jun 24, 2015

jongyoul commented Jun 24, 2015

Leemoonsoo commented Jun 24, 2015

jongyoul commented Jun 24, 2015

Leemoonsoo commented Jun 24, 2015

jongyoul commented Jun 25, 2015

jongyoul commented Jun 25, 2015

Leemoonsoo commented Jun 25, 2015

bzz commented Jun 26, 2015

jongyoul commented Jun 26, 2015

Leemoonsoo Jun 27, 2015

felixcheung Jun 27, 2015

jongyoul Jun 27, 2015

jongyoul commented Jun 30, 2015

jongyoul commented Jul 3, 2015

jongyoul commented Jul 4, 2015

Leemoonsoo commented Jul 4, 2015

[ZEPPELIN-18] Running pyspark without deploying python libraries to every yarn node #118

[ZEPPELIN-18] Running pyspark without deploying python libraries to every yarn node #118

Conversation

jongyoul commented Jun 24, 2015

jongyoul Jun 24, 2015

Choose a reason for hiding this comment

bzz commented Jun 24, 2015

jongyoul commented Jun 24, 2015

jongyoul commented Jun 24, 2015

Leemoonsoo commented Jun 24, 2015

jongyoul commented Jun 24, 2015

Leemoonsoo commented Jun 24, 2015

jongyoul commented Jun 25, 2015

jongyoul commented Jun 25, 2015

Leemoonsoo commented Jun 25, 2015

bzz commented Jun 26, 2015

jongyoul commented Jun 26, 2015

Leemoonsoo Jun 27, 2015

Choose a reason for hiding this comment

felixcheung Jun 27, 2015

Choose a reason for hiding this comment

jongyoul Jun 27, 2015

Choose a reason for hiding this comment

jongyoul commented Jun 30, 2015

jongyoul commented Jul 3, 2015

jongyoul commented Jul 4, 2015

Leemoonsoo commented Jul 4, 2015