Include the sbin/spark-config.sh in spark-executor #651
Conversation
This is needed because broadcast values are broken on pyspark on Mesos, it tries to import pyspark but can't.
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Mind creating a JIRA for this? If you could post the exception or issue that occurs it would be helfpul. |
|
JIRA: https://issues.apache.org/jira/browse/SPARK-1725 The error that happens is |
sbin/spark-executor
Outdated
There was a problem hiding this comment.
You can just re-use $FWDIR from above
There was a problem hiding this comment.
I.e. $FWDIR/sbin/spark-config.sh.
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
LGTM - @bouk did you test this new fix and make sure it works? |
|
I went ahead and merged this - thanks! |
This is needed because broadcast values are broken on pyspark on Mesos, it tries to import pyspark but can't, as the PYTHONPATH is not set due to changes in ff5be9a https://issues.apache.org/jira/browse/SPARK-1725 Author: Bouke van der Bijl <boukevanderbijl@gmail.com> Closes #651 from bouk/include-spark-config-in-mesos-executor and squashes the following commits: b2f1295 [Bouke van der Bijl] Inline PYTHONPATH in spark-executor eedbbcc [Bouke van der Bijl] Include the sbin/spark-config.sh in spark-executor (cherry picked from commit 2fd2752) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
This is needed because broadcast values are broken on pyspark on Mesos, it tries to import pyspark but can't, as the PYTHONPATH is not set due to changes in ff5be9a https://issues.apache.org/jira/browse/SPARK-1725 Author: Bouke van der Bijl <boukevanderbijl@gmail.com> Closes apache#651 from bouk/include-spark-config-in-mesos-executor and squashes the following commits: b2f1295 [Bouke van der Bijl] Inline PYTHONPATH in spark-executor eedbbcc [Bouke van der Bijl] Include the sbin/spark-config.sh in spark-executor
###### _excavator_ is a bot for automating changes across repositories. Changes produced by the excavator/bulldozer-oss check. To enable or disable this check, please contact the maintainers of Excavator.
This is needed because broadcast values are broken on pyspark on Mesos, it tries to import pyspark but can't, as the PYTHONPATH is not set due to changes in ff5be9a
https://issues.apache.org/jira/browse/SPARK-1725