Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-19307][pyspark] Make sure user conf is propagated to SparkContext. #16682

Closed
wants to merge 3 commits into from

Conversation

@vanzin
Copy link
Contributor

commented Jan 23, 2017

The code was failing to propagate the user conf in the case where the
JVM was already initialized, which happens when a user submits a
python script via spark-submit.

Tested with new unit test and by running a python script in a real cluster.

[SPARK-19307][pyspark] Make sure user conf is propagated to SparkCont…
…ext.

The code was failing to propagate the user conf in the case where the
JVM was already initialized, which happens when a user submits a
python script via spark-submit.

Tested with new unit test and by running a python script in a real cluster.
@SparkQA

This comment has been minimized.

Copy link

commented Jan 23, 2017

Test build #71876 has finished for PR 16682 at commit 35eeafe.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@SparkQA

This comment has been minimized.

Copy link

commented Jan 23, 2017

Test build #71882 has finished for PR 16682 at commit cef5136.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@SparkQA

This comment has been minimized.

Copy link

commented Jan 24, 2017

Test build #71893 has finished for PR 16682 at commit ef8349a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@vanzin

This comment has been minimized.

Copy link
Contributor Author

commented Jan 24, 2017

@zjffdu

This comment has been minimized.

Copy link
Contributor

commented Jan 24, 2017

LGTM

@vanzin

This comment has been minimized.

Copy link
Contributor Author

commented Jan 25, 2017

Merging to master / 2.1.

asfgit pushed a commit that referenced this pull request Jan 25, 2017
[SPARK-19307][PYSPARK] Make sure user conf is propagated to SparkCont…
…ext.

The code was failing to propagate the user conf in the case where the
JVM was already initialized, which happens when a user submits a
python script via spark-submit.

Tested with new unit test and by running a python script in a real cluster.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #16682 from vanzin/SPARK-19307.

(cherry picked from commit 92afaa9)
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

@asfgit asfgit closed this in 92afaa9 Jan 25, 2017

@vanzin vanzin deleted the vanzin:SPARK-19307 branch Jan 27, 2017

uzadude added a commit to uzadude/spark that referenced this pull request Jan 27, 2017
[SPARK-19307][PYSPARK] Make sure user conf is propagated to SparkCont…
…ext.

The code was failing to propagate the user conf in the case where the
JVM was already initialized, which happens when a user submits a
python script via spark-submit.

Tested with new unit test and by running a python script in a real cluster.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes apache#16682 from vanzin/SPARK-19307.
cmonkey added a commit to cmonkey/spark that referenced this pull request Feb 15, 2017
[SPARK-19307][PYSPARK] Make sure user conf is propagated to SparkCont…
…ext.

The code was failing to propagate the user conf in the case where the
JVM was already initialized, which happens when a user submits a
python script via spark-submit.

Tested with new unit test and by running a python script in a real cluster.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes apache#16682 from vanzin/SPARK-19307.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.