[SPARK-1550] [PySpark] Allow SparkContext creation after failed attempts #1606
Conversation
|
QA tests have started for PR 1606. This patch merges cleanly. |
| @@ -249,17 +258,14 @@ def defaultMinPartitions(self): | |||
| """ | |||
| return self._jsc.sc().defaultMinPartitions() | |||
|
|
|||
| def __del__(self): | |||
There was a problem hiding this comment.
Because we keep a reference to this object in SparkContext._active_spark_context, this method never gets called except when cleaning up after a SparkContext creation attempt that failed because another context was already running. In that case, the call to sc.stop() clears SparkContext._active_spark_context and we lose track of the active context, which can allow the creation of multiple running contexts.
|
QA results for PR 1606: |
|
+1 lgtm i've been hitting this issue repeatedly, but assumed it was a corner case that wouldn't get much attention. my assumption is that use of spark-submit and pyspark are the primary ways to get a context. |
|
Thanks Josh, merged this. |
This addresses a PySpark issue where a failed attempt to construct SparkContext would prevent any future SparkContext creation. Author: Josh Rosen <joshrosen@apache.org> Closes apache#1606 from JoshRosen/SPARK-1550 and squashes the following commits: ec7fadc [Josh Rosen] [SPARK-1550] [PySpark] Allow SparkContext creation after failed attempts
(cherry picked from commit d6a02ec59cd1997858a2e63e85e834361a769dc3) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
This addresses a PySpark issue where a failed attempt to construct SparkContext would prevent any future SparkContext creation.