[SPARK-8532][SQL] In Python's DataFrameWriter, save/saveAsTable/json/parquet/jdbc always override mode #6937
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #35462 has finished for PR 6937 at commit
|
| @@ -276,7 +276,15 @@ def save(self, path=None, format=None, mode="error", **options): | |||
|
|
|||
| >>> df.write.mode('append').parquet(os.path.join(tempfile.mkdtemp(), 'data')) | |||
| """ | |||
| self.mode(mode).options(**options) | |||
| if mode is not "error": | |||
|
Test build #35465 has finished for PR 6937 at commit
|
…de already uses "error" as the default value.
| @@ -346,13 +352,14 @@ def parquet(self, path, mode="error"): | |||
| * ``overwrite``: Overwrite existing data. | |||
| * ``ignore``: Silently ignore this operation if data already exists. | |||
| * ``error`` (default case): Throw an exception if data already exists. | |||
|
|
|||
| :param partitionBy: names of partitioning columns | |||
| >>> df.write.parquet(os.path.join(tempfile.mkdtemp(), 'data')) | |||
|
LGTM |
|
Test build #35475 has finished for PR 6937 at commit
|
| @@ -336,7 +342,7 @@ def json(self, path, mode="error"): | |||
| self._jwrite.mode(mode).json(path) | |||
There was a problem hiding this comment.
self.mode(mode)._jwrite.json(path)
|
Test build #35470 has finished for PR 6937 at commit
|
|
Oh, this one hits |
|
I will merge it into master and branch-1.4 once SparkQA is happy. |
|
Test build #35477 has finished for PR 6937 at commit
|
|
Merged build finished. Test FAILed. |
asfgit
pushed a commit
that referenced
this pull request
Jun 22, 2015
…/parquet/jdbc always override mode https://issues.apache.org/jira/browse/SPARK-8532 This PR has two changes. First, it fixes the bug that save actions (i.e. `save/saveAsTable/json/parquet/jdbc`) always override mode. Second, it adds input argument `partitionBy` to `save/saveAsTable/parquet`. Author: Yin Huai <yhuai@databricks.com> Closes #6937 from yhuai/SPARK-8532 and squashes the following commits: f972d5d [Yin Huai] davies's comment. d37abd2 [Yin Huai] style. d21290a [Yin Huai] Python doc. 889eb25 [Yin Huai] Minor refactoring and add partitionBy to save, saveAsTable, and parquet. 7fbc24b [Yin Huai] Use None instead of "error" as the default value of mode since JVM-side already uses "error" as the default value. d696dff [Yin Huai] Python style. 88eb6c4 [Yin Huai] If mode is "error", do not call mode method. c40c461 [Yin Huai] Regression test. (cherry picked from commit 5ab9fcf) Signed-off-by: Yin Huai <yhuai@databricks.com>
nemccarthy
pushed a commit
to nemccarthy/spark
that referenced
this pull request
Jun 22, 2015
…/parquet/jdbc always override mode https://issues.apache.org/jira/browse/SPARK-8532 This PR has two changes. First, it fixes the bug that save actions (i.e. `save/saveAsTable/json/parquet/jdbc`) always override mode. Second, it adds input argument `partitionBy` to `save/saveAsTable/parquet`. Author: Yin Huai <yhuai@databricks.com> Closes apache#6937 from yhuai/SPARK-8532 and squashes the following commits: f972d5d [Yin Huai] davies's comment. d37abd2 [Yin Huai] style. d21290a [Yin Huai] Python doc. 889eb25 [Yin Huai] Minor refactoring and add partitionBy to save, saveAsTable, and parquet. 7fbc24b [Yin Huai] Use None instead of "error" as the default value of mode since JVM-side already uses "error" as the default value. d696dff [Yin Huai] Python style. 88eb6c4 [Yin Huai] If mode is "error", do not call mode method. c40c461 [Yin Huai] Regression test. (cherry picked from commit 5ab9fcf) Signed-off-by: Yin Huai <yhuai@databricks.com>
animeshbaranawal
pushed a commit
to animeshbaranawal/spark
that referenced
this pull request
Jun 25, 2015
…/parquet/jdbc always override mode https://issues.apache.org/jira/browse/SPARK-8532 This PR has two changes. First, it fixes the bug that save actions (i.e. `save/saveAsTable/json/parquet/jdbc`) always override mode. Second, it adds input argument `partitionBy` to `save/saveAsTable/parquet`. Author: Yin Huai <yhuai@databricks.com> Closes apache#6937 from yhuai/SPARK-8532 and squashes the following commits: f972d5d [Yin Huai] davies's comment. d37abd2 [Yin Huai] style. d21290a [Yin Huai] Python doc. 889eb25 [Yin Huai] Minor refactoring and add partitionBy to save, saveAsTable, and parquet. 7fbc24b [Yin Huai] Use None instead of "error" as the default value of mode since JVM-side already uses "error" as the default value. d696dff [Yin Huai] Python style. 88eb6c4 [Yin Huai] If mode is "error", do not call mode method. c40c461 [Yin Huai] Regression test.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/SPARK-8532
This PR has two changes. First, it fixes the bug that save actions (i.e.
save/saveAsTable/json/parquet/jdbc) always override mode. Second, it adds input argumentpartitionBytosave/saveAsTable/parquet.