Skip to content

Commit

Permalink
[SPARK-8355] [SQL] Python DataFrameReader/Writer should mirror Scala
Browse files Browse the repository at this point in the history
I compared PySpark DataFrameReader/Writer against Scala ones. `Option` function is missing in both reader and writer, but the rest seems to all match.

I added `Option` to reader and writer and updated the `pyspark-sql` test.

Author: Cheolsoo Park <cheolsoop@netflix.com>

Closes #7078 from piaozhexiu/SPARK-8355 and squashes the following commits:

c63d419 [Cheolsoo Park] Fix version
524e0aa [Cheolsoo Park] Add option function to df reader and writer
  • Loading branch information
Cheolsoo Park authored and rxin committed Jun 29, 2015
1 parent 0b10662 commit ac2e17b
Show file tree
Hide file tree
Showing 2 changed files with 15 additions and 0 deletions.
14 changes: 14 additions & 0 deletions python/pyspark/sql/readwriter.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,13 @@ def schema(self, schema):
self._jreader = self._jreader.schema(jschema)
return self

@since(1.5)
def option(self, key, value):
"""Adds an input option for the underlying data source.
"""
self._jreader = self._jreader.option(key, value)
return self

@since(1.4)
def options(self, **options):
"""Adds input options for the underlying data source.
Expand Down Expand Up @@ -235,6 +242,13 @@ def format(self, source):
self._jwrite = self._jwrite.format(source)
return self

@since(1.5)
def option(self, key, value):
"""Adds an output option for the underlying data source.
"""
self._jwrite = self._jwrite.option(key, value)
return self

@since(1.4)
def options(self, **options):
"""Adds output options for the underlying data source.
Expand Down
1 change: 1 addition & 0 deletions python/pyspark/sql/tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -564,6 +564,7 @@ def test_save_and_load_builder(self):
self.assertEqual(sorted(df.collect()), sorted(actual.collect()))

df.write.mode("overwrite").options(noUse="this options will not be used in save.")\
.option("noUse", "this option will not be used in save.")\
.format("json").save(path=tmpPath)
actual =\
self.sqlCtx.read.format("json")\
Expand Down

0 comments on commit ac2e17b

Please sign in to comment.