ks.read_spark_io / DataFrame.to_spark_io #447
Conversation
Codecov Report
@@ Coverage Diff @@
## master #447 +/- ##
==========================================
+ Coverage 93.06% 93.07% +0.01%
==========================================
Files 27 27
Lines 3344 3349 +5
==========================================
+ Hits 3112 3117 +5
Misses 232 232
Continue to review full report at Codecov.
|
rxin
changed the title
|
@floscha want to take a look at this? |
There was a problem hiding this comment.
This is some really useful functionality to have since, in practice, I read/write most data from/to HDFS rather than a local file system.
I left some comments regarding the documentation. Since the implementation itself is pretty straightforward, I don't have any complaints there 😉
databricks/koalas/frame.py
Outdated
| path : string, optional | ||
| Path to the data source. | ||
| format : string, optional | ||
| Name of the data source in Spark. |
There was a problem hiding this comment.
Name sounds more like file name if you asked me. Why not instead go for "Specifies the input data source format." like the Spark docs describe it.
Also, like you did for mode, it would be great to provide a list of supported formats, namely: CSV, JDBC, JSON, ORC, and Parquet.
There was a problem hiding this comment.
Thanks. Good point. Will do the changes.
databricks/koalas/namespace.py
Outdated
| path : string, optional | ||
| Path to the data source. | ||
| format : string, optional | ||
| Name of the data source in Spark. |
There was a problem hiding this comment.
See my comment on format above.
|
|
||
| # Write out partitioned by one column | ||
| expected.to_spark_io(tmp, format='json', mode='overwrite', partition_cols='i32') | ||
| # reset column order, as once the data is written out, Spark rearranges partition |
There was a problem hiding this comment.
This line and 107 could start with a capital letter and end with a period, but that's rather cosmetic 😉
Softagram Impact Report for pull/447 (head commit: e2726df)⭐ Change Overview
📄 Full report
Give feedback on this report to support@softagram.com |
Resolves #446