-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-8355] [SQL] Python DataFrameReader/Writer should mirror Scala #7078
Conversation
Can one of the admins verify this patch? |
Jenkins, add to whitelist. |
Jenkins, ok to test. |
@@ -74,6 +74,13 @@ def schema(self, schema): | |||
return self | |||
|
|||
@since(1.4) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should be 1.5
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for reviewing. Fixed.
Merged build triggered. |
Merged build started. |
Test build #35966 has started for PR 7078 at commit |
Merged build triggered. |
Merged build started. |
Test build #35968 has started for PR 7078 at commit |
lgtm. |
Test build #35966 has finished for PR 7078 at commit
|
Merged build finished. Test FAILed. |
@JoshRosen maybe a bug in the reporting script? it fails because it cannot find unit-tests.log |
Going to merge this. Thanks! |
Test build #35968 has finished for PR 7078 at commit
|
Merged build finished. Test FAILed. |
I compared PySpark DataFrameReader/Writer against Scala ones. `Option` function is missing in both reader and writer, but the rest seems to all match. I added `Option` to reader and writer and updated the `pyspark-sql` test. Author: Cheolsoo Park <cheolsoop@netflix.com> Closes #7078 from piaozhexiu/SPARK-8355 and squashes the following commits: c63d419 [Cheolsoo Park] Fix version 524e0aa [Cheolsoo Park] Add option function to df reader and writer (cherry picked from commit ac2e17b) Signed-off-by: Reynold Xin <rxin@databricks.com>
I compared PySpark DataFrameReader/Writer against Scala ones.
Option
function is missing in both reader and writer, but the rest seems to all match.I added
Option
to reader and writer and updated thepyspark-sql
test.