New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FOLLOW-UP][SPARK-23772][SQL] Provide an option to ignore column of all null values or empty array during JSON schema inference #22002
Conversation
@maropu Please, have a look at the PR. |
Test build #94249 has finished for PR 22002 at commit
|
Test build #94250 has finished for PR 22002 at commit
|
jenkins, retest this, please |
Test build #94253 has finished for PR 22002 at commit
|
@@ -267,7 +267,7 @@ def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None, | |||
mode=mode, columnNameOfCorruptRecord=columnNameOfCorruptRecord, dateFormat=dateFormat, | |||
timestampFormat=timestampFormat, multiLine=multiLine, | |||
allowUnquotedControlChars=allowUnquotedControlChars, lineSep=lineSep, | |||
samplingRatio=samplingRatio, encoding=encoding) | |||
samplingRatio=samplingRatio, dropFieldIfAllNull=dropFieldIfAllNull, encoding=encoding) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh... good catch. thanks.
LGTM cc: @HyukjinKwon |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Merged to master. |
What changes were proposed in this pull request?
The
dropFieldIfAllNull
parameter of thejson
method wasn't set as an option. This PR fixes that.How was this patch tested?
I added a test to
sql/test.py