[SPARK-48807][SQL] Binary Support for CSV datasource #47212
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
only thing from me is that we won't be able to read/write roundtrip. Can we do this with the newer binary string format? |
|
For IO roundtrip, the UFT8 output style can play it directly. Other styles can play with/ functions, or we can add an extra read option to help |
|
If we specify the schema as binary, can we read it back as binary? |
|
I remember we do similar things in thriftserver (cc @wangyum ) so I am fine with this but just want to make sure we can read it back |
Yes, I have added the above tests to verify read-as-raw-string and read-w/-binary-schema |
HyukjinKwon
approved these changes
Jul 10, 2024
HyukjinKwon
reviewed
Jul 10, 2024
| .option("ds_option", "value") | ||
| .format(dataSourceFormat) | ||
| .save(path.getCanonicalPath) | ||
| val expectedStr = ToStringBase.getBinaryFormatter("Spark SQL".getBytes()) |
There was a problem hiding this comment.
Can we change the value as non UTF8 output instead?
There was a problem hiding this comment.
This helper method gets a binary formatter based on BINARY_OUTPUT_STYLE and converts the raw bytes here to both UTF8 and non-UTF8 outputs
|
Thanks you @HyukjinKwon @dongjoon-hyun Merged to master |
biruktesf-db
pushed a commit
to biruktesf-db/spark
that referenced
this pull request
Jul 11, 2024
### What changes were proposed in this pull request? SPARK-42237 disabled binary output for CSV because the binary values use `java.lang.Object.toString` for outputting. Now we have meaningful binary string representations support in UnivocityGenerator, we can support it now. ### Why are the changes needed? improve csv with spark sql types ### Does this PR introduce _any_ user-facing change? Yes, but it's from failures to success with binary csv tables ### How was this patch tested? new tests ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#47212 from yaooqinn/SPARK-48807. Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>
jingz-db
pushed a commit
to jingz-db/spark
that referenced
this pull request
Jul 22, 2024
### What changes were proposed in this pull request? SPARK-42237 disabled binary output for CSV because the binary values use `java.lang.Object.toString` for outputting. Now we have meaningful binary string representations support in UnivocityGenerator, we can support it now. ### Why are the changes needed? improve csv with spark sql types ### Does this PR introduce _any_ user-facing change? Yes, but it's from failures to success with binary csv tables ### How was this patch tested? new tests ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#47212 from yaooqinn/SPARK-48807. Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>
attilapiros
pushed a commit
to attilapiros/spark
that referenced
this pull request
Oct 4, 2024
### What changes were proposed in this pull request? SPARK-42237 disabled binary output for CSV because the binary values use `java.lang.Object.toString` for outputting. Now we have meaningful binary string representations support in UnivocityGenerator, we can support it now. ### Why are the changes needed? improve csv with spark sql types ### Does this PR introduce _any_ user-facing change? Yes, but it's from failures to success with binary csv tables ### How was this patch tested? new tests ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#47212 from yaooqinn/SPARK-48807. Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>
himadripal
pushed a commit
to himadripal/spark
that referenced
this pull request
Oct 19, 2024
### What changes were proposed in this pull request? SPARK-42237 disabled binary output for CSV because the binary values use `java.lang.Object.toString` for outputting. Now we have meaningful binary string representations support in UnivocityGenerator, we can support it now. ### Why are the changes needed? improve csv with spark sql types ### Does this PR introduce _any_ user-facing change? Yes, but it's from failures to success with binary csv tables ### How was this patch tested? new tests ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#47212 from yaooqinn/SPARK-48807. Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
SPARK-42237 disabled binary output for CSV because the binary values use
java.lang.Object.toStringfor outputting. Now we have meaningful binary string representations support in UnivocityGenerator, we can support it now.Why are the changes needed?
improve csv with spark sql types
Does this PR introduce any user-facing change?
Yes, but it's from failures to success with binary csv tables
How was this patch tested?
new tests
Was this patch authored or co-authored using generative AI tooling?
no