New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-26584][SQL] Remove spark.sql.orc.copyBatchToSpark
internal conf
#23503
Conversation
cc @cloud-fan , @viirya , @gengliangwang |
Thank you for review and approval, @cloud-fan . |
@@ -86,12 +85,8 @@ | |||
// The memory mode of the columnarBatch | |||
private final MemoryMode MEMORY_MODE; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we still need this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops. Thank you.
Thank you, @HyukjinKwon and @viirya . |
Test build #100998 has finished for PR 23503 at commit
|
Test build #101004 has finished for PR 23503 at commit
|
retest this please. |
Thanks, @viirya . |
+1, LGTM |
Test build #101009 has finished for PR 23503 at commit
|
Thank you, @gengliangwang . |
Merged to master. |
## What changes were proposed in this pull request? This PR aims to remove internal ORC configuration to simplify the code path for Spark 3.0.0. This removes the configuration `spark.sql.orc.copyBatchToSpark` and related ORC codes including tests and benchmarks. ## How was this patch tested? Pass the Jenkins with the reduced test coverage. Closes apache#23503 from dongjoon-hyun/SPARK-26584. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
What changes were proposed in this pull request?
This PR aims to remove internal ORC configuration to simplify the code path for Spark 3.0.0. This removes the configuration
spark.sql.orc.copyBatchToSpark
and related ORC codes including tests and benchmarks.How was this patch tested?
Pass the Jenkins with the reduced test coverage.