Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[KSHC] Support Parquet/Orc provider is splitable #5017

Closed
wants to merge 1 commit into from

Conversation

yikf
Copy link
Contributor

@yikf yikf commented Jul 4, 2023

Why are the changes needed?

This PR amins to support Parquet/Orc provider is splitable.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before make a pull request

@pan3793
Copy link
Member

pan3793 commented Jul 4, 2023

Another question not related to this PR, what's the output filename pattern written by this connector? does it have .<codec>.<format> suffix? e.g. .gzip.parquet

@yikf
Copy link
Contributor Author

yikf commented Jul 4, 2023

Another question not related to this PR, what's the output filename pattern written by this connector? does it have .<codec>.<format> suffix? e.g. .gzip.parquet

Like Spark Hive v1, it has no suffix

@pan3793
Copy link
Member

pan3793 commented Jul 4, 2023

Another question not related to this PR, what's the output filename pattern written by this connector? does it have .<codec>.<format> suffix? e.g. .gzip.parquet

Like Spark Hive v1, it has no suffix

It's inconsistent in built-in Hive, parquet has no suffix but ORC does.

@codecov-commenter
Copy link

codecov-commenter commented Jul 4, 2023

Codecov Report

Merging #5017 (79b2aa0) into master (1d5ac07) will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #5017   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         563     563           
  Lines       31167   31167           
  Branches     4070    4070           
======================================
  Misses      31167   31167           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@yikf yikf self-assigned this Jul 5, 2023
@yikf yikf added this to the v1.8.0 milestone Jul 5, 2023
@yikf
Copy link
Contributor Author

yikf commented Jul 6, 2023

Kindly ping @pan3793 @cxzl25 , Any comments?

@yikf yikf closed this in 46f8e0c Jul 6, 2023
@yikf
Copy link
Contributor Author

yikf commented Jul 6, 2023

Thanks all, merged to master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants