Enable using multithreaded setup_page_index in hybrid scan reader - #20721
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
| }; | ||
|
|
||
| // Use parallel processing only if we have enough columns to justify the overhead | ||
| constexpr std::size_t parallel_threshold = 256; |
There was a problem hiding this comment.
@vuule did you pick 512 for this empirically or did you run any experiments. I am seeing more benefit with 256 for the hybrid scan example but can revert.
There was a problem hiding this comment.
I didn't look for the break-even point, so I'm not surprised this number is too high. Would not object to adjusting it if you have a benchmark that shows 256 to be a better threshold.
There was a problem hiding this comment.
Okay, thanks for confirming
mhaseeb123
added
3 - Ready for Review
| return; | ||
| } | ||
|
|
||
| auto& row_groups = per_file_metadata.front().row_groups; |
There was a problem hiding this comment.
No need for all this anymore. Just call metadata.setup_page_index(..) method.
| int64_t const length = max_offset - min_offset; | ||
| auto const idx_buf = source->host_read(min_offset, length); | ||
| // Flatten all columns into a single vector for easier task distribution | ||
| std::vector<std::reference_wrapper<ColumnChunk>> all_column_chunks; |
There was a problem hiding this comment.
Moved everything beyond this point as-is into the setup_page_index function.
| { | ||
| CUDF_FUNC_RANGE(); | ||
|
|
||
| // Flatten all columns into a single vector for easier task distribution |
There was a problem hiding this comment.
Moved as-is from the constructor
| metadata& operator=(metadata const& other) = delete; | ||
| metadata& operator=(metadata&& other) = default; | ||
|
|
||
| ~metadata() = default; |
There was a problem hiding this comment.
Use base reader's destructor that uses multiple threads
mhaseeb123
added
4 - Needs Review
|
/merge |
mhaseeb123
removed
the
4 - Needs Review
mhaseeb123
added
the
5 - Ready to Merge
3 participants
Description
This PR refactors the multithreaded page index setup code in main parquet reader so that the child class used in hybrid scan reader can also benefit from it. No new code is introduced.
Checklist