[improve](nereids) filter nereidsPrunedTabletIds per partition in distributionPrune#63851
Open
Larborator wants to merge 1 commit into
Open
[improve](nereids) filter nereidsPrunedTabletIds per partition in distributionPrune#63851Larborator wants to merge 1 commit into
Larborator wants to merge 1 commit into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
ee08fcc to
04cc2a6
Compare
…tributionPrune apache#53403 short-circuited `distributionPrune` to return the entire `nereidsPrunedTabletIds` set when running under Nereids. However, the caller `computeTabletInfo` invokes `distributionPrune` inside a per-partition loop and then iterates the returned ids, calling `MaterializedIndex.getTablet(id)` on each. When `nereidsPrunedTabletIds` contains tablets across many partitions, every per-partition iteration walks the entire global set and does a `getTablet` hash lookup on ids that belong to other partitions (which are then filtered out by the null check), yielding O(partitionNum * globalPrunedSize) lookups. The short-circuit also copies the full HashSet into a new ArrayList once per partition. Filter the global set down to the current partition's tablet ids (`tabletIdsInOrder`, already prepared by the caller) before returning. The result is identical to what the caller's null-check would have produced, so behavior is unchanged; only the redundant lookups and copies are eliminated. The non-Nereids path, the `sampleTabletIds` path and the empty-set fallback are untouched. Also cache the `selectedTable.getTablet(id)` result in the caller's loop so the lookup runs once per id instead of twice.
04cc2a6 to
a7bc775
Compare
Contributor
Author
|
/review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
#53403 short-circuited distributionPrune to return the entire nereidsPrunedTabletIds set when running under Nereids. However, the caller computeTabletInfo invokes distributionPrune inside a per-partition loop and then iterates the returned ids, calling MaterializedIndex.getTablet(id) on each. When nereidsPrunedTabletIds contains tablets across many
partitions, every per-partition iteration walks the entire global set and does a getTablet hash lookup on ids that belong to other partitions (which are then filtered out by the null check), yielding O(partitionNum * globalPrunedSize) lookups. The short-circuit also copies the full HashSet into a new ArrayList once per partition.
Filter the global set down to the current partition's tablet ids (tabletIdsInOrder, already prepared by the caller) before returning. The result is identical to what the caller's null-check would have produced, so behavior is unchanged; only the redundant lookups and copies are eliminated. The non-Nereids path, the sampleTabletIds path and the empty-set
fallback are untouched.
Issue Number: close #63854
Related PR: #53403
Problem Summary:
Plan time of OlapScan queries with many partitions and many globally pruned tablets degrades quadratically due to redundant per-partition iterations over the global pruned tablet set in OlapScanNode.distributionPrune. Restore per-partition complexity by filtering the global set down to the current partition's tablets before returning.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)