refactor(interactive): Support Nested Branch Logical Plan #3151
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Committed-by: bingqing.lbq from Dev container
Committed-by: bingqing.lbq from Dev container
…hScope into ir_multiple_match
1. Fix bugs in `get_merge_node` and `get_branch_node` method when facing more comlicated logical plan, now it will uses more strict condition for final merge/first branch checking 2. Fix a bug in `append_branch_plan`, when branch plan having a node with id already exist in the original plan, now it will not cover the existing node, but it will now merge the children of the two nodes. It solves the problem in finding subplans that two branch nodes overlap 3. Add more test cases to validate the new `subplan` and `get_branch_plans` method
1. Add 6 types logical plans for testing 1.1 Verify the correctness `get_merge_node`, `get_branch_node`, `subplan`, `get_branch_plans` with these logical plans 1.2 Verify whether these logical plan can converted to expected physical plan 1.3 Verify whether the query based on these logical plans can generate expected results 2. Use flow algorithm in `get_merge_node` and `get_branch_node`, greatly improve the efficiency 2.1 Implement a Fraction abstraction for the precision during flow comparison 3. Solve a problem in `append_branch_plans`, now it supports to append branch plans which having nodes alreading existing in the original plan 4. Completely rewrite the logic of `get_merge_node`, `get_branch_node`, `subplan`, `get_branch_plans 4.1 Now it in theory can support arbitrary DAG logical plan as long as it is reasonable 5. Solve a problem in `extract_subplans`, now the last node of a subplan can be a merge node 6. Make the code much safer that now it will check the existence of merge node subplans much more strictly 6.1 Remove `expect`, instead throws None or Error 7. Add more comments which suggests the logic for handling nested branches
MeloYang05
changed the title
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #3151 +/- ##
=======================================
Coverage 42.05% 42.05%
=======================================
Files 101 101
Lines 10985 10985
=======================================
Hits 4620 4620
Misses 6365 6365 Continue to review full report in Codecov by Sentry.
|
This commit edit constructors for LogicalPlan: Previous Constructor -> Current Constructor 1.LogicalPlan::with_root() -> LogicalPlan::with_node(), to distringuish from the pb::Root operator 2.LogicalPlan::default() -> LogicalPlan::with_root(), the LogicalPlan is initialized with a pb::Root Node 3.LogicalPlan::empty() -> LogicalPlan::default(), the LogicalPlan is initialized as empty
longbinlai
previously approved these changes
Aug 29, 2023
| } | ||
| } | ||
|
|
||
| impl Add for Fraction { |
There was a problem hiding this comment.
Fraction and GCD, using some rust package.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What do these changes do?
In the previous implementation, a branch node in the GIE logical plan was required to correspond to exactly one merge node. However, it is common to encounter nested branch plans in real-world scenarios, where a branch node may correspond to multiple merge nodes at different levels.
For example:
root(1)
/ |
2 3 4
\ / /
5 /
\ /
6
To support these situations, this pull request updates the implementation of
get_branch_plans,sub_plans,get_merge_node, and other related methods. As a result, the logical plan can theoretically accommodate arbitrary branches, as long as they are reasonable.