Skip to content

Conversation

@mirober
Copy link
Contributor

@mirober mirober commented Jun 25, 2022

Which issue does this PR close?

Closes #2573.

Rationale for this change

What changes are included in this PR?

Adds a new type of logical plan for DISTINCT operations, which is converted into an aggregation during physical planning.

Something I am unsure of: all of the LogicalPlan structs contain Arc<LogicalPlan> where they refer to other plans, except Union, which has a Vec<LogicalPlan>. I think this caused a couple of extra LogicalPlan clones due to needing an owned LogicalPlan to put into a union. Should Union contain Vec<Arc<LogicalPlan>> instead?

Are there any user-facing changes?

@github-actions github-actions bot added core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules sql SQL Planner labels Jun 25, 2022
@codecov-commenter
Copy link

Codecov Report

Merging #2792 (bc4193e) into master (3da59e8) will increase coverage by 0.18%.
The diff coverage is 84.21%.

@@            Coverage Diff             @@
##           master    #2792      +/-   ##
==========================================
+ Coverage   84.94%   85.13%   +0.18%     
==========================================
  Files         272      273       +1     
  Lines       48221    48293      +72     
==========================================
+ Hits        40963    41113     +150     
+ Misses       7258     7180      -78     
Impacted Files Coverage Δ
...tafusion/optimizer/src/common_subexpr_eliminate.rs 94.11% <ø> (ø)
datafusion/optimizer/src/projection_push_down.rs 98.08% <ø> (ø)
datafusion/proto/src/logical_plan.rs 17.63% <22.22%> (+0.01%) ⬆️
datafusion/expr/src/logical_plan/plan.rs 74.21% <83.33%> (+0.30%) ⬆️
datafusion/expr/src/logical_plan/builder.rs 89.73% <96.29%> (+0.22%) ⬆️
datafusion/core/src/physical_plan/planner.rs 81.02% <100.00%> (+0.15%) ⬆️
datafusion/expr/src/utils.rs 90.73% <100.00%> (+0.05%) ⬆️
datafusion/sql/src/planner.rs 81.34% <100.00%> (-0.02%) ⬇️
datafusion/core/src/datasource/listing/table.rs 89.13% <0.00%> (-1.23%) ⬇️
datafusion/core/src/config.rs 90.32% <0.00%> (-0.79%) ⬇️
... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3da59e8...bc4193e. Read the comment docs.

@alamb alamb requested a review from andygrove June 26, 2022 13:10
@alamb
Copy link
Contributor

alamb commented Jun 27, 2022

cc @andygrove

@andygrove
Copy link
Member

Thanks @mrob95. I plan on reviewing this later this week.

Copy link
Member

@andygrove andygrove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mrob95 this looks great. I think you raise a good point about Union containing Vec<LogicalPlan> rather than Vec<Arc<LogicalPlan>> but I think that should be fixed as a separate issue/PR.

@andygrove
Copy link
Member

@mrob95 There is a conflict that needs resolving. I can merge once that is resolved.

@mirober
Copy link
Contributor Author

mirober commented Jun 28, 2022

Thanks, should be fixed now 🙂

@alamb
Copy link
Contributor

alamb commented Jun 30, 2022

Filed #2816 to track the Union inconstency

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules sql SQL Planner

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve UX for UNION vs UNION ALL (introduce a LogicalPlan::Distinct)

4 participants