Skip to content

Conversation

@jimexist
Copy link
Member

@jimexist jimexist commented Jul 19, 2021

Which issue does this PR close?

fix explain verbose with limit push down

re #733

Rationale for this change

What changes are included in this PR?

before fix:

> create external table test stored as PARQUET location '/Users/jiayu_liu/Desktop/aggregate_test_100.parquet';
0 rows in set. Query took 0.001 seconds.
> explain verbose select c3, c9 from test limit 3;
+-----------------------------------------+----------------------------------------------------------------------------------------------------------------+
| plan_type                               | plan                                                                                                           |
+-----------------------------------------+----------------------------------------------------------------------------------------------------------------+
| logical_plan                            | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=None                                                                            |
| logical_plan after projection_push_down | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8])                                                                    |
| logical_plan after simplify_expressions | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8])                                                                    |
| logical_plan after projection_push_down | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8])                                                                    |
| logical_plan after simplify_expressions | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8])                                                                    |
| physical_plan                           | GlobalLimitExec: limit=3                                                                                       |
|                                         |   ProjectionExec: expr=[c3@0 as c3, c9@1 as c9]                                                                |
|                                         |     ParquetExec: batch_size=8192, limit=None, partitions=[/Users/jiayu_liu/Desktop/aggregate_test_100.parquet] |
+-----------------------------------------+----------------------------------------------------------------------------------------------------------------+

after the fix

> create external table test stored as PARQUET location '/Users/jiayu_liu/Desktop/aggregate_test_100.parquet';
0 rows in set. Query took 0.001 seconds.
> explain verbose select c3, c9 from test limit 3;
+-----------------------------------------+----------------------------------------------------------------------------------------------------------------+
| plan_type                               | plan                                                                                                           |
+-----------------------------------------+----------------------------------------------------------------------------------------------------------------+
| logical_plan                            | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=None                                                                            |
| logical_plan after projection_push_down | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8])                                                                    |
| logical_plan after simplify_expressions | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8])                                                                    |
| logical_plan after limit_push_down      | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8]), limit=3                                                           |
| logical_plan after projection_push_down | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8]), limit=3                                                           |
| logical_plan after simplify_expressions | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8]), limit=3                                                           |
| logical_plan after limit_push_down      | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8]), limit=3                                                           |
| physical_plan                           | GlobalLimitExec: limit=3                                                                                       |
|                                         |   ProjectionExec: expr=[c3@0 as c3, c9@1 as c9]                                                                |
|                                         |     ParquetExec: batch_size=3, limit=Some(3), partitions=[/Users/jiayu_liu/Desktop/aggregate_test_100.parquet] |
+-----------------------------------------+----------------------------------------------------------------------------------------------------------------+

Are there any user-facing changes?

@alamb alamb changed the title fix explain verbose with limit push down Add explain verbose with limit push down Jul 19, 2021
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @jimexist -- this definitely is an improvement (and relates to #733)

One thing that bothers me about #733 is that each optimizer pass needs to handle explain plans specially, but I think that means we end up missing some of them (and also not catching user defined optimizer passes)

I am thinking it might be better to (eventually) handle explain plan capturing in the planners. I'll try and work on that after #744

@alamb alamb merged commit 6f9681d into apache:master Jul 19, 2021
@jimexist jimexist deleted the fix-explain-limit-push-down branch July 19, 2021 14:19
@Dandandan
Copy link
Contributor

Nice fix @jimexist

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants