planner: add aggregation hints `TIDB_HASHAGG` and `TIDB_STREAMAGG` #11364

foreyes · 2019-07-22T06:28:59Z

What problem does this PR solve?

Add Optimizer Hints TIDB_HASHAGG and TIDB_STREAMAGG.

What is changed and how it works?

Handle the hint from parser, and enforce planner to choose the aggregation type.
Related parser PR: pingcap/parser#394

mysql> explain select count(*) from t t1, t t2 where t1.a = t2.b;
+--------------------------+----------+------+--------------------------------------------------------------------+
| id                       | count    | task | operator info                                                      |
+--------------------------+----------+------+--------------------------------------------------------------------+
| StreamAgg_13             | 1.00     | root | funcs:count(1)                                                     |
| └─HashLeftJoin_26        | 12500.00 | root | inner join, inner:TableReader_20, equal:[eq(test.t1.a, test.t2.b)] |
|   ├─TableReader_22       | 10000.00 | root | data:TableScan_21                                                  |
|   │ └─TableScan_21       | 10000.00 | cop  | table:t1, range:[-inf,+inf], keep order:false, stats:pseudo        |
|   └─TableReader_20       | 10000.00 | root | data:TableScan_19                                                  |
|     └─TableScan_19       | 10000.00 | cop  | table:t2, range:[-inf,+inf], keep order:false, stats:pseudo        |
+--------------------------+----------+------+--------------------------------------------------------------------+
6 rows in set (0.00 sec)

mysql> explain select /*+ TIDB_HASHAGG() */ count(*) from t t1, t t2 where t1.a = t2.b;
+--------------------------+----------+------+--------------------------------------------------------------------+
| id                       | count    | task | operator info                                                      |
+--------------------------+----------+------+--------------------------------------------------------------------+
| HashAgg_11               | 1.00     | root | funcs:count(1)                                                     |
| └─HashLeftJoin_15        | 12500.00 | root | inner join, inner:TableReader_18, equal:[eq(test.t1.a, test.t2.b)] |
|   ├─TableReader_20       | 10000.00 | root | data:TableScan_19                                                  |
|   │ └─TableScan_19       | 10000.00 | cop  | table:t1, range:[-inf,+inf], keep order:false, stats:pseudo        |
|   └─TableReader_18       | 10000.00 | root | data:TableScan_17                                                  |
|     └─TableScan_17       | 10000.00 | cop  | table:t2, range:[-inf,+inf], keep order:false, stats:pseudo        |
+--------------------------+----------+------+--------------------------------------------------------------------+
6 rows in set (0.01 sec)

mysql> explain select count(t1.a) from t t1, t t2 where t1.a = t2.a*2 group by t1.a;
+--------------------------+----------+------+---------------------------------------------------------------------------+
| id                       | count    | task | operator info                                                             |
+--------------------------+----------+------+---------------------------------------------------------------------------+
| HashAgg_13               | 8000.00  | root | group by:test.t1.a, funcs:count(test.t1.a)                                |
| └─HashLeftJoin_16        | 12500.00 | root | inner join, inner:Projection_21, equal:[eq(test.t1.a, mul(test.t2.a, 2))] |
|   ├─TableReader_20       | 10000.00 | root | data:TableScan_19                                                         |
|   │ └─TableScan_19       | 10000.00 | cop  | table:t1, range:[-inf,+inf], keep order:false, stats:pseudo               |
|   └─Projection_21        | 10000.00 | root | test.t2.a, mul(test.t2.a, 2)                                              |
|     └─TableReader_23     | 10000.00 | root | data:TableScan_22                                                         |
|       └─TableScan_22     | 10000.00 | cop  | table:t2, range:[-inf,+inf], keep order:false, stats:pseudo               |
+--------------------------+----------+------+---------------------------------------------------------------------------+
7 rows in set (0.00 sec)

mysql> explain select /*+ TIDB_STREAMAGG() */ count(t1.a) from t t1, t t2 where t1.a = t2.a*2 group by t1.a;
+----------------------------+----------+------+---------------------------------------------------------------------------+
| id                         | count    | task | operator info                                                             |
+----------------------------+----------+------+---------------------------------------------------------------------------+
| StreamAgg_15               | 8000.00  | root | group by:test.t1.a, funcs:count(test.t1.a)                                |
| └─Sort_24                  | 12500.00 | root | test.t1.a:asc                                                             |
|   └─HashLeftJoin_16        | 12500.00 | root | inner join, inner:Projection_21, equal:[eq(test.t1.a, mul(test.t2.a, 2))] |
|     ├─TableReader_20       | 10000.00 | root | data:TableScan_19                                                         |
|     │ └─TableScan_19       | 10000.00 | cop  | table:t1, range:[-inf,+inf], keep order:false, stats:pseudo               |
|     └─Projection_21        | 10000.00 | root | test.t2.a, mul(test.t2.a, 2)                                              |
|       └─TableReader_23     | 10000.00 | root | data:TableScan_22                                                         |
|         └─TableScan_22     | 10000.00 | cop  | table:t2, range:[-inf,+inf], keep order:false, stats:pseudo               |
+----------------------------+----------+------+---------------------------------------------------------------------------+
8 rows in set (0.00 sec)

Check List

Tests

Unit test

Code changes

Change plan builder to handling aggregation hints.
Change exhaust physical plan to apply aggregation hints.

Side effects

Change optimizer behaviors.

Related changes

Add new rule in parser

codecov · 2019-07-22T07:38:30Z

Codecov Report

Merging #11364 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #11364   +/-   ##
===========================================
  Coverage   81.6243%   81.6243%           
===========================================
  Files           426        426           
  Lines         93640      93640           
===========================================
  Hits          76433      76433           
  Misses        11807      11807           
  Partials       5400       5400

foreyes · 2019-07-24T06:03:33Z

/run-all-tests

foreyes · 2019-07-24T06:20:39Z

PTAL. @zz-jason @lamxTyler

planner/core/exhaust_physical_plans.go

planner/core/logical_plans.go

foreyes · 2019-07-24T06:57:54Z

/run-all-tests

planner/core/logical_plans.go

planner/core/logical_plan_builder.go

planner/core/exhaust_physical_plans.go

foreyes · 2019-07-24T09:56:14Z

Code improved, PTAL. @zz-jason @XuHuaiyu

XuHuaiyu

LGTM

XuHuaiyu · 2019-07-25T02:36:16Z

We should update the version of parser in go.mod before merging this commit.

executor/join_test.go

foreyes · 2019-08-06T04:32:43Z

PTAL. @XuHuaiyu @eurekaka

XuHuaiyu · 2019-08-06T05:29:02Z

planner/core/exhaust_physical_plans.go

 	all, desc := prop.AllSameOrder()
-	if len(la.possibleProperties) == 0 || !all {
+	if !all {


This is not resolved?

XuHuaiyu · 2019-08-06T05:31:54Z

planner/core/physical_plan_test.go

@@ -1570,3 +1570,71 @@ func (s *testPlanSuite) TestIndexJoinHint(c *C) {
 		}
 	}
 }
+
+func (s *testPlanSuite) TestAggregationHints(c *C) {


Add a test case which contains subquery?

For the first one, because possibleChildProperties are only possible... We'd better not rely too much on it, I handle this in line 1272 - 1275, you can take a look.

For the test case, I will add them soon.

When adding test case, find another bug. Fix it soon...

foreyes · 2019-08-06T07:50:39Z

Add tests and fix a Merge Join bug, PTAL. @eurekaka @XuHuaiyu

eurekaka

LGTM

XuHuaiyu

LGTM

foreyes · 2019-08-07T02:45:51Z

/run-all-tests

XuHuaiyu · 2019-08-07T02:50:18Z

I'm still curious that, is this expectable?
Or, this will be fixed in another PR?

CREATE TABLE `t` (
  `a` int(11) DEFAULT NULL,
  `b` int(11) DEFAULT NULL,
  KEY `a` (`a`)
);
tidb> desc select /*+ TIDB_HJ(t)  */ a, count(b) from t group by a order by a;
+--------------------------+----------+------+--------------------------------------------------------------------+
| id                       | count    | task | operator info                                                      |
+--------------------------+----------+------+--------------------------------------------------------------------+
| Projection_22            | 8000.00  | root | test.t.a, 2_col_0                                                  |
| └─StreamAgg_24           | 8000.00  | root | group by:test.t.a, funcs:count(test.t.b), firstrow(test.t.a)       |
|   └─Projection_21        | 10000.00 | root | test.t.a, test.t.b                                                 |
|     └─IndexLookUp_20     | 10000.00 | root |                                                                    |
|       ├─IndexScan_18     | 10000.00 | cop  | table:t, index:a, range:[NULL,+inf], keep order:true, stats:pseudo |
|       └─TableScan_19     | 10000.00 | cop  | table:t, keep order:false, stats:pseudo                            |
+--------------------------+----------+------+--------------------------------------------------------------------+
6 rows in set, 1 warning (0.00 sec)

tidb> show warnings;
+---------+------+-----------------------------------------------------------------------------------------------------------------------+
| Level   | Code | Message                                                                                                               |
+---------+------+-----------------------------------------------------------------------------------------------------------------------+
| Warning | 1815 | There are no matching table names for (t) in optimizer hint /*+ TIDB_HJ(t) */. Maybe you can use the table alias name |
+---------+------+-----------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

foreyes · 2019-08-07T02:53:07Z

I'm still curious that, is this expectable?
Or, this will be fixed in another PR?

CREATE TABLE `t` (
  `a` int(11) DEFAULT NULL,
  `b` int(11) DEFAULT NULL,
  KEY `a` (`a`)
);
tidb> desc select /*+ TIDB_HJ(t)  */ a, count(b) from t group by a order by a;
+--------------------------+----------+------+--------------------------------------------------------------------+
| id                       | count    | task | operator info                                                      |
+--------------------------+----------+------+--------------------------------------------------------------------+
| Projection_22            | 8000.00  | root | test.t.a, 2_col_0                                                  |
| └─StreamAgg_24           | 8000.00  | root | group by:test.t.a, funcs:count(test.t.b), firstrow(test.t.a)       |
|   └─Projection_21        | 10000.00 | root | test.t.a, test.t.b                                                 |
|     └─IndexLookUp_20     | 10000.00 | root |                                                                    |
|       ├─IndexScan_18     | 10000.00 | cop  | table:t, index:a, range:[NULL,+inf], keep order:true, stats:pseudo |
|       └─TableScan_19     | 10000.00 | cop  | table:t, keep order:false, stats:pseudo                            |
+--------------------------+----------+------+--------------------------------------------------------------------+
6 rows in set, 1 warning (0.00 sec)

tidb> show warnings;
+---------+------+-----------------------------------------------------------------------------------------------------------------------+
| Level   | Code | Message                                                                                                               |
+---------+------+-----------------------------------------------------------------------------------------------------------------------+
| Warning | 1815 | There are no matching table names for (t) in optimizer hint /*+ TIDB_HJ(t) */. Maybe you can use the table alias name |
+---------+------+-----------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

I know this case, it's expected, but looks weird, I will fix it in another PR. @XuHuaiyu

…ingcap#11364)

foreyes added sig/planner SIG: Planner status/WIP type/new-feature labels Jul 22, 2019

foreyes mentioned this pull request Jul 22, 2019

parser: add TableOptimizerHint TIDB_HASHAGG and TIDB_STREAMAGG pingcap/parser#394

Merged

foreyes force-pushed the dev/add_agg_hints branch 2 times, most recently from ecd3e00 to ff1f239 Compare July 23, 2019 07:05

foreyes removed the status/WIP label Jul 24, 2019

foreyes requested review from alivxxx and zz-jason July 24, 2019 06:19

foreyes changed the title ~~[WIP] planner: add aggregation hints TIDB_HASHAGG and TIDB_STREAMAGG~~ planner: add aggregation hints TIDB_HASHAGG and TIDB_STREAMAGG Jul 24, 2019

alivxxx reviewed Jul 24, 2019

View reviewed changes

planner/core/exhaust_physical_plans.go Outdated Show resolved Hide resolved

planner/core/logical_plans.go Outdated Show resolved Hide resolved

foreyes force-pushed the dev/add_agg_hints branch from 37eff4e to 5accbd8 Compare July 24, 2019 06:53

XuHuaiyu reviewed Jul 24, 2019

View reviewed changes

planner/core/logical_plans.go Outdated Show resolved Hide resolved

planner/core/logical_plan_builder.go Outdated Show resolved Hide resolved

planner/core/exhaust_physical_plans.go Outdated Show resolved Hide resolved

zz-jason removed their request for review July 24, 2019 08:39

foreyes force-pushed the dev/add_agg_hints branch from fa37100 to 2c46284 Compare July 24, 2019 09:42

XuHuaiyu reviewed Jul 25, 2019

View reviewed changes

XuHuaiyu added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 25, 2019

foreyes force-pushed the dev/add_agg_hints branch 3 times, most recently from 80af97d to 74c1bf3 Compare July 26, 2019 09:52

foreyes added 4 commits July 30, 2019 10:15

add aggregation hints

e87fad5

try to fix ci

0184392

add tests

1f0e8c4

add enforce flag to stream agg hint & fix test

9a1a791

resolve comments

7dd61f9

alivxxx reviewed Aug 6, 2019

View reviewed changes

executor/join_test.go Outdated Show resolved Hide resolved

avoid test leak

d2246e5

foreyes mentioned this pull request Aug 6, 2019

Extract a common error message pattern for conflicted SQL hints and hint alias #11632

Open

foreyes requested review from eurekaka and XuHuaiyu August 6, 2019 04:32

XuHuaiyu reviewed Aug 6, 2019

View reviewed changes

foreyes added 3 commits August 6, 2019 15:10

add tests & fix bug

0a122d6

fix explain test

8c49116

fix test

9d4c316

eurekaka reviewed Aug 6, 2019

View reviewed changes

Merge branch 'master' into dev/add_agg_hints

2609f8b

eurekaka added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 6, 2019

XuHuaiyu approved these changes Aug 7, 2019

View reviewed changes

Merge branch 'master' into dev/add_agg_hints

081b602

XuHuaiyu added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Aug 7, 2019

foreyes merged commit a530f87 into pingcap:master Aug 7, 2019

foreyes deleted the dev/add_agg_hints branch August 7, 2019 02:51

foreyes mentioned this pull request Aug 14, 2019

planner: revert optimizer hint scope #11740

Merged

foreyes added a commit to foreyes/tidb that referenced this pull request Sep 16, 2019

planner: add aggregation hints TIDB_HASHAGG and TIDB_STREAMAGG (p…

2dd2878

…ingcap#11364)

foreyes mentioned this pull request Sep 16, 2019

planner: cherry-pick sql hint related PRs into release-3.1 #12192

Merged

zz-jason mentioned this pull request Oct 10, 2019

Feature Request: Richer SQL Hints #12304

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

planner: add aggregation hints `TIDB_HASHAGG` and `TIDB_STREAMAGG` #11364

planner: add aggregation hints `TIDB_HASHAGG` and `TIDB_STREAMAGG` #11364

foreyes commented Jul 22, 2019 •

edited

codecov bot commented Jul 22, 2019 •

edited

foreyes commented Jul 24, 2019

foreyes commented Jul 24, 2019

foreyes commented Jul 24, 2019

foreyes commented Jul 24, 2019

XuHuaiyu left a comment

XuHuaiyu commented Jul 25, 2019

foreyes commented Aug 6, 2019

XuHuaiyu Aug 6, 2019

XuHuaiyu Aug 6, 2019

foreyes Aug 6, 2019

foreyes Aug 6, 2019

foreyes commented Aug 6, 2019

eurekaka left a comment

XuHuaiyu left a comment

foreyes commented Aug 7, 2019

XuHuaiyu commented Aug 7, 2019

foreyes commented Aug 7, 2019 •

edited

planner: add aggregation hints TIDB_HASHAGG and TIDB_STREAMAGG #11364

planner: add aggregation hints TIDB_HASHAGG and TIDB_STREAMAGG #11364

Conversation

foreyes commented Jul 22, 2019 • edited

What problem does this PR solve?

What is changed and how it works?

Check List

codecov bot commented Jul 22, 2019 • edited

Codecov Report

foreyes commented Jul 24, 2019

foreyes commented Jul 24, 2019

foreyes commented Jul 24, 2019

foreyes commented Jul 24, 2019

XuHuaiyu left a comment

Choose a reason for hiding this comment

XuHuaiyu commented Jul 25, 2019

foreyes commented Aug 6, 2019

XuHuaiyu Aug 6, 2019

Choose a reason for hiding this comment

XuHuaiyu Aug 6, 2019

Choose a reason for hiding this comment

foreyes Aug 6, 2019

Choose a reason for hiding this comment

foreyes Aug 6, 2019

Choose a reason for hiding this comment

foreyes commented Aug 6, 2019

eurekaka left a comment

Choose a reason for hiding this comment

XuHuaiyu left a comment

Choose a reason for hiding this comment

foreyes commented Aug 7, 2019

XuHuaiyu commented Aug 7, 2019

foreyes commented Aug 7, 2019 • edited

planner: add aggregation hints `TIDB_HASHAGG` and `TIDB_STREAMAGG` #11364

planner: add aggregation hints `TIDB_HASHAGG` and `TIDB_STREAMAGG` #11364

foreyes commented Jul 22, 2019 •

edited

codecov bot commented Jul 22, 2019 •

edited

foreyes commented Aug 7, 2019 •

edited