plan: support topn push down in plan phase. #1769

hanfei1991 · 2016-09-27T08:38:43Z

It's a additional pr for #1760
It tries to push topn to table scan and index scan.
@coocood @shenli @zimulala @tiancaiamao @XuHuaiyu PTAL

coocood · 2016-09-27T10:12:46Z

Any test covers the case when selection condition can not be converted to PB, then topN will not be pushed?

coocood · 2016-09-27T09:19:06Z

plan/physical_plans.go

@@ -114,6 +113,7 @@ func (p *physicalTableSource) addTopN(prop *requiredProperty) {
 	for _, item := range prop.props {
 		p.SortItems = append(p.SortItems, sortByItemToPB(p.client, item.col, item.desc))


sortByItem may return nil, then addTopN should return false.

hanfei1991 · 2016-09-27T12:46:39Z

@coocood PTAL

XuHuaiyu · 2016-09-28T08:58:46Z

plan/physical_plans.go

-	if prop.limit != nil {
-		count := int64(prop.limit.Count + prop.limit.Offset)
-		p.LimitCount = &count
+	if prop.limit == nil || len(prop.props) == 0 {


There may be no need to judge "prop.limit == nil", it has been promised before invoking this function.

It's safer to check this again inside this function.

XuHuaiyu · 2016-09-28T09:10:17Z

plan/physical_plans.go

@@ -74,6 +72,7 @@ type physicalTableSource struct {
 	AggFuncs   []*tipb.Expr
 	GbyItems   []*tipb.ByItem

+	// ConditionPBExpr is the pb structure of conditions that pushed down.


pushed down --> be pushed down

coocood · 2016-09-28T11:54:41Z

The added test case doesn't make much sense as the test use a mock context which returns a nil Client, and TopN is not supported by local store.
So TopN will never be pushed down.

hanfei1991 · 2016-09-29T03:21:05Z

@coocood PTAL

coocood · 2016-09-29T03:28:29Z

plan/plan_test.go

@@ -373,6 +535,10 @@ func (s *testPlanSuite) TestCBO(c *C) {
 			best: "Index(t.c_d_e)[[-inf,10000)]->Sort + Limit(2) + Offset(0)->Projection",
 		},
 		{
+			sql:  "select * from t a where a.c < 10000 and a.d in (1000, a.e) order by a.a limit 2",


move this case to TestTopnPushDown and verify that top n is not pushed down.

@hanfei1991

XuHuaiyu · 2016-09-29T12:09:33Z

plan/match_property.go

@@ -106,21 +141,38 @@ func (is *PhysicalIndexScan) matchProperty(prop *requiredProperty, infos ...*phy
 		if allAsc {


if allAsc || allDesc{ if allDesc { sortedIs.Desc = true } // Duplicated code }

would this be more brief

@hanfei1991

XuHuaiyu · 2016-09-30T03:43:39Z

Please take a look at function handleLeftJoin/handleRightJoin in physical_plan_builder.go, the local variable allLeft/allRight been checked if false three times, but if allLeft/allRight is false, the function will be returned just in the first check, and there is no value write of allLeft/allRight after this check.

shenli · 2016-09-30T08:41:13Z

executor/executor_distsql.go

-		selIdxReq.OrderBy = append(selIdxReq.OrderBy, &tipb.ByItem{Desc: e.indexPlan.Desc})
+	if len(e.indexPlan.SortItems) > 0 {
+		selIdxReq.OrderBy = e.indexPlan.SortItems
+	} else if e.indexPlan.Desc {


How about e.indexPlan.SortItems > 0 and e.indexPlan.Desc is true?

@hanfei1991

If e.indexPlan.SortItems > 0, we don't care if the table is desc scanned or not.

XuHuaiyu · 2016-09-30T08:52:24Z

plan/plan_test.go

@@ -46,6 +50,10 @@ func (s *testPlanSuite) SetUpSuite(c *C) {
 	s.Parser = parser.New()
 }

+func newType() types.FieldType {


newTypeLong() be better?

shenli · 2016-10-08T02:26:46Z

executor/executor_distsql.go

+	if len(e.orderByList) > 0 {
+		selReq.OrderBy = e.orderByList
+	} else if e.supportDesc && e.desc {
+		selReq.OrderBy = []*tipb.ByItem{{Desc: e.desc}}


When will e.desc is true and len(e.orderByList) == 0?

If no topn is pushed, the order by list will be empty.

shenli · 2016-10-08T02:30:28Z

plan/match_property.go

 	if len(prop.props) == 0 {
-		return enforceProperty(prop, &physicalPlanInfo{p: ts, cost: cost, count: infos[0].count})
+		p := tryToAddUnionScan(ts.txn, ts.conditions, ts)


Why add union scan here?

"try to add union scan" will check if the txn is read-only. If it is, it will add union scan for it.

shenli · 2016-10-08T02:33:30Z

plan/match_property.go

 	}
 	if len(prop.props) == 1 && ts.pkCol != nil && ts.pkCol == prop.props[0].col {
 		sortedTs := *ts
 		sortedTs.Desc = prop.props[0].desc
 		sortedTs.KeepOrder = true
+		p := tryToAddUnionScan(ts.txn, ts.conditions, &sortedTs)


Should we also addTopN in this condition?

This condition means the pk column can match the required property.

shenli · 2016-10-08T02:33:58Z

plan/match_property.go

@@ -106,21 +141,38 @@ func (is *PhysicalIndexScan) matchProperty(prop *requiredProperty, infos ...*phy
 		if allAsc {


@hanfei1991

shenli · 2016-10-08T02:36:11Z

plan/plan.go

@@ -259,12 +259,8 @@ func (p *basePlan) initID() {
 // basePlan implements base Plan interface.
 // Should be used as embedded struct in Plan implementations.
 type basePlan struct {
-	fields      []*ast.ResultField
-	startupCost float64
-	totalCost   float64


These attributes are useless?

shenli · 2016-10-08T02:36:48Z

plan/plan_test.go

@@ -46,6 +50,10 @@ func (s *testPlanSuite) SetUpSuite(c *C) {
 	s.Parser = parser.New()
 }

+func newType() types.FieldType {


newLongType is better.

shenli · 2016-10-08T02:37:03Z

plan/plan_test.go

@@ -373,6 +535,10 @@ func (s *testPlanSuite) TestCBO(c *C) {
 			best: "Index(t.c_d_e)[[-inf,10000)]->Sort + Limit(2) + Offset(0)->Projection",
 		},
 		{
+			sql:  "select * from t a where a.c < 10000 and a.d in (1000, a.e) order by a.a limit 2",


@hanfei1991

hanfei1991 · 2016-10-08T08:05:51Z

@shenli @coocood @XuHuaiyu PTAL

shenli · 2016-10-10T08:51:46Z

plan/physical_plan_builder.go

@@ -406,9 +385,6 @@ func (p *Join) handleLeftJoin(prop *requiredProperty, innerJoin bool) (*physical
 			allLeft = false
 		}
 	}
-	if !allLeft {


Make best effort to push down sort. Because if there is a limit and sort, it will be converted to top-n. Running top-n as early as possible will save running time.

shenli · 2016-10-10T08:52:45Z

LGTM

coocood · 2016-10-10T06:09:10Z

plan/physical_plans.go

@@ -54,15 +53,14 @@ type PhysicalIndexScan struct {

 	accessEqualCount int
 	AccessCondition  []expression.Expression
-	// ConditionPBExpr is the pb structure of conditions that pushed down.
-	ConditionPBExpr *tipb.Expr
+	conditions       []expression.Expression

 	TableAsName *model.CIStr
 }

 type physicalDistSQLPlan interface {


Add comment about this interface and the methods, describe why we need this interface, how those method should be implemented and used.

coocood · 2016-10-10T06:20:05Z

plan/physical_plans.go

@@ -40,7 +39,7 @@ type PhysicalIndexScan struct {
 	basePlan
 	physicalTableSource

-	ctx        context.Context
+	txn        kv.Transaction


We don't need to keep a txn reference, we only need to know if it is read only.

coocood · 2016-10-10T06:21:05Z

plan/physical_plans.go

@@ -178,7 +179,7 @@ type PhysicalTableScan struct {
 	basePlan
 	physicalTableSource

-	ctx     context.Context
+	txn     kv.Transaction


coocood · 2016-10-11T03:44:38Z

LGTM

support topn in plan phase.

5673491

hanfei1991 changed the title ~~support topn in plan phase.~~ plan: support topn push down in plan phase. Sep 27, 2016

coocood reviewed Sep 27, 2016

View reviewed changes

add test

f7eedaf

XuHuaiyu reviewed Sep 28, 2016

View reviewed changes

add test.

c0cd833

hanfei1991 force-pushed the hanfei/topn branch from 731e8ef to c0cd833 Compare September 29, 2016 03:25

coocood reviewed Sep 29, 2016

View reviewed changes

XuHuaiyu reviewed Sep 29, 2016

View reviewed changes

shenli reviewed Sep 30, 2016

View reviewed changes

XuHuaiyu reviewed Sep 30, 2016

View reviewed changes

shenli reviewed Oct 8, 2016

View reviewed changes

hanfei1991 added 3 commits October 8, 2016 14:17

tiny change.

b52d6b0

Merge branch 'master' into hanfei/topn

d87dbf4

tiny change.

3d6faad

shenli reviewed Oct 10, 2016

View reviewed changes

coocood reviewed Oct 10, 2016

View reviewed changes

address comment.

967c560

Merge branch 'master' into hanfei/topn

0382a65

hanfei1991 merged commit 9b9c011 into master Oct 11, 2016

hanfei1991 deleted the hanfei/topn branch October 11, 2016 07:17

		@@ -114,6 +113,7 @@ func (p physicalTableSource) addTopN(prop requiredProperty) {
		for _, item := range prop.props {
		p.SortItems = append(p.SortItems, sortByItemToPB(p.client, item.col, item.desc))

		@@ -106,21 +141,38 @@ func (is PhysicalIndexScan) matchProperty(prop requiredProperty, infos ...*phy
		if allAsc {

plan: support topn push down in plan phase. #1769

plan: support topn push down in plan phase. #1769

Conversation

hanfei1991 commented Sep 27, 2016

coocood commented Sep 27, 2016 • edited

Choose a reason for hiding this comment

hanfei1991 commented Sep 27, 2016

XuHuaiyu Sep 28, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coocood commented Sep 28, 2016

hanfei1991 commented Sep 29, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XuHuaiyu Sep 29, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XuHuaiyu commented Sep 30, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hanfei1991 commented Oct 8, 2016

Choose a reason for hiding this comment

shenli commented Oct 10, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coocood commented Oct 11, 2016

coocood commented Sep 27, 2016 •

edited

XuHuaiyu Sep 28, 2016 •

edited

XuHuaiyu Sep 29, 2016 •

edited