Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

planner,expression: use constraint propagation in partition pruning #8885

Merged
merged 26 commits into from Jan 17, 2019

Conversation

@tiancaiamao
Copy link
Contributor

tiancaiamao commented Dec 29, 2018

What problem does this PR solve?

Fix #7516

Now we can prune some of the to_days function:

CREATE TABLE t1 (
a int(10) unsigned NOT NULL,
b DATETIME NOT NULL,
PRIMARY KEY (a, b)
) PARTITION BY RANGE (TO_DAYS(b))
(PARTITION p20090401 VALUES LESS THAN (TO_DAYS('2009-04-02')),
PARTITION p20090402 VALUES LESS THAN (TO_DAYS('2009-04-03')),
PARTITION p20090403 VALUES LESS THAN (TO_DAYS('2009-04-04')),
PARTITION p20090404 VALUES LESS THAN (TO_DAYS('2009-04-05')),
PARTITION p20090405 VALUES LESS THAN MAXVALUE);

The partition expression is TO_DAYS(b) < XXX and TO_DAYS(b) >= YYY, it has a function to_days.

EXPLAIN SELECT * FROM t1 WHERE b <= CAST('2009-04-03' AS DATE);
id	count	task	operator info
Union_9	9970.00	root	
├─TableReader_12	3323.33	root	data:Selection_11
│ └─Selection_11	3323.33	cop	le(test.t1.b, 2009-04-03)
│   └─TableScan_10	10000.00	cop	table:t1, partition:p20090401, range:[-inf,+inf], keep order:false, stats:pseudo
├─TableReader_15	3323.33	root	data:Selection_14
│ └─Selection_14	3323.33	cop	le(test.t1.b, 2009-04-03)
│   └─TableScan_13	10000.00	cop	table:t1, partition:p20090402, range:[-inf,+inf], keep order:false, stats:pseudo
└─TableReader_18	3323.33	root	data:Selection_17
  └─Selection_17	3323.33	cop	le(test.t1.b, 2009-04-03)
    └─TableScan_16	10000.00	cop	table:t1, partition:p20090403, range:[-inf,+inf], keep order:false, stats:pseudo

From the result we can see that p20090404, p20090405 is pruned.

What is changed and how it works?

Fix those two restrictions mentioned in the issue:

  1. it doesn't support filter conditions that can't be push down to datasource
  2. it doesn't support expressions that can't calculate range

Now prune will consider both push down conditions and the filter conditions in the selection.
Leverage the constraint propagate rule in #8640
to handle expressions (function) that can't calculate range.

There are still two minor problems, but they are not introduced by this commit:

  1. The partition expressions for the first partition is col < const or col is null, current partition pruning can't prune or col is null
  2. Range calculating is still kept in the code, we should get rid of it eventually

Check List

Tests

  • Unit test
  • Integration test

This change is Reviewable

@zz-jason

This comment has been minimized.

Copy link
Member

zz-jason commented Jan 2, 2019

@tiancaiamao Please add some proper labels. For example, component, type.

@tiancaiamao

This comment has been minimized.

Copy link
Contributor Author

tiancaiamao commented Jan 2, 2019

This is still WIP, and the changes are based on #8640
@zz-jason

@tiancaiamao tiancaiamao removed the status/WIP label Jan 4, 2019

@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Jan 4, 2019

Codecov Report

❗️ No coverage uploaded for pull request base (master@59c7b69). Click here to learn what that means.
The diff coverage is 72.72%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #8885   +/-   ##
=========================================
  Coverage          ?   67.14%           
=========================================
  Files             ?      372           
  Lines             ?    76968           
  Branches          ?        0           
=========================================
  Hits              ?    51679           
  Misses            ?    20657           
  Partials          ?     4632
Impacted Files Coverage Δ
planner/core/logical_plans.go 73.72% <ø> (ø)
planner/core/rule_predicate_push_down.go 90.63% <100%> (ø)
expression/constraint_propagation.go 73.45% <54.54%> (ø)
planner/core/rule_partition_processor.go 74.02% <80.95%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 59c7b69...89e597f. Read the comment docs.

@tiancaiamao

This comment has been minimized.

Copy link
Contributor Author

tiancaiamao commented Jan 4, 2019

@tiancaiamao tiancaiamao added this to In Progress in Table Partitioning via automation Jan 4, 2019

@zz-jason zz-jason requested review from eurekaka and zz-jason Jan 6, 2019

@tiancaiamao

This comment has been minimized.

Copy link
Contributor Author

tiancaiamao commented Jan 7, 2019

Show resolved Hide resolved expression/constraint_propagation.go
Show resolved Hide resolved planner/core/planbuilder_test.go Outdated
Show resolved Hide resolved planner/core/rule_partition_processor.go Outdated
Show resolved Hide resolved planner/core/rule_partition_processor.go Outdated
Show resolved Hide resolved planner/core/rule_partition_processor.go Outdated
Show resolved Hide resolved planner/core/rule_partition_processor.go Outdated
Show resolved Hide resolved planner/core/rule_partition_processor.go Outdated
Show resolved Hide resolved planner/core/rule_partition_processor.go Outdated
Show resolved Hide resolved planner/core/rule_partition_processor.go Outdated
Show resolved Hide resolved planner/core/rule_partition_processor.go Outdated
Show resolved Hide resolved planner/core/rule_partition_processor.go Outdated

tiancaiamao added some commits Jan 7, 2019

@tiancaiamao

This comment has been minimized.

Copy link
Contributor Author

tiancaiamao commented Jan 8, 2019

@eurekaka
Copy link
Contributor

eurekaka left a comment

LGTM

@eurekaka eurekaka added the status/LGT1 label Jan 8, 2019

Show resolved Hide resolved expression/constraint_propagation.go
Show resolved Hide resolved planner/core/logical_plans.go Outdated
Show resolved Hide resolved planner/core/planbuilder_test.go Outdated
Show resolved Hide resolved planner/core/rule_partition_processor.go Outdated
Show resolved Hide resolved planner/core/rule_partition_processor.go Outdated
}
}

if partCol == nil {

This comment has been minimized.

Copy link
@zz-jason

zz-jason Jan 8, 2019

Member

can we check partCol == nil first when entering this function?

This comment has been minimized.

Copy link
@tiancaiamao

tiancaiamao Jan 8, 2019

Author Contributor

Do you mean check partCol == nil at the beginning of this function and return false? No we can't do that.

In to_days(c) > xx and c < yy, partCol is nil, but I want to handle it.
If this branch moved to the beginning, the canBePruned will return immediately.

This comment has been minimized.

Copy link
@zz-jason

zz-jason Jan 8, 2019

Member

Got it. Could you add a comment about this to let others know why we intentionally put this check after solver.Solve()?

Show resolved Hide resolved planner/core/rule_partition_processor.go
Show resolved Hide resolved planner/core/rule_partition_processor.go Outdated
@zz-jason
Copy link
Member

zz-jason left a comment

LGTM

@zz-jason zz-jason requested a review from winoros Jan 11, 2019

@tiancaiamao

This comment has been minimized.

Copy link
Contributor Author

tiancaiamao commented Jan 15, 2019

/run-all-tests

@tiancaiamao

This comment has been minimized.

Copy link
Contributor Author

tiancaiamao commented Jan 15, 2019

/run-unit-test

@tiancaiamao

This comment has been minimized.

Copy link
Contributor Author

tiancaiamao commented Jan 15, 2019

/run-all-tests

@tiancaiamao

This comment has been minimized.

Copy link
Contributor Author

tiancaiamao commented Jan 15, 2019

@tiancaiamao

This comment has been minimized.

Copy link
Contributor Author

tiancaiamao commented Jan 16, 2019

PTAL @winoros

@tiancaiamao

This comment has been minimized.

Copy link
Contributor Author

tiancaiamao commented Jan 17, 2019

@tiancaiamao

This comment has been minimized.

Copy link
Contributor Author

tiancaiamao commented Jan 17, 2019

/run-all-tests

@tiancaiamao tiancaiamao merged commit b339c02 into pingcap:master Jan 17, 2019

13 checks passed

ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
idc-jenkins-ci-tidb/build Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/check_dev Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/common-test job succeeded
Details
idc-jenkins-ci-tidb/integration-common-test Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/integration-compatibility-test Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/integration-ddl-test Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/mybatis-test job succeeded
Details
idc-jenkins-ci-tidb/sqllogic-test-1 Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/sqllogic-test-2 Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/unit-test Jenkins job succeeded.
Details
license/cla Contributor License Agreement is signed.
Details

Table Partitioning automation moved this from In Progress to Done Jan 17, 2019

@tiancaiamao tiancaiamao deleted the tiancaiamao:re-partition-prune branch Jan 17, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.