# planner,expression: use constraint propagation in partition pruning #8885

Merged
merged 26 commits into from Jan 17, 2019

## Conversation

Projects
6 participants
Contributor

### What problem does this PR solve?

Fix #7516

Now we can prune some of the `to_days` function:

``````CREATE TABLE t1 (
a int(10) unsigned NOT NULL,
b DATETIME NOT NULL,
PRIMARY KEY (a, b)
) PARTITION BY RANGE (TO_DAYS(b))
(PARTITION p20090401 VALUES LESS THAN (TO_DAYS('2009-04-02')),
PARTITION p20090402 VALUES LESS THAN (TO_DAYS('2009-04-03')),
PARTITION p20090403 VALUES LESS THAN (TO_DAYS('2009-04-04')),
PARTITION p20090404 VALUES LESS THAN (TO_DAYS('2009-04-05')),
PARTITION p20090405 VALUES LESS THAN MAXVALUE);
``````

The partition expression is `TO_DAYS(b) < XXX and TO_DAYS(b) >= YYY`, it has a function `to_days`.

``````EXPLAIN SELECT * FROM t1 WHERE b <= CAST('2009-04-03' AS DATE);
Union_9	9970.00	root
│ └─Selection_11	3323.33	cop	le(test.t1.b, 2009-04-03)
│   └─TableScan_10	10000.00	cop	table:t1, partition:p20090401, range:[-inf,+inf], keep order:false, stats:pseudo
│ └─Selection_14	3323.33	cop	le(test.t1.b, 2009-04-03)
│   └─TableScan_13	10000.00	cop	table:t1, partition:p20090402, range:[-inf,+inf], keep order:false, stats:pseudo
└─Selection_17	3323.33	cop	le(test.t1.b, 2009-04-03)
└─TableScan_16	10000.00	cop	table:t1, partition:p20090403, range:[-inf,+inf], keep order:false, stats:pseudo
``````

From the result we can see that `p20090404`, `p20090405` is pruned.

### What is changed and how it works?

Fix those two restrictions mentioned in the issue:

1. it doesn't support filter conditions that can't be push down to datasource
2. it doesn't support expressions that can't calculate range

Now prune will consider both push down conditions and the filter conditions in the `selection`.
Leverage the constraint propagate rule in #8640
to handle expressions (function) that can't calculate range.

There are still two minor problems, but they are not introduced by this commit:

1. The partition expressions for the first partition is `col < const or col is null`, current partition pruning can't prune `or col is null`
2. Range calculating is still kept in the code, we should get rid of it eventually

### Check List

Tests

• Unit test
• Integration test

This change is

### tiancaiamao added some commits Dec 10, 2018

``` expression: constraint propagate for '>' and monotonous function ```
``` 6cbf528 ```
``` Merge branch 'master' into propagate-pruning ```
``` f966b75 ```
``` Merge master ```
``` e11f0ed ```
``` gofmt ```
``` 12d9212 ```
``` fix CI ```
``` 3376571 ```
``` fix CI ```
``` 9a3659b ```
``` address comment ```
``` b3287fc ```
``` Merge branch 'master' into propagate-pruning ```
``` 0f08c3e ```
``` planner,expression: use constraint propagation in partition pruning ```
``` 934bddd ```

Member

### tiancaiamao added some commits Jan 2, 2019

``` make the code more general and remove useless part ```
``` bdcf799 ```
``` Merge branch 'propagate-pruning' into re-partition-prune ```
``` c661aa7 ```
``` update test ```
``` c3bddb8 ```

### tiancaiamao added component/planner component/expression labels Jan 2, 2019

Contributor Author

### tiancaiamao commented Jan 2, 2019

 This is still WIP, and the changes are based on #8640 @zz-jason

### tiancaiamao added some commits Jan 4, 2019

``` Merge branch 'master' into re-partition-prune ```
``` 1939df5 ```
``` handle filter conditions that do not push down in partition pruning ```
``` 32dae16 ```
``` update partition pruning test in explain ```
``` 4a5735e ```

# Codecov Report

❗️ No coverage uploaded for pull request base (`master@59c7b69`). Click here to learn what that means.
The diff coverage is `72.72%`.

```@@            Coverage Diff            @@
##             master    #8885   +/-   ##
=========================================
Coverage          ?   67.14%
=========================================
Files             ?      372
Lines             ?    76968
Branches          ?        0
=========================================
Hits              ?    51679
Misses            ?    20657
Partials          ?     4632```
Impacted Files Coverage Δ
planner/core/logical_plans.go `73.72% <ø> (ø)`
planner/core/rule_predicate_push_down.go `90.63% <100%> (ø)`
expression/constraint_propagation.go `73.45% <54.54%> (ø)`
planner/core/rule_partition_processor.go `74.02% <80.95%> (ø)`

`Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`

Contributor Author

### zz-jason requested review from eurekaka and zz-jasonJan 6, 2019

Contributor Author

### zz-jason reviewed Jan 7, 2019

expression/constraint_propagation.go
planner/core/planbuilder_test.go Outdated
planner/core/rule_partition_processor.go Outdated
planner/core/rule_partition_processor.go Outdated
planner/core/rule_partition_processor.go Outdated
planner/core/rule_partition_processor.go Outdated
planner/core/rule_partition_processor.go Outdated
planner/core/rule_partition_processor.go Outdated

### eurekaka reviewed Jan 7, 2019

planner/core/rule_partition_processor.go Outdated
planner/core/rule_partition_processor.go Outdated
planner/core/rule_partition_processor.go Outdated
``` address comment ```
``` bf90c2a ```

### tiancaiamao added some commits Jan 7, 2019

``` fix ci ```
``` 10fb9b1 ```
``` address comment ```
``` 084ce7a ```
``` address comment ```
``` 8518cc0 ```
Contributor Author

Contributor

 LGTM

### zz-jason reviewed Jan 8, 2019

expression/constraint_propagation.go
planner/core/logical_plans.go Outdated
planner/core/planbuilder_test.go Outdated
planner/core/rule_partition_processor.go Outdated
planner/core/rule_partition_processor.go Outdated
 } } if partCol == nil {

#### zz-jason Jan 8, 2019

Member

can we check `partCol == nil` first when entering this function?

#### tiancaiamao Jan 8, 2019

Author Contributor

Do you mean check `partCol == nil` at the beginning of this function and return false? No we can't do that.

In `to_days(c) > xx and c < yy`, `partCol` is nil, but I want to handle it.
If this branch moved to the beginning, the `canBePruned` will return immediately.

#### zz-jason Jan 8, 2019

Member

Got it. Could you add a comment about this to let others know why we intentionally put this check after `solver.Solve()`?

planner/core/rule_partition_processor.go
planner/core/rule_partition_processor.go Outdated
``` address comment ```
``` 03aff65 ```

Member

 LGTM

### winoros reviewed Jan 14, 2019

planner/core/partition_pruning_test.go Outdated
``` address comment ```
``` 5bec8fc ```
Contributor Author

### tiancaiamao commented Jan 15, 2019

 /run-all-tests
Contributor Author

### tiancaiamao commented Jan 15, 2019

 /run-unit-test
``` Merge branch 'master' into re-partition-prune ```
``` 7d57f56 ```
Contributor Author

### tiancaiamao commented Jan 15, 2019

 /run-all-tests
Contributor Author

### tiancaiamao added some commits Jan 15, 2019

``` fix typo ```
``` 0a52d72 ```
``` Merge branch 'master' into re-partition-prune ```
``` 5f5acb1 ```
``` address comment ```
``` 1b0825b ```
Contributor Author

### tiancaiamao commented Jan 16, 2019

 PTAL @winoros
Contributor Author

### zz-jason approved these changes Jan 17, 2019

``` Merge branch 'master' into re-partition-prune ```
``` 89e597f ```
Contributor Author

### tiancaiamao commented Jan 17, 2019

 /run-all-tests

### tiancaiamao merged commit `b339c02` into pingcap:master Jan 17, 2019 13 checks passed

#### 13 checks passed

ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
idc-jenkins-ci-tidb/build Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/check_dev Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/common-test job succeeded
Details
idc-jenkins-ci-tidb/integration-common-test Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/integration-compatibility-test Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/integration-ddl-test Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/mybatis-test job succeeded
Details
idc-jenkins-ci-tidb/sqllogic-test-1 Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/sqllogic-test-2 Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/unit-test Jenkins job succeeded.
Details