New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

expression: constraint propagate for '>' and monotonous function #8640

Merged
merged 12 commits into from Jan 4, 2019

Conversation

@tiancaiamao
Copy link
Contributor

tiancaiamao commented Dec 10, 2018

What problem does this PR solve?

Propagate some thing like:

TO_DAYS(col) < TO_DAYS('2001-01-01') && col > '2009-04-03'  => false

What is changed and how it works?

Add a propagate rule ruleColumnGTConst leverage this PR #7643

Some simple rules:

	col > c1, col > c2, c1 > c2 => col > c1
        col > c1, col < c2, c1 >= c2 => false

A more complicate rule:

	col > c1, f(col) < c2, f is monotonous, f(c1) <= c2 => false

Prove:

        col > c1, f is monotonous => f(col) > f(c1)
        f(col) > f(c1), f(col) < c2, f(c1) >= c2 => false

Check List

Tests

  • Unit test

Related changes

Ref #7516


This change is Reviewable

@tiancaiamao tiancaiamao force-pushed the tiancaiamao:propagate-pruning branch from dcb25d5 to e11f0ed Dec 11, 2018

@tiancaiamao tiancaiamao removed the status/WIP label Dec 11, 2018

tiancaiamao added some commits Dec 11, 2018

@tiancaiamao tiancaiamao added this to In Progress in Table Partitioning via automation Dec 11, 2018

@tiancaiamao

This comment has been minimized.

Copy link
Contributor

tiancaiamao commented Dec 11, 2018

Show resolved Hide resolved expression/constraint_propagation.go Outdated
Show resolved Hide resolved expression/constraint_propagation.go Outdated
Show resolved Hide resolved expression/constraint_propagation.go Outdated
Show resolved Hide resolved expression/constraint_propagation.go Outdated
Show resolved Hide resolved expression/constraint_propagation.go
@tiancaiamao

This comment has been minimized.

Copy link
Contributor

tiancaiamao commented Dec 29, 2018

PTAL @winoros

@tiancaiamao

This comment has been minimized.

Copy link
Contributor

tiancaiamao commented Dec 30, 2018

PTAL @winoros

@tiancaiamao

This comment has been minimized.

Copy link
Contributor

tiancaiamao commented Jan 2, 2019

Tiny update. I remove this rule, it's not useful for partition pruning and adds code complexity

col > c1, col > c2, c1 > c2 => col > c1

PTAL @winoros

@winoros
Copy link
Member

winoros left a comment

lgtm

@winoros winoros added the status/LGT1 label Jan 2, 2019

@tiancaiamao

This comment has been minimized.

Copy link
Contributor

tiancaiamao commented Jan 2, 2019

@zz-jason zz-jason requested review from eurekaka and zz-jason Jan 2, 2019

@tiancaiamao

This comment has been minimized.

Copy link
Contributor

tiancaiamao commented Jan 3, 2019

con1, ok = f1.GetArgs()[1].(*Constant)
if !ok {
return
}

This comment has been minimized.

@eurekaka

eurekaka Jan 3, 2019

Contributor

We don't consider const op col?

This comment has been minimized.

@tiancaiamao

tiancaiamao Jan 4, 2019

Contributor

There are only two rules that really useful to partition pruning:

  1. col < const
  2. col >= const

Why?

create table ... partition by col 
p1 values less than const1
p2 values less than const2
...

Let's take a look at p2, the filter expressions are:

col >= const1 and col < const2

The final conditions would be:

col >= const1 and col < const2 and query

So the problem become that whether we can propagate:

col >= const1 and query => false
col < const2 and query => false

ruleColumnOPConst means use col op const as rule, perfectly cover the above two.


You'll find the PR can't handle the const op col.
It doesn't matter, we can add another rule that transform
const op col to col op const

Then you'll find the rule order matters.

Not a serious problem, we can control the order by
newConstraintSolver(ruleTransformConstOPColumn, ruleColumnOPConst)

I'll make pruning for select ... where to_days(c) > '2018-01-04' work, and improve some cases later.

This comment has been minimized.

@eurekaka

eurekaka Jan 4, 2019

Contributor

I'll make pruning for select ... where to_days(c) > '2018-01-04' work, and improve some cases later.

Thant would be great.

@@ -32,12 +32,16 @@ var _ = Suite(&testExpressionSuite{})
type testExpressionSuite struct{}

func newColumn(id int) *Column {

This comment has been minimized.

@zz-jason

zz-jason Jan 4, 2019

Member

how about:
s/newColumn/newInt64Column/
s/newColumnWithType/newColumn/

This comment has been minimized.

@tiancaiamao

tiancaiamao Jan 4, 2019

Contributor

There are so many newColumn in this file https://github.com/pingcap/tidb/blob/3cf386b5ca2300a0e0cebeae64d58eac9fcbff6c/expression/constant_test.go
If the function signature changes, it bring about ~50 (lines) irrelevant changes to this PR.

And this is _test.go, in this file newColumn indicates newInt64Column

@zz-jason zz-jason requested a review from lamxTyler Jan 4, 2019

@eurekaka
Copy link
Contributor

eurekaka left a comment

LGTM

@eurekaka eurekaka added status/LGT2 and removed status/LGT1 labels Jan 4, 2019

@eurekaka

This comment has been minimized.

Copy link
Contributor

eurekaka commented Jan 4, 2019

/run-all-tests

@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Jan 4, 2019

Codecov Report

❗️ No coverage uploaded for pull request base (master@e0b30fe). Click here to learn what that means.
The diff coverage is 62.22%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #8640   +/-   ##
=========================================
  Coverage          ?   67.52%           
=========================================
  Files             ?      363           
  Lines             ?    75073           
  Branches          ?        0           
=========================================
  Hits              ?    50693           
  Misses            ?    19905           
  Partials          ?     4475
Impacted Files Coverage Δ
expression/constraint_propagation.go 74.67% <62.22%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e0b30fe...c2e6940. Read the comment docs.

@tiancaiamao tiancaiamao merged commit 3708a83 into pingcap:master Jan 4, 2019

4 checks passed

ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
idc-jenkins-ci-tidb/build Jenkins job succeeded.
Details
license/cla Contributor License Agreement is signed.
Details

Table Partitioning automation moved this from In Progress to Done Jan 4, 2019

@tiancaiamao tiancaiamao deleted the tiancaiamao:propagate-pruning branch Jan 4, 2019

zimulala added a commit that referenced this pull request Jan 4, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment