New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: derive `col is not null` from `col op col` conditions #8603

Merged
merged 3 commits into from Jan 30, 2019

Conversation

@eurekaka
Copy link
Contributor

eurekaka commented Dec 6, 2018

What problem does this PR solve?

Fix #8587

What is changed and how it works?

  • Check whether the col op col condition is null rejected, if yes, derive col is not null for both sides of inner join, and for inner side of outer join.
  • Remove duplicate conditions before pushing them down. Note that we check if the condition contains mutable functions before de-duplicating.
  • Check NotNull flag of child column before generating is not null expression to avoid unnecessary conditions.
  • Handle anti join specially.

Check List

Tests

  • Unit test

Code changes

  • Has exported function/method change

Side effects

  • Increased code complexity

Related changes

N/A


This change is Reviewable

@eurekaka

This comment has been minimized.

Copy link
Contributor Author

eurekaka commented Dec 11, 2018

/run-all-tests

@eurekaka eurekaka force-pushed the eurekaka:derive_not_null branch from 848d607 to 4d76bb4 Dec 11, 2018

@eurekaka

This comment has been minimized.

Copy link
Contributor Author

eurekaka commented Dec 11, 2018

/run-all-tests

@eurekaka

This comment has been minimized.

Copy link
Contributor Author

eurekaka commented Dec 12, 2018

/run-unit-test

@eurekaka

This comment has been minimized.

Copy link
Contributor Author

eurekaka commented Dec 18, 2018

/rebuid

@eurekaka

This comment has been minimized.

Copy link
Contributor Author

eurekaka commented Dec 18, 2018

/run-all-tests

@eurekaka eurekaka removed the status/WIP label Dec 19, 2018

@eurekaka

This comment has been minimized.

Copy link
Contributor Author

eurekaka commented Dec 19, 2018

randgen test failures can be fixed by #8725, wait it to be merged first.

@eurekaka eurekaka force-pushed the eurekaka:derive_not_null branch from f5e9164 to afb8bb0 Dec 19, 2018

@XuHuaiyu XuHuaiyu changed the title plan: derive `col is not null` from `col op col` conditions *: derive `col is not null` from `col op col` conditions Dec 19, 2018

@eurekaka

This comment has been minimized.

Copy link
Contributor Author

eurekaka commented Dec 19, 2018

/run-all-tests

@eurekaka

This comment has been minimized.

Copy link
Contributor Author

eurekaka commented Dec 19, 2018

/run-mybatis-test

@eurekaka

This comment has been minimized.

Copy link
Contributor Author

eurekaka commented Dec 19, 2018

@eurekaka

This comment has been minimized.

Copy link
Contributor Author

eurekaka commented Dec 20, 2018

@lamxTyler comments addressed, PTAL

Show resolved Hide resolved expression/util.go Outdated
Show resolved Hide resolved expression/util.go Outdated
@eurekaka

This comment has been minimized.

Copy link
Contributor Author

eurekaka commented Dec 21, 2018

@lamxTyler comments addressed, PTAL

@lamxTyler
Copy link
Member

lamxTyler left a comment

LGTM

@eurekaka eurekaka force-pushed the eurekaka:derive_not_null branch 2 times, most recently from 88075a6 to d61a4ef Dec 28, 2018

@eurekaka

This comment has been minimized.

Copy link
Contributor Author

eurekaka commented Dec 29, 2018

@zz-jason PTAL. I have updated the PR to not derive is not null from column equal condition of LeftOuterSemiJoin either, for example, for queries like select a, a in (select a from t t2 where t2.b > t1.b) from t t1, no t2.a is not null is generated, but we would still generate t2.b is not null, which looks correct?

mysql> explain select a, a in (select a from t t2 where t2.b > t1.b) from t t1;
+--------------------------+-------+------+-----------------------------------------------------------------------------------------------+
| id                       | count | task | operator info                                                                                 |
+--------------------------+-------+------+-----------------------------------------------------------------------------------------------+
| Projection_7             | 4.00  | root | t1.a, 6_aux_0                                                                                 |
| └─HashLeftJoin_8         | 4.00  | root | left outer semi join, inner:TableReader_13, equal:[eq(t1.a, t2.a)], other cond:gt(t2.b, t1.b) |
|   ├─TableReader_10       | 4.00  | root | data:TableScan_9                                                                              |
|   │ └─TableScan_9        | 4.00  | cop  | table:t1, range:[-inf,+inf], keep order:false, stats:pseudo                                   |
|   └─TableReader_13       | 4.00  | root | data:Selection_12                                                                             |
|     └─Selection_12       | 4.00  | cop  | not(isnull(t2.b))                                                                             |
|       └─TableScan_11     | 4.00  | cop  | table:t2, range:[-inf,+inf], keep order:false, stats:pseudo                                   |
+--------------------------+-------+------+-----------------------------------------------------------------------------------------------+
Show resolved Hide resolved executor/join_test.go
Show resolved Hide resolved expression/constant_propagation.go

@zz-jason zz-jason requested review from winoros and zz-jason Jan 2, 2019

plan: derive `col is not null` from `col op col` conditions
Check whether the `op` condition is null rejected, if yes, derive
`col is not null` for both sides of inner join, and for inner side
of outer join.

Also, remove duplicate conditions before pushing them down. Check
if the condition is mutable before removing duplicates.

For AntiSemiJoin, AntiLeftOuterSemiJoin and LeftOuterSemiJoin, do
not generate new `is not null` conditions.

@eurekaka eurekaka force-pushed the eurekaka:derive_not_null branch from 38d570f to 3e864de Jan 16, 2019

@eurekaka

This comment has been minimized.

Copy link
Contributor Author

eurekaka commented Jan 16, 2019

/run-all-tests

@eurekaka

This comment has been minimized.

Copy link
Contributor Author

eurekaka commented Jan 16, 2019

/run-integration-ddl-test
/run-unit-test

@eurekaka eurekaka requested review from zz-jason and winoros and removed request for winoros Jan 16, 2019

@zz-jason
Copy link
Member

zz-jason left a comment

LGTM

@zz-jason zz-jason added status/LGT2 and removed status/LGT1 labels Jan 29, 2019

@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Jan 29, 2019

Codecov Report

Merging #8603 into master will increase coverage by <.01%.
The diff coverage is 86.45%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #8603      +/-   ##
==========================================
+ Coverage    67.2%   67.21%   +<.01%     
==========================================
  Files         371      371              
  Lines       76985    77050      +65     
==========================================
+ Hits        51741    51791      +50     
- Misses      20624    20631       +7     
- Partials     4620     4628       +8
Impacted Files Coverage Δ
expression/builtin.go 84% <ø> (ø) ⬆️
planner/core/logical_plans.go 73.72% <100%> (ø) ⬆️
planner/core/expression_rewriter.go 73.61% <100%> (-0.17%) ⬇️
planner/core/logical_plan_builder.go 74.86% <100%> (+0.17%) ⬆️
expression/constant_propagation.go 95.56% <100%> (+0.74%) ⬆️
expression/util.go 72% <77.77%> (+0.48%) ⬆️
planner/core/rule_predicate_push_down.go 89.16% <79.41%> (-1.48%) ⬇️
util/systimemon/systime_mon.go 80% <0%> (-20%) ⬇️
ddl/delete_range.go 75.13% <0%> (-4.24%) ⬇️
executor/merge_join.go 80.45% <0%> (-1.15%) ⬇️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 20c1b95...b04bd72. Read the comment docs.

@eurekaka eurekaka merged commit 09a5c80 into pingcap:master Jan 30, 2019

5 checks passed

ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
idc-jenkins-ci-tidb/build Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/check_dev Jenkins job succeeded.
Details
license/cla Contributor License Agreement is signed.
Details

@eurekaka eurekaka deleted the eurekaka:derive_not_null branch Jan 30, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment