planner, executor: implement the null-aware antiSemiJoin and null-aware antiLeftOuterSemiJoin (hash join with inner build) #37512

AilinKid · 2022-08-31T09:02:50Z

Signed-off-by: AilinKid 314806019@qq.com

What problem does this PR solve?

Issue Number: close #37525

Problem Summary:

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

implement the null-aware antiSemiJoin and null-aware antiLeftOuterSemiJoin

ti-chi-bot · 2022-08-31T09:02:51Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

fixdb
windtalker

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

ti-chi-bot · 2022-08-31T09:08:41Z

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/9974aefb74ef8f5cc8106b960bd8d967a415eefd

fixdb · 2022-08-31T20:44:41Z

Here is the issue: #37525

AilinKid · 2022-09-01T07:14:36Z

/run-mysql-test tidb-test=pr/1953

AilinKid · 2022-09-02T03:24:33Z

/run-mysql-test tidb-test=pr/1953

AilinKid · 2022-09-02T04:55:42Z

/run-mysql-test tidb-test=pr/1953

AilinKid · 2022-09-02T05:08:49Z

/run-mysql-test tidb-test=pr/1953

AilinKid · 2022-09-02T05:37:41Z

/run-mysql-test tidb-test=pr/1953

winoros · 2022-09-05T10:15:21Z

planner/core/logical_plans.go

@@ -142,6 +142,7 @@ type LogicalJoin struct {
 	preferJoinOrder bool

 	EqualConditions []*expression.ScalarFunction
+	NAEQConditions  []*expression.ScalarFunction


Will it be better if we use a bool value to tell whether is null aware or not?

// Step2: when step1 is finished, then we can determine whether we need to extract NA-EQ from OtherCondition to NAEQConditions.
// when there are still no EqualConditions, let's try to be a NAAJ.

In where the comment is, we only fill the NAEQConditions when there are no common EqualConditions.

winoros · 2022-09-05T10:24:53Z

executor/joiner.go

@@ -575,12 +646,84 @@ func (j *leftOuterSemiJoiner) Clone() joiner {
 	return &leftOuterSemiJoiner{baseJoiner: j.baseJoiner.Clone()}
 }

+type nullAwareAntiLeftOuterSemiJoiner struct {
+	baseJoiner


Can we put the NAAJType as a member variable of the struct?

NAAJType is not a fixed joiner parameter, but the runtime status according to the left and right rows characteristic dynamically.

now it is only used by AntiLeftOuterSemiJoiner to fill the specified result according to runtime NAAJType.

On only the OnMisMatch(), we will fill it with 1. quite different like before, handling null in onMatch is a kind of acceleration of execution.

Looks like null-aware join is so different from the normal join, this pr introduce too many if isNA related code, even the top interface tryMatchInners/onMatch are polluted. I wonder that comparing to the current implementation, if it can be more clear by adding a new NAHashJoinExec instead? @XuHuaiyu what do you think?

Yeah, i'm also thinking whether we could split to two join exec that extract the comment part as an interface and baseHashJoinExec.

That seems reasonable, the code size will expand dramatically in this pr (some copy work is unavoidable even if extracting an interface for baseHashJoinExec), we can put the clear work in the next pr?

planner/core/exhaust_physical_plans.go

AilinKid · 2022-09-08T07:40:04Z

/run-mysql-test tidb-test=pr/1953

AilinKid · 2022-09-08T07:55:06Z

/run-mysql-test tidb-test=pr/1953

AilinKid · 2022-09-08T08:06:32Z

/run-mysql-test tidb-test=pr/1953

AilinKid · 2022-09-08T08:33:07Z

/run-mysql-test tidb-test=pr/1953

AilinKid · 2022-09-14T06:42:26Z

/run-mysql-test tidb-test=pr/1953

fixdb · 2022-09-14T06:52:03Z

planner/core/physical_plans.go

+	// IsNullEQ is used for cases like Except statement where null key should be matched with null key.
+	// <1,null> is exactly matched with <1,null>, where the null value should not be filtered and
+	// the null is exactly matched with null only. (while in NAAJ null value should also be matched
+	// with other non-null item as well)


I am confused, null value matched with non-null values?

yes, null in (y set), for lhs, it doesn't care what the y actually is.

additional, when match <1, null> probe row to hash table, what are you ganna fetch from hash-table? the pattern should be <1, whatever>

planner/core/physical_plans.go

planner/core/explain.go

fixdb · 2022-09-15T05:52:28Z

cmd/explaintest/t/naaj.test

+delete from naaj_B;
+insert into naaj_B values(2, 2, 2);
+select (a, b) not in (select a, b from naaj_B) from naaj_A;
+select * from naaj_A where (a, b) not in (select a, b from naaj_B);


Can we have cases where naaj_A has (null, null, null) tuple?

windtalker · 2022-09-15T07:46:59Z

cmd/explaintest/r/naaj.result

+1	NULL	1
+explain format = 'brief' select (a, b) not in (select a, b from naaj_B where naaj_A.c = naaj_B.c) from naaj_A;
+id	estRows	task	access object	operator info
+HashJoin	10000.00	root		anti left outer semi join, equal:[eq(test.naaj_a.c, test.naaj_b.c)], other cond:eq(test.naaj_a.a, test.naaj_b.a), eq(test.naaj_a.b, test.naaj_b.b)


Is the other condition eq(test.naaj_a.a, test.naaj_b.a), eq(test.naaj_a.b, test.naaj_b.b) a normal eq condition or need to be null aware?

yes, it does.

here is the old path, since it already has the correlated EQ condition "naaj_A.c = naaj_B.c", which could be used as a hash join key. So the na-eq condition should be put in the other condition as it was before. (mixture EQ cond and NA-EQ cond in hash join key is little tricky)

This can be future improvements. We can open another issue to track this optimization (EQ and NAEQ as the hash keys).

windtalker · 2022-09-15T07:52:14Z

cmd/explaintest/r/subquery.result

@@ -55,7 +55,7 @@ insert into exam values(1, 'math', 100);
 set names utf8 collate utf8_general_ci;
 explain format = 'brief' select * from stu where stu.name not in (select 'guo' from exam where exam.stu_id = stu.id);
 id	estRows	task	access object	operator info
-Apply	10000.00	root		CARTESIAN anti semi join, other cond:eq(test.stu.name, Column#8)
+Apply	10000.00	root		Null-aware anti semi join, equal:[eq(test.stu.name, Column#8)]


Just curious why this sql need using Apply?

interesting, collation block this, see issue #37032

in apply mode, even there is a na-eq here, it will be appended as other conds in running

windtalker · 2022-09-15T07:59:53Z

executor/builder.go

@@ -1972,7 +1993,10 @@ func (b *executorBuilder) buildApply(v *plannercore.PhysicalApply) Executor {
 	if b.err != nil {
 		return nil
 	}
-	otherConditions := append(expression.ScalarFuncs2Exprs(v.EqualConditions), v.OtherConditions...)
+	// test is in the explain/naaj.test#part5.
+	// although we prepared the NAEqualConditions, but for Apply mode, we still need move it to other conditions like eq condition did here.


So Apply does not need care about null/empty things?

actually, it does, it feels the null & empty of other conditions(na-eq condition is marked) in joiner logic. doesn't like hash join, you can do this at the matching phase before joiner detail works.

windtalker · 2022-09-15T08:25:35Z

planner/core/testdata/expression_rewriter_suite_out.json

@@ -22,7 +22,7 @@
      {
        "SQL": "select * from t1 where (t1.a, t1.b) not in (select a, b from t2)",
        "Plan": [
-          "HashJoin 3.20 root  CARTESIAN anti semi join, other cond:eq(test.t1.a, test.t2.a), eq(test.t1.b, test.t2.b)",
+          "HashJoin 3.20 root  Null-aware anti semi join, equal:[eq(test.t1.b, test.t2.b) eq(test.t1.a, test.t2.a)]",


why condition order is changed.

yes, some misorder before (>,<), but it doesn't affect the correctness by now.

windtalker · 2022-09-15T08:28:01Z

planner/core/stats.go

-	leftKeyNDV := getColsNDV(h.leftJoinKeys, h.leftSchema, h.leftProfile)
-	rightKeyNDV := getColsNDV(h.rightJoinKeys, h.rightSchema, h.rightProfile)
+	var leftKeyNDV, rightKeyNDV float64
+	if len(h.leftJoinKeys) > 0 {


If both leftJoinKyes and leftNAJoinKeys are non-empty, the ndv will be based only on the leftJoinKyes, is it a by-design behavior?

len(h.leftJoinKeys) > 0 || len(h.rightJoinKeys) > 0, I want this actually, I take it for granted that left keys are always same size of right keys. (join key and na-join key is mutually exclusive)

fixed

planner/core/logical_plans.go

windtalker · 2022-09-15T09:44:49Z

planner/core/exhaust_physical_plans.go

@@ -1942,6 +1950,8 @@ func (p *LogicalJoin) tryToGetMppHashJoin(prop *property.PhysicalProperty, useBC
 		return nil
 	}
 	lkeys, rkeys, _, _ := p.GetJoinKeys()
+	lNAkeys, rNAKeys := p.GetNAJoinKeys()


since mpp does not support na-keys, looks like we should return nil once na-keys is not empty?

windtalker · 2022-09-16T02:47:10Z

planner/core/exhaust_physical_plans.go

@@ -1809,6 +1809,11 @@ func (p *LogicalJoin) shouldUseMPPBCJ() bool {
 	return checkChildFitBC(p.children[0]) || checkChildFitBC(p.children[1])
 }

+// canPushToCop checks if it can be pushed to some stores.
+func (p *LogicalJoin) canPushToCop(storeTp kv.StoreType) bool {
+	return len(p.NAEQConditions) == 0 && p.baseLogicalPlan.canPushToCop(storeTp)


There will be some compatible issues here:
Before this pr, for sql like select * from t where nullable_col not in (select xx from t1), if the subquery meet the broadcast threshold, it can be pushed to TiFlash, but after this pr, there is no way to pushdown such join to TiFlash? I think before TiFlash's support of naaj is ready, it's better to add a switch to let user decide to use naaj or cross join

Since it will be controlled by the switch before tiflash supports the exec logic. Just forbid pushing down for now is okay?

yes, after the switch, the len(p.NAEQConditions) == 0 should exactly be 0 here

fixdb

+1

Signed-off-by: AilinKid <314806019@qq.com>

winoros · 2022-09-17T19:15:39Z

executor/hash_table.go

+				needCheckBuildRowPos = append(needCheckBuildRowPos, c.hCtx.naKeyColIdx[i])
+				needCheckProbeRowPos = append(needCheckProbeRowPos, probeHCtx.naKeyColIdx[i])
+			}
+			// check the idxs-th value of the join columns.


What does idxs here mean?

join column（a,b,c) (e,f,g)

find the non-null bit, for example, (b,c) - (e,f)
use the b,c's, and e,f's column index to fetch the value from the chunk to check the same.
the idxs (means b,c's column index)/(e,f's column index)

Looks like needCheckProbeRowPos and needCheckBuildRowPos can be generated outside the loop since it only related to probeKeyNullBits?

executor/hash_table.go

winoros · 2022-09-17T19:26:04Z

executor/joiner.go

@@ -575,12 +646,84 @@ func (j *leftOuterSemiJoiner) Clone() joiner {
 	return &leftOuterSemiJoiner{baseJoiner: j.baseJoiner.Clone()}
 }

+type nullAwareAntiLeftOuterSemiJoiner struct {
+	baseJoiner


Yeah, i'm also thinking whether we could split to two join exec that extract the comment part as an interface and baseHashJoinExec.

winoros · 2022-09-17T19:28:33Z

planner/core/exhaust_physical_plans.go

@@ -1809,6 +1809,11 @@ func (p *LogicalJoin) shouldUseMPPBCJ() bool {
 	return checkChildFitBC(p.children[0]) || checkChildFitBC(p.children[1])
 }

+// canPushToCop checks if it can be pushed to some stores.
+func (p *LogicalJoin) canPushToCop(storeTp kv.StoreType) bool {
+	return len(p.NAEQConditions) == 0 && p.baseLogicalPlan.canPushToCop(storeTp)


Since it will be controlled by the switch before tiflash supports the exec logic. Just forbid pushing down for now is okay?

Co-authored-by: Yiding Cui <winoros@gmail.com>

windtalker · 2022-09-19T07:01:10Z

executor/hash_table.go

+				needCheckBuildRowPos = append(needCheckBuildRowPos, c.hCtx.naKeyColIdx[i])
+				needCheckProbeRowPos = append(needCheckProbeRowPos, probeHCtx.naKeyColIdx[i])
+			}
+			// check the idxs-th value of the join columns.


Looks like needCheckProbeRowPos and needCheckBuildRowPos can be generated outside the loop since it only related to probeKeyNullBits?

windtalker · 2022-09-19T07:45:40Z

executor/hash_table.go

+	hashVals        []hash.Hash64
+	hasNull         []bool
+	naHasNull       []bool
+	naColNullBitMap []*bitmap.ConcurrentBitmap


why need a concurrentBitmap here?

yes, it's unnecessary here

If it's unnecessary, I think we should not use concurrentBitMap here, please use a normal bitmap instead.

windtalker · 2022-09-19T08:13:25Z

executor/join.go

-	hCtx.initHash(probeSideChk.NumRows())
+	numRows := probeSideChk.NumRows()
+	hCtx.initHash(numRows)
+	// By now, path 1 and 2 won't be conducted at the same time.


Then why not use if isNAAJ to make sure path 1 and path 2 could not be conducted at the same time?

cause we guarantee the EQcondition and NAEQcondition couldn't be filled at same time before.

AilinKid · 2022-09-19T09:02:25Z

Looks like needCheckProbeRowPos and needCheckBuildRowPos can be generated outside the loop since it only related to probeKeyNullBits?

Yes it could, for the same code style of GetNullBucketRows, we put it here, fixed

Signed-off-by: AilinKid <314806019@qq.com>

windtalker

LGTM

AilinKid · 2022-09-19T13:56:11Z

/merge

ti-chi-bot · 2022-09-19T13:56:14Z

This pull request has been accepted and is ready to merge.

Commit hash: 3f3f9b7

AilinKid · 2022-09-19T15:04:23Z

/merge

AilinKid · 2022-09-19T15:56:23Z

/merge

AilinKid · 2022-09-20T06:16:24Z

/run-cherry-picker

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2022-09-20T06:17:15Z

cherry pick to release-6.2 in PR #37980

sre-bot · 2022-09-20T10:53:20Z

TiDB MergeCI notify

🔴 Bad News! New failing [1] after this pr merged.
These new failed integration tests seem to be caused by the current PR, please try to fix these new failed integration tests, thanks!

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci-tidb/tics-test	🟥 failed 1, success 0, total 1	4 min 18 sec	New failing
idc-jenkins-ci-tidb/common-test	🔴 failed 1, success 10, total 11	52 min	Existing failure
idc-jenkins-ci-tidb/integration-ddl-test	🔴 failed 1, success 5, total 6	25 min	Existing failure
idc-jenkins-ci-tidb/integration-common-test	🔴 failed 2, success 15, total 17	12 min	Existing failure
idc-jenkins-ci/integration-cdc-test	🟢 all 37 tests passed	27 min	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	5 min 17 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	4 min 58 sec	Existing passed
idc-jenkins-ci-tidb/mybatis-test	🟢 all 1 tests passed	3 min 17 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	2 min 38 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed

ti-chi-bot added release-note size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 31, 2022

AilinKid requested review from windtalker, winoros and fixdb August 31, 2022 11:09

AilinKid changed the title ~~planner, executor: implement the null-aware antiSemiJoin and null-aware antiLeftOuterSemiJoin~~ planner, executor: implement the null-aware antiSemiJoin and null-aware antiLeftOuterSemiJoin (hash join with inner build) Aug 31, 2022

winoros reviewed Sep 5, 2022

View reviewed changes

fixdb reviewed Sep 14, 2022

View reviewed changes

planner/core/physical_plans.go Show resolved Hide resolved

fixdb reviewed Sep 14, 2022

View reviewed changes

planner/core/physical_plans.go Show resolved Hide resolved

fixdb reviewed Sep 15, 2022

View reviewed changes

windtalker reviewed Sep 16, 2022

View reviewed changes

fixdb approved these changes Sep 16, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 16, 2022

AilinKid requested a review from a team as a code owner September 17, 2022 05:31

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 17, 2022

add naaj switch

406facb

Signed-off-by: AilinKid <314806019@qq.com>

AilinKid force-pushed the NAAJ branch from c27042c to 406facb Compare September 17, 2022 06:58

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 17, 2022

winoros reviewed Sep 17, 2022

View reviewed changes

AilinKid and others added 2 commits September 18, 2022 09:26

Update executor/hash_table.go

0ac2d30

Co-authored-by: Yiding Cui <winoros@gmail.com>

Merge branch 'master' into NAAJ

6656184

windtalker reviewed Sep 19, 2022

View reviewed changes

AilinKid added 2 commits September 19, 2022 17:31

move the null bits collection out of loop

c93e419

Signed-off-by: AilinKid <314806019@qq.com>

add unsafeSet for concurrentBitmap to speed up

053aeb6

Signed-off-by: AilinKid <314806019@qq.com>

AilinKid force-pushed the NAAJ branch from 84689e1 to 053aeb6 Compare September 19, 2022 12:42

Merge branch 'master' into NAAJ

3f3f9b7

windtalker approved these changes Sep 19, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 19, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 19, 2022

Merge branch 'master' into NAAJ

b3f486e

jackysp approved these changes Sep 19, 2022

View reviewed changes

ti-chi-bot merged commit 0823fdb into pingcap:master Sep 19, 2022

AilinKid added the needs-cherry-pick-6.2 label Sep 20, 2022

ti-srebot mentioned this pull request Sep 20, 2022

planner, executor: implement the null-aware antiSemiJoin and null-aware antiLeftOuterSemiJoin (hash join with inner build) (#37512) #37980

Closed

12 tasks

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Sep 20, 2022

cherry pick pingcap#37512 to release-6.2

c0d4b29

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

planner, executor: implement the null-aware antiSemiJoin and null-aware antiLeftOuterSemiJoin (hash join with inner build) #37512

planner, executor: implement the null-aware antiSemiJoin and null-aware antiLeftOuterSemiJoin (hash join with inner build) #37512

Conversation

AilinKid commented Aug 31, 2022 • edited

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Aug 31, 2022 • edited

ti-chi-bot commented Aug 31, 2022 • edited

fixdb commented Aug 31, 2022

AilinKid commented Sep 1, 2022

AilinKid commented Sep 2, 2022

AilinKid commented Sep 2, 2022

AilinKid commented Sep 2, 2022

AilinKid commented Sep 2, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AilinKid Sep 8, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AilinKid Sep 8, 2022 • edited

Choose a reason for hiding this comment

windtalker Sep 17, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AilinKid commented Sep 8, 2022

AilinKid commented Sep 8, 2022

AilinKid commented Sep 8, 2022

AilinKid commented Sep 8, 2022

AilinKid commented Sep 14, 2022

Choose a reason for hiding this comment

AilinKid Sep 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AilinKid Sep 17, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AilinKid Sep 17, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AilinKid Sep 17, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AilinKid Sep 17, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AilinKid Sep 17, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AilinKid Sep 18, 2022 • edited

Choose a reason for hiding this comment

fixdb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AilinKid Sep 18, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AilinKid commented Sep 19, 2022 • edited

windtalker left a comment

Choose a reason for hiding this comment

AilinKid commented Sep 19, 2022

AilinKid commented Aug 31, 2022 •

edited

ti-chi-bot commented Aug 31, 2022 •

edited

ti-chi-bot commented Aug 31, 2022 •

edited

AilinKid Sep 8, 2022 •

edited

AilinKid Sep 8, 2022 •

edited

windtalker Sep 17, 2022 •

edited

AilinKid Sep 14, 2022 •

edited

AilinKid Sep 17, 2022 •

edited

AilinKid Sep 17, 2022 •

edited

AilinKid Sep 17, 2022 •

edited

AilinKid Sep 17, 2022 •

edited

AilinKid Sep 17, 2022 •

edited

AilinKid Sep 18, 2022 •

edited

AilinKid Sep 18, 2022 •

edited

AilinKid commented Sep 19, 2022 •

edited