planner: add limit control for intersection case of index merge reader #46862

AilinKid · 2023-09-11T10:05:01Z

What problem does this PR solve?

Issue Number: close #46863
Problem Summary:

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

planner: add limit control for intersection case of index merge reader

Signed-off-by: AilinKid <314806019@qq.com>

tiprow · 2023-09-11T10:05:24Z

Hi @AilinKid. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

codecov · 2023-09-11T10:16:23Z

Codecov Report

Merging #46862 (d0d83f9) into master (aac330b) will decrease coverage by 0.7125%.
Report is 40 commits behind head on master.
The diff coverage is 74.3750%.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #46862        +/-   ##
================================================
- Coverage   73.3596%   72.6471%   -0.7125%     
================================================
  Files          1324       1352        +28     
  Lines        396657     403772      +7115     
================================================
+ Hits         290986     293329      +2343     
- Misses        87110      91779      +4669     
- Partials      18561      18664       +103

Flag	Coverage Δ
integration	`27.8054% <8.7500%> (?)`
unit	`73.2624% <74.3750%> (-0.0972%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`54.0444% <ø> (ø)`
parser	`84.9911% <ø> (+0.0302%)`	⬆️
br	`48.5483% <ø> (-4.1076%)`	⬇️

Signed-off-by: AilinKid <314806019@qq.com>

planner/core/task.go

qw4990

LGTM for the optimizer part

qw4990 · 2023-09-12T03:30:15Z

planner/core/plan_cost_ver2.go

+	// give a bias to pushDown limit, since it will get the same cost with NON_PUSH_DOWN_LIMIT case via expect count.
+	// push down limit case may reduce cop request consumption if any in some cases.
+	if p.PushedLimit != nil {
+		p.planCostVer2 = mulCostVer2(p.planCostVer2, 0.99)


We'd better implement this rule in physical optimization like pushLimitOrTopNForcibly instead of in the cost model. However the current IndexMerge implementation prevents us from doing this. Can we add a comment here to notify us to remove this tricky cost formula once we can implement this rule in the physical optimization?

make sense, definitely

executor/index_merge_reader.go

Signed-off-by: AilinKid <314806019@qq.com>

wshwsh12 · 2023-09-12T07:24:49Z

/review default

ti-chi-bot · 2023-09-12T07:25:09Z

@wshwsh12:

Potential Problems

Missing integration tests for the new changes.

Suggestions

Add integration tests to cover the new features and changes.

Review

PR Title

The PR title is clear and indicates what the change is about.

PR Description

The PR description is detailed and explains the problem and the solution well. It also provides a link to the related issue and follows the PR Title Format guidelines. The checklist for tests, side effects, documentation, and release notes is also properly filled out.

Code Changes

The code changes seem reasonable and address the problem as described in the issue. There are no obvious issues in the code changes.

However, there is a missing feature fetchLoopIntersectionWithOrderBy that might need to be implemented in the future as mentioned in the TODO comment. Keep track of this missing feature for future improvements.

Test Cases

Unit tests are included in the PR, which is good. However, there are no integration tests included. It would be better to add integration tests to ensure the new changes work as expected in the real environment.

In response to this:

/review default

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Defined2014 · 2023-09-12T07:30:52Z

executor/index_merge_reader.go

+			if w.pushedLimit != nil {
+				if w.pushedLimit.Count == 0 {
+					// close limitDone channel to notify intersectionProcessWorkers * N to exit.
+					close(w.limitDone)
+					return
+				}
+				rows := uint64(len(task.handles))
+				// The number of handles is less than the offset, discard all handles.
+				if rows <= w.pushedLimit.Offset {
+					w.pushedLimit.Offset -= rows
+					// wait next intersection result task
+					continue
+				}
+				task.handles = task.handles[w.pushedLimit.Offset:]
+				w.pushedLimit.Offset = 0
+
+				rows = uint64(len(task.handles))
+				// The number of handles is greater than the limit, only keep limit count.
+				if rows > w.pushedLimit.Count {
+					task.handles = task.handles[:w.pushedLimit.Count]
+				}
+				w.pushedLimit.Count -= mathutil.Min(w.pushedLimit.Count, rows)
+			}


This part could be a single function, and also could be used in fetchLoopUnion.

make sense, done

Signed-off-by: AilinKid <314806019@qq.com>

AilinKid · 2023-09-13T03:23:54Z

/test unit-test

tiprow · 2023-09-13T03:24:27Z

@AilinKid: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: AilinKid <314806019@qq.com>

Defined2014 · 2023-09-13T06:52:44Z

executor/test/indexmergereadtest/index_merge_reader_test.go

+	}
+	tk.MustExec("analyze table t")
+	tk.MustExec("insert into t values " + strings.Join(valsInsert, ","))
+	for i := 0; i < 100; i++ {


Maybe 10 is enough, ci is too slow. And better reduce row count to 500.

Signed-off-by: AilinKid <314806019@qq.com>

ti-chi-bot · 2023-09-13T07:24:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Defined2014, qw4990

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Defined2014,qw4990]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2023-09-13T07:24:42Z

[LGTM Timeline notifier]

Timeline:

2023-09-13 07:00:57.913524084 +0000 UTC m=+68823.881112120: ☑️ agreed by Defined2014.
2023-09-13 07:24:40.985455712 +0000 UTC m=+70246.953043744: ☑️ agreed by qw4990.

tiprow · 2023-09-13T07:56:02Z

@AilinKid: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
tiprow_fast_test	`d0d83f9`	link	true	`/test tiprow_fast_test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

AilinKid · 2023-09-13T08:20:58Z

/test tiprow_fast_test

ti-chi-bot · 2023-09-13T08:21:00Z

@AilinKid: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build
/test canary-scan-security
/test check-dev
/test check-dev2
/test mysql-test
/test pingcap/tidb/canary_ghpr_unit_test
/test pull-br-integration-test
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-ddl-test
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-prisma-test
/test pull-mysql-connector-test
/test pull-sqllogic-test
/test pull-tiflash-test
/test unit-test

The following commands are available to trigger optional jobs:

/test pull-notify-when-compatibility-sections-changed

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test

In response to this:

/test tiprow_fast_test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tiprow · 2023-09-13T08:21:31Z

@AilinKid: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test tiprow_fast_test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

AilinKid · 2023-09-13T08:21:44Z

/ok-to-test

AilinKid · 2023-09-13T08:22:00Z

/test tiprow_fast_test

ti-chi-bot · 2023-09-13T08:22:03Z

@AilinKid: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build
/test canary-scan-security
/test check-dev
/test check-dev2
/test mysql-test
/test pingcap/tidb/canary_ghpr_unit_test
/test pull-br-integration-test
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-ddl-test
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-prisma-test
/test pull-mysql-connector-test
/test pull-sqllogic-test
/test pull-tiflash-test
/test unit-test

The following commands are available to trigger optional jobs:

/test pull-notify-when-compatibility-sections-changed

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test

In response to this:

/test tiprow_fast_test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pingcap#46862) close pingcap#46863

add limit control for intersection case of index merge reader

b2a5a96

Signed-off-by: AilinKid <314806019@qq.com>

ti-chi-bot bot added do-not-merge/needs-linked-issue release-note size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 11, 2023

ti-chi-bot bot removed the do-not-merge/needs-linked-issue label Sep 11, 2023

add task switch

a7ae95b

Signed-off-by: AilinKid <314806019@qq.com>

qw4990 reviewed Sep 12, 2023

View reviewed changes

planner/core/task.go Outdated Show resolved Hide resolved

qw4990 reviewed Sep 12, 2023

View reviewed changes

Defined2014 reviewed Sep 12, 2023

View reviewed changes

executor/index_merge_reader.go Show resolved Hide resolved

address hangjie's comment

510819f

Signed-off-by: AilinKid <314806019@qq.com>

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 12, 2023

Delete docs/design/2023-09-12-grouping-sets.md

7cd5aea

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 12, 2023

Defined2014 reviewed Sep 12, 2023

View reviewed changes

AilinKid added 2 commits September 12, 2023 18:01

address yuanjia's comment

efbbd97

Signed-off-by: AilinKid <314806019@qq.com>

address hangjie's comment

f03d93f

Signed-off-by: AilinKid <314806019@qq.com>

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 12, 2023

AilinKid added 3 commits September 13, 2023 11:33

make bazel

b47a838

Signed-off-by: AilinKid <314806019@qq.com>

.

1853509

Signed-off-by: AilinKid <314806019@qq.com>

.

c0d80b3

Signed-off-by: AilinKid <314806019@qq.com>

.

8b3c8fb

Signed-off-by: AilinKid <314806019@qq.com>

Defined2014 reviewed Sep 13, 2023

View reviewed changes

Defined2014 approved these changes Sep 13, 2023

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm labels Sep 13, 2023

reduce ci burden

d0d83f9

Signed-off-by: AilinKid <314806019@qq.com>

qw4990 approved these changes Sep 13, 2023

View reviewed changes

ti-chi-bot bot added the lgtm label Sep 13, 2023

ti-chi-bot bot removed the needs-1-more-lgtm label Sep 13, 2023

ti-chi-bot bot added the ok-to-test label Sep 13, 2023

ti-chi-bot bot merged commit 3ad4f74 into pingcap:master Sep 13, 2023
6 of 16 checks passed

yibin87 pushed a commit to yibin87/tidb that referenced this pull request Oct 31, 2023

planner: add limit control for intersection case of index merge reader (

a55fee1

pingcap#46862) close pingcap#46863

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

planner: add limit control for intersection case of index merge reader #46862

planner: add limit control for intersection case of index merge reader #46862

AilinKid commented Sep 11, 2023 •

edited

tiprow bot commented Sep 11, 2023

codecov bot commented Sep 11, 2023 •

edited

qw4990 left a comment

qw4990 Sep 12, 2023

AilinKid Sep 12, 2023

wshwsh12 commented Sep 12, 2023

ti-chi-bot bot commented Sep 12, 2023

Defined2014 Sep 12, 2023

AilinKid Sep 12, 2023

AilinKid commented Sep 13, 2023

tiprow bot commented Sep 13, 2023

Defined2014 Sep 13, 2023 •

edited

ti-chi-bot bot commented Sep 13, 2023

ti-chi-bot bot commented Sep 13, 2023

tiprow bot commented Sep 13, 2023

AilinKid commented Sep 13, 2023

ti-chi-bot bot commented Sep 13, 2023

tiprow bot commented Sep 13, 2023

AilinKid commented Sep 13, 2023

AilinKid commented Sep 13, 2023

ti-chi-bot bot commented Sep 13, 2023

planner: add limit control for intersection case of index merge reader #46862

planner: add limit control for intersection case of index merge reader #46862

Conversation

AilinKid commented Sep 11, 2023 • edited

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

tiprow bot commented Sep 11, 2023

codecov bot commented Sep 11, 2023 • edited

Codecov Report

qw4990 left a comment

Choose a reason for hiding this comment

qw4990 Sep 12, 2023

Choose a reason for hiding this comment

AilinKid Sep 12, 2023

Choose a reason for hiding this comment

wshwsh12 commented Sep 12, 2023

ti-chi-bot bot commented Sep 12, 2023

Potential Problems

Suggestions

Review

PR Title

PR Description

Code Changes

Test Cases

Defined2014 Sep 12, 2023

Choose a reason for hiding this comment

AilinKid Sep 12, 2023

Choose a reason for hiding this comment

AilinKid commented Sep 13, 2023

tiprow bot commented Sep 13, 2023

Defined2014 Sep 13, 2023 • edited

Choose a reason for hiding this comment

ti-chi-bot bot commented Sep 13, 2023

ti-chi-bot bot commented Sep 13, 2023

[LGTM Timeline notifier]

tiprow bot commented Sep 13, 2023

AilinKid commented Sep 13, 2023

ti-chi-bot bot commented Sep 13, 2023

tiprow bot commented Sep 13, 2023

AilinKid commented Sep 13, 2023

AilinKid commented Sep 13, 2023

ti-chi-bot bot commented Sep 13, 2023

AilinKid commented Sep 11, 2023 •

edited

codecov bot commented Sep 11, 2023 •

edited

Defined2014 Sep 13, 2023 •

edited