executor: support IndexMergeReaderExecutor #12305

hailanwhu · 2019-09-23T05:27:41Z

What problem does this PR solve?

In #11245 PR, we generate the PhysicalPlan for IndexMerge. In this PR, we add the executor and execution framework for IndexMerge.

What is changed and how it works?

Add new executor, IndexMergeReaderExecutor
Add execution framework of IndexMergeReaderExecutor, most of the code is borrowed from IndexLookUp and TableReader. However, it is difficult to refactor.

Check List

Tests

No code
Test will be added after IndexMerge hint added.

Code changes

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Release note

-new feature.

codecov · 2019-09-23T05:34:37Z

Codecov Report

❗ No coverage uploaded for pull request base (master@933715f). Click here to learn what that means.
The diff coverage is 69.9422%.

@@             Coverage Diff             @@
##             master     #12305   +/-   ##
===========================================
  Coverage          ?   80.1134%           
===========================================
  Files             ?        484           
  Lines             ?     121992           
  Branches          ?          0           
===========================================
  Hits              ?      97732           
  Misses            ?      16458           
  Partials          ?       7802

planner/core/resolve_indices.go

executor/distsql.go

hailanwhu · 2019-09-24T13:31:07Z

@zz-jason I have fixed the comments. However, the unit-test failed and the log is [2019-09-24T13:25:49.372Z] gzip: stdin: unexpected end of file . I cannot get any useful information from it to fix this error.

sre-bot · 2019-09-26T08:19:59Z

@zz-jason, @SunRunAway, @XuHuaiyu, PTAL.

sre-bot · 2019-09-28T08:20:00Z

@zz-jason, @SunRunAway, @XuHuaiyu, PTAL.

sre-bot · 2019-10-01T08:19:35Z

@zz-jason, @SunRunAway, @XuHuaiyu, PTAL.

executor/builder.go

sre-bot · 2019-10-10T03:51:01Z

@zz-jason, @XuHuaiyu, @SunRunAway, PTAL.

sre-bot · 2019-10-12T03:51:01Z

@zz-jason, @XuHuaiyu, @SunRunAway, PTAL.

zyxbest · 2019-10-14T05:45:54Z

/run-unit-test

sre-bot · 2019-10-15T03:51:01Z

@zz-jason, @XuHuaiyu, @SunRunAway, PTAL.

hailanwhu · 2019-10-20T21:40:07Z

@XuHuaiyu I add the comments and force executing using IndexMerge path.

hailanwhu · 2019-12-01T22:44:56Z

@XuHuaiyu I meet some problems. I am not sure why it fails using testSuit1.

hailanwhu · 2019-12-04T06:09:39Z

@eurekaka I do not have a clear idea why using the testSuit1 failed.

eurekaka · 2019-12-04T06:18:35Z

I manually run the test SQL sequence in MySQL client, the case fails also. Looks like there are bugs in code logic.

hailanwhu · 2019-12-08T13:58:27Z

@eurekaka last time, the manual test failed because of the setting of startTS. I updated the code. However, the test still failed.

hailanwhu · 2019-12-16T14:06:35Z

@eurekaka all pass. Maybe the feedback is not handled well. Because every partial scan could have a feedback, it is not sure how to deal with them.

planner/core/physical_plans.go

executor/index_merge_reader.go

XuHuaiyu · 2019-10-31T07:09:40Z

executor/index_merge_reader.go

+	for i := 0; i < len(e.partialPlans); i++ {
+		_, ok := e.partialPlans[i][0].(*plannercore.PhysicalIndexScan)
+		if ok {
+			kvRanges, err := distsql.IndexRangesToKVRanges(e.ctx.GetSessionVars().StmtCtx, getPhysicalTableID(e.table), e.indexes[i].ID, e.ranges[i], e.feedback)


s/ kvRanges/ ranges

XuHuaiyu · 2019-10-31T07:09:50Z

executor/index_merge_reader.go

+
+// Open implements the Executor Open interface
+func (e *IndexMergeReaderExecutor) Open(ctx context.Context) error {
+	kvRangess := make([][]kv.KeyRange, 0, len(e.partialPlans))


s/ kvRangess/ kvRanges

Do I need to rename the variables?

Yes, it's a typo.

executor/builder.go

executor/index_merge_reader.go

sre-bot · 2019-12-25T12:24:37Z

@zz-jason, @XuHuaiyu, @qw4990, @SunRunAway, PTAL.

sre-bot · 2019-12-28T05:22:37Z

@zz-jason, @XuHuaiyu, @qw4990, @SunRunAway, PTAL.

qw4990

Should we check the enableIndexMerge before generating an IndexMerge plan at find_best_task.go:423？

XuHuaiyu · 2019-12-30T06:41:01Z

/run-all-tests

XuHuaiyu · 2019-12-30T12:20:09Z

/run-all-tests

XuHuaiyu · 2019-12-31T01:53:14Z

LGTM

qw4990 · 2019-12-31T07:13:22Z

LGTM

sre-bot · 2019-12-31T07:16:11Z

/run-all-tests

sre-bot · 2019-12-31T07:19:11Z

@hailanwhu merge failed.

XuHuaiyu · 2019-12-31T07:22:03Z

/run-unit-test

sre-bot added the contribution This PR is from a community contributor. label Sep 23, 2019

francis0407 added sig/execution SIG execution status/PTAL labels Sep 23, 2019

francis0407 requested review from SunRunAway, zz-jason and XuHuaiyu September 23, 2019 05:53

zz-jason reviewed Sep 24, 2019

View reviewed changes

planner/core/resolve_indices.go Outdated Show resolved Hide resolved

planner/core/resolve_indices.go Outdated Show resolved Hide resolved

executor/distsql.go Outdated Show resolved Hide resolved

XuHuaiyu reviewed Oct 8, 2019

View reviewed changes

executor/builder.go Outdated Show resolved Hide resolved

executor/builder.go Outdated Show resolved Hide resolved

hailanwhu requested a review from a team as a code owner December 4, 2019 05:59

ghost requested review from eurekaka and winoros and removed request for a team December 4, 2019 05:59

eurekaka removed their request for review December 4, 2019 06:07

XuHuaiyu reviewed Dec 18, 2019

View reviewed changes

executor/index_merge_reader.go Outdated Show resolved Hide resolved

executor/index_merge_reader.go Outdated Show resolved Hide resolved

executor/index_merge_reader.go Outdated Show resolved Hide resolved

executor/index_merge_reader.go Outdated Show resolved Hide resolved

qw4990 reviewed Dec 23, 2019

View reviewed changes

executor/index_merge_reader.go Show resolved Hide resolved

qw4990 reviewed Dec 23, 2019

View reviewed changes

executor/index_merge_reader.go Show resolved Hide resolved

qw4990 reviewed Dec 23, 2019

View reviewed changes

executor/index_merge_reader.go Show resolved Hide resolved

qw4990 reviewed Dec 23, 2019

View reviewed changes

executor/index_merge_reader.go Outdated Show resolved Hide resolved

qw4990 reviewed Dec 30, 2019

View reviewed changes

hailanwhu added 7 commits December 30, 2019 17:18

indexmerge executor

1ba97b3

fix dev

ea2356c

fix partial comments

3b49e7d

fix some comments

3c778bd

fix dev

5c8742f

fix some comments

744b990

add test for disabling indexmerge

0d623ce

qw4990 added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 31, 2019

qw4990 added status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. status/PTAL labels Dec 31, 2019

Merge branch 'master' into IndexMergeExecute

f89dca0

XuHuaiyu approved these changes Dec 31, 2019

View reviewed changes

XuHuaiyu changed the title ~~executor: add IndexMergeReaderExecutor and its execution framework~~ executor: support IndexMergeReaderExecutor Dec 31, 2019

XuHuaiyu merged commit ae106f2 into pingcap:master Dec 31, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

executor: support IndexMergeReaderExecutor #12305

executor: support IndexMergeReaderExecutor #12305

hailanwhu commented Sep 23, 2019

codecov bot commented Sep 23, 2019 •

edited

Loading

hailanwhu commented Sep 24, 2019

sre-bot commented Sep 26, 2019

sre-bot commented Sep 28, 2019

sre-bot commented Oct 1, 2019

sre-bot commented Oct 10, 2019

sre-bot commented Oct 12, 2019

zyxbest commented Oct 14, 2019

sre-bot commented Oct 15, 2019

hailanwhu commented Oct 20, 2019

hailanwhu commented Dec 1, 2019

hailanwhu commented Dec 4, 2019

eurekaka commented Dec 4, 2019

hailanwhu commented Dec 8, 2019

hailanwhu commented Dec 16, 2019

XuHuaiyu Oct 31, 2019

XuHuaiyu Oct 31, 2019

hailanwhu Dec 19, 2019

zz-jason Dec 29, 2019

sre-bot commented Dec 25, 2019

sre-bot commented Dec 28, 2019

qw4990 left a comment

XuHuaiyu commented Dec 30, 2019

XuHuaiyu commented Dec 30, 2019

XuHuaiyu commented Dec 31, 2019

qw4990 commented Dec 31, 2019

sre-bot commented Dec 31, 2019

sre-bot commented Dec 31, 2019

XuHuaiyu commented Dec 31, 2019

executor: support IndexMergeReaderExecutor #12305

executor: support IndexMergeReaderExecutor #12305

Conversation

hailanwhu commented Sep 23, 2019

What problem does this PR solve?

What is changed and how it works?

Check List

codecov bot commented Sep 23, 2019 • edited Loading

Codecov Report

hailanwhu commented Sep 24, 2019

sre-bot commented Sep 26, 2019

sre-bot commented Sep 28, 2019

sre-bot commented Oct 1, 2019

sre-bot commented Oct 10, 2019

sre-bot commented Oct 12, 2019

zyxbest commented Oct 14, 2019

sre-bot commented Oct 15, 2019

hailanwhu commented Oct 20, 2019

hailanwhu commented Dec 1, 2019

hailanwhu commented Dec 4, 2019

eurekaka commented Dec 4, 2019

hailanwhu commented Dec 8, 2019

hailanwhu commented Dec 16, 2019

XuHuaiyu Oct 31, 2019

Choose a reason for hiding this comment

XuHuaiyu Oct 31, 2019

Choose a reason for hiding this comment

hailanwhu Dec 19, 2019

Choose a reason for hiding this comment

zz-jason Dec 29, 2019

Choose a reason for hiding this comment

sre-bot commented Dec 25, 2019

sre-bot commented Dec 28, 2019

qw4990 left a comment

Choose a reason for hiding this comment

XuHuaiyu commented Dec 30, 2019

XuHuaiyu commented Dec 30, 2019

XuHuaiyu commented Dec 31, 2019

qw4990 commented Dec 31, 2019

sre-bot commented Dec 31, 2019

sre-bot commented Dec 31, 2019

XuHuaiyu commented Dec 31, 2019

codecov bot commented Sep 23, 2019 •

edited

Loading