executor: fix global index scan, index join, and query filtering reorg partition for global index #41992

L-maple · 2023-03-07T11:25:14Z

What problem does this PR solve?

Issue Number: close #41991
Issue Number: close #42065

Problem Summary:

What is changed and how it works?

fix global index scan when specify partition
fix global index join
fix drop/truncate partition reorg for global index

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

ti-chi-bot · 2023-03-07T11:25:16Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Defined2014
mjonss

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

ti-chi-bot · 2023-03-07T11:25:24Z

Hi @L-maple. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

L-maple · 2023-03-07T11:30:36Z

/assign @mjonss

L-maple · 2023-03-09T09:26:31Z

/cc @Defined2014

L-maple · 2023-03-09T09:27:03Z

/cc @mjonss

Defined2014 · 2023-03-09T09:35:12Z

Does the PR ready to be reviewed? Maybe you could remove [WIP] in title.

L-maple · 2023-03-09T09:48:17Z

WIP

Does the PR ready to be reviewed? Maybe you could remove [WIP] in title.

@Defined2014 Hi, could you run /ok-to-test?
I will contribute a littile more code for drop partition in this PR, which also has bug.

L-maple · 2023-04-07T11:42:30Z

@L-maple please update/merge with upstream (pingcap/master) and resolve the ddl/db_partition_test.go conflict.

@mjonss Hi, I resolved the conflict. But later I find building failed, and I cannot understand what it means and what I should do.

executor/distsql.go

mjonss · 2023-04-12T17:33:30Z

@L-maple please update/merge with upstream (pingcap/master) and resolve the ddl/db_partition_test.go conflict.

@mjonss Hi, I resolved the conflict. But later I find building failed, and I cannot understand what it means and what I should do.

Current CI failure seems to be:

ERROR: /home/jenkins/agent/workspace/pingcap/tidb/ghpr_build/tidb/executor/BUILD.bazel:3:11: GoCompilePkg executor/executor.a failed: (Exit 1): builder failed: error executing command (from target //executor:executor) bazel-out/k8-opt-exec-2B5CBBC6/bin/external/go_sdk/builder_reset/builder compilepkg -sdk external/go_sdk -installsuffix linux_amd64 -src executor/adapter.go -src executor/admin.go -src ... (remaining 521 arguments skipped)



Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging

compilepkg: nogo: errors found by nogo during build-time code analysis:

executor/distsql.go:344:16: ineffectual assignment to err (ineffassign)

Basically this.

mjonss

LGTM

L-maple · 2023-04-18T07:29:10Z

@Defined2014 please help me review this PR, thanks~

Defined2014 · 2023-04-19T05:47:17Z

executor/builder.go

+		tmp, ok := b.is.TableByID(ret.table.Meta().ID)
+		if !ok {
+			b.err = err
+			return nil
+		}
+		tbl, ok := tmp.(table.PartitionedTable)
+		if !ok {
+			b.err = exeerrors.ErrBuildExecutor
+			return nil
+		}


maybe could remove those codes and take L3786 - L3787 to L3766

L3766 is actually more clean. But @mjonss told me I should check explicitly before. You can talk about it and I will follow your common advice.

I handle the err (ErrTableNotExists) and still check the type transform explicitly here. @Defined2014

Defined2014 · 2023-04-19T05:51:37Z

executor/builder.go

+		if !ok {
+			return nil, err
+		}


Forgot set err here. By the way, I think no need check ok here, just like L4612.

Also please fix other similar codes.

Yes👍🏻，You are right. But I find other places in this method also not set err. When the builder.err should be set?

I think you could change it to return nil, infoschema.ErrTableNotExists here.

The err without setting maybe always equal to nil.

planner/core/find_best_task.go

Defined2014 · 2023-04-23T02:27:35Z

/merge

ti-chi-bot · 2023-04-23T02:27:39Z

This pull request has been accepted and is ready to merge.

Commit hash: bafae2a

ti-chi-bot · 2023-04-23T03:09:06Z

In response to a cherrypick label: new pull request created to branch release-7.1: #43314.

L-maple requested a review from a team as a code owner March 7, 2023 11:25

ti-chi-bot added release-note-none do-not-merge/needs-triage-completed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 7, 2023

ti-chi-bot added the needs-ok-to-test label Mar 7, 2023

L-maple force-pushed the feature/enhance_ddl_for_global_index branch from e4e4aaa to bd5461a Compare March 7, 2023 11:26

ti-chi-bot assigned mjonss Mar 7, 2023

L-maple force-pushed the feature/enhance_ddl_for_global_index branch from bd5461a to 4bac118 Compare March 7, 2023 12:03

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 7, 2023

L-maple changed the title ~~executor: fix global index scan when specify partition~~ [WIP] executor: fix global index scan, index join, and partition reorg for global index Mar 7, 2023

ti-chi-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. and removed do-not-merge/needs-triage-completed labels Mar 7, 2023

L-maple force-pushed the feature/enhance_ddl_for_global_index branch from 4bac118 to fec2992 Compare March 9, 2023 09:24

ti-chi-bot requested a review from Defined2014 March 9, 2023 09:26

ti-chi-bot requested a review from mjonss March 9, 2023 09:27

L-maple force-pushed the feature/enhance_ddl_for_global_index branch from fec2992 to 7de3f70 Compare March 9, 2023 09:32

Defined2014 added ok-to-test and removed needs-ok-to-test labels Mar 9, 2023

L-maple force-pushed the feature/enhance_ddl_for_global_index branch 4 times, most recently from a2f055a to c56ccc9 Compare March 13, 2023 06:21

L-maple force-pushed the feature/enhance_ddl_for_global_index branch from 1cecff9 to 0fa36f5 Compare April 7, 2023 11:11

L-maple requested review from hawkingrei and mjonss April 7, 2023 11:13

L-maple force-pushed the feature/enhance_ddl_for_global_index branch from 0fa36f5 to 3bfdcf3 Compare April 7, 2023 11:30

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 7, 2023

mjonss reviewed Apr 12, 2023

View reviewed changes

executor/distsql.go Show resolved Hide resolved

L-maple force-pushed the feature/enhance_ddl_for_global_index branch from 3bfdcf3 to 78a5bce Compare April 13, 2023 11:46

L-maple requested a review from mjonss April 13, 2023 11:46

L-maple force-pushed the feature/enhance_ddl_for_global_index branch from 78a5bce to 7d28afc Compare April 13, 2023 12:06

L-maple mentioned this pull request Apr 13, 2023

ddl: fix truncate partition for global index #42433

Merged

12 tasks

mjonss approved these changes Apr 13, 2023

View reviewed changes

Defined2014 removed the needs-cherry-pick-release-7.0 label Apr 19, 2023

Defined2014 reviewed Apr 19, 2023

View reviewed changes

fix: fix global index scan, index join, and dml in drop partition

bafae2a

L-maple force-pushed the feature/enhance_ddl_for_global_index branch from 7d28afc to bafae2a Compare April 23, 2023 02:24

ti-chi-bot bot added the needs-cherry-pick-release-7.1 label Apr 23, 2023

L-maple requested a review from Defined2014 April 23, 2023 02:25

Defined2014 approved these changes Apr 23, 2023

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 23, 2023

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 23, 2023

ti-chi-bot bot deleted a comment from ti-chi-bot Apr 23, 2023

ti-chi-bot bot merged commit 9825089 into pingcap:master Apr 23, 2023
8 checks passed

ti-chi-bot mentioned this pull request Apr 23, 2023

executor: fix global index scan, index join, and query filtering reorg partition for global index (#41992) #43314

Closed

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

executor: fix global index scan, index join, and query filtering reorg partition for global index #41992

executor: fix global index scan, index join, and query filtering reorg partition for global index #41992

L-maple commented Mar 7, 2023 •

edited

ti-chi-bot commented Mar 7, 2023 •

edited by ti-chi-bot bot

ti-chi-bot commented Mar 7, 2023

L-maple commented Mar 7, 2023

L-maple commented Mar 9, 2023

L-maple commented Mar 9, 2023

Defined2014 commented Mar 9, 2023

L-maple commented Mar 9, 2023

L-maple commented Apr 7, 2023

mjonss commented Apr 12, 2023

mjonss left a comment

L-maple commented Apr 18, 2023

Defined2014 Apr 19, 2023

L-maple Apr 19, 2023 •

edited

L-maple Apr 23, 2023

Defined2014 Apr 19, 2023

L-maple Apr 19, 2023

Defined2014 Apr 21, 2023

Defined2014 Apr 21, 2023 •

edited

L-maple Apr 23, 2023

Defined2014 commented Apr 23, 2023

ti-chi-bot bot commented Apr 23, 2023

ti-chi-bot commented Apr 23, 2023

executor: fix global index scan, index join, and query filtering reorg partition for global index #41992

executor: fix global index scan, index join, and query filtering reorg partition for global index #41992

Conversation

L-maple commented Mar 7, 2023 • edited

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Mar 7, 2023 • edited by ti-chi-bot bot

ti-chi-bot commented Mar 7, 2023

L-maple commented Mar 7, 2023

L-maple commented Mar 9, 2023

L-maple commented Mar 9, 2023

Defined2014 commented Mar 9, 2023

L-maple commented Mar 9, 2023

L-maple commented Apr 7, 2023

mjonss commented Apr 12, 2023

mjonss left a comment

Choose a reason for hiding this comment

L-maple commented Apr 18, 2023

Defined2014 Apr 19, 2023

Choose a reason for hiding this comment

L-maple Apr 19, 2023 • edited

Choose a reason for hiding this comment

L-maple Apr 23, 2023

Choose a reason for hiding this comment

Defined2014 Apr 19, 2023

Choose a reason for hiding this comment

L-maple Apr 19, 2023

Choose a reason for hiding this comment

Defined2014 Apr 21, 2023

Choose a reason for hiding this comment

Defined2014 Apr 21, 2023 • edited

Choose a reason for hiding this comment

L-maple Apr 23, 2023

Choose a reason for hiding this comment

Defined2014 commented Apr 23, 2023

ti-chi-bot bot commented Apr 23, 2023

ti-chi-bot commented Apr 23, 2023

L-maple commented Mar 7, 2023 •

edited

ti-chi-bot commented Mar 7, 2023 •

edited by ti-chi-bot bot

L-maple Apr 19, 2023 •

edited

Defined2014 Apr 21, 2023 •

edited