Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

executor: reset kvRange in TableReaderExec.Close #19447

Merged
merged 3 commits into from Aug 26, 2020
Merged

executor: reset kvRange in TableReaderExec.Close #19447

merged 3 commits into from Aug 26, 2020

Conversation

ichn-hu
Copy link
Contributor

@ichn-hu ichn-hu commented Aug 25, 2020 •
edited

What problem does this PR solve?

Issue Number: close #19431 and close #19435

Problem Summary:

When union scan is under apply's inner side, it will be called multiple times, and for the memTableReader that reads the dirty table, it uses kvRanges from the base table scan as the range, while the base table scanner will update the kvRanges, it does not set it back when it is closed, therefore the kvRange array will grow, and duplicated rows will be read from the dirty table, resulted in errors.

What is changed and how it works?

What's Changed:

Add a back up kvRange variable, reset it upon the close of table reader.

Related changes

  • Need to cherry-pick to the release branch 4.0, 3.1 and 3.0

Check List

Tests

  • Unit test

Side effects

N/A

Release note

  • No release note.

@ichn-hu ichn-hu requested a review from a team as a code owner August 25, 2020 12:02
@ichn-hu ichn-hu requested review from wshwsh12 and removed request for a team August 25, 2020 12:02
@ichn-hu
Copy link
Contributor Author

ichn-hu commented Aug 25, 2020

@XuHuaiyu PTAL and please help assign reviewer.

@ichn-hu ichn-hu changed the title Fix kvRanges duplication for union scan when it is run mutiple times executor: Fix kvRanges duplication for union scan when it is run mutiple times Aug 25, 2020
@ichn-hu
Copy link
Contributor Author

ichn-hu commented Aug 25, 2020

/assign @ichn-hu

@github-actions github-actions bot added the sig/execution SIG execution label Aug 25, 2020
@ichn-hu ichn-hu changed the title executor: Fix kvRanges duplication for union scan when it is run mutiple times executor: fix kvRanges duplication for union scan when it is run mutiple times Aug 26, 2020
lzmhhh123
lzmhhh123 reviewed Aug 26, 2020
View reviewed changes
executor/table_reader.go Outdated
dagPB *tipb.DAGRequest
startTS uint64
kvRanges []kv.KeyRange
kvRangesBackup []kv.KeyRange
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A better way is moving the buildResp out of Open to Next.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A Backup makes me feel weird.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am feeling ackward as well, perhaps a better name like initialKvRanges would make it less weird, while I did not quite get it what you mean by moving the buildResp to Next, and I don't think it would solve the problem here. The problem is, whenever buildResp is called, new ranges will be added to kvRanges, even if it is put under Next, as long as it is not cleaned, the redundant ranges will be remained.

@lzmhhh123 lzmhhh123 mentioned this pull request Aug 26, 2020
Closed
@XuHuaiyu
Copy link
Contributor

LGTM

@ti-srebot ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 26, 2020
@XuHuaiyu XuHuaiyu changed the title executor: fix kvRanges duplication for union scan when it is run mutiple times executor: fix kvRanges duplication for union scan when it run mutiple times Aug 26, 2020
lzmhhh123
lzmhhh123 reviewed Aug 26, 2020
View reviewed changes
Copy link
Member

@lzmhhh123 lzmhhh123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-srebot ti-srebot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Aug 26, 2020
@lzmhhh123
Copy link
Member

/merge

@ti-srebot ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 26, 2020
@ti-srebot
Copy link
Contributor

/run-all-tests

@ti-srebot
Copy link
Contributor

@ichn-hu merge failed.

@XuHuaiyu
Copy link
Contributor

/run-common-test

@XuHuaiyu XuHuaiyu changed the title executor: fix kvRanges duplication for union scan when it run mutiple times executor: reset kvRange in TableReaderExec.Close Aug 26, 2020
@XuHuaiyu XuHuaiyu merged commit 2a05244 into pingcap:master Aug 26, 2020
@ti-srebot ti-srebot mentioned this pull request Aug 26, 2020
Merged
ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Aug 26, 2020
@ichn-hu @ti-srebot
cherry pick pingcap#19447 to release-3.0
e5e32a5 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot
Copy link
Contributor

cherry pick to release-3.0 in PR #19468

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Aug 26, 2020
@ichn-hu @ti-srebot
cherry pick pingcap#19447 to release-3.1
685833d 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot mentioned this pull request Aug 26, 2020
Merged
@ti-srebot
Copy link
Contributor

cherry pick to release-3.1 in PR #19469

winoros
winoros reviewed Aug 26, 2020
View reviewed changes
Copy link
Member

@winoros winoros left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ti-srebot ti-srebot added status/LGT3 The PR has already had 3 LGTM. and removed status/LGT2 Indicates that a PR has LGTM 2. labels Aug 26, 2020
ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Aug 26, 2020
@ichn-hu @ti-srebot
cherry pick pingcap#19447 to release-4.0
7498b42 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot mentioned this pull request Aug 26, 2020
Merged
@ti-srebot
Copy link
Contributor

cherry pick to release-4.0 in PR #19470

ti-srebot added a commit that referenced this pull request Aug 26, 2020
@ti-srebot
executor: reset kvRange in TableReaderExec.Close (#19447) (#19470)
9c1c212 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
ti-srebot added a commit that referenced this pull request Aug 27, 2020
@ti-srebot
executor: reset kvRange in TableReaderExec.Close (#19447) (#19469)
1fec254 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
ti-srebot added a commit that referenced this pull request Aug 27, 2020
@ti-srebot
executor: reset kvRange in TableReaderExec.Close (#19447) (#19468)
e510546 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@winoros winoros winoros left review comments

@lzmhhh123 lzmhhh123 lzmhhh123 left review comments

@ti-srebot ti-srebot ti-srebot approved these changes

@wshwsh12 wshwsh12 Awaiting requested review from wshwsh12

Assignees

@ichn-hu ichn-hu

Labels
sig/execution SIG execution status/can-merge Indicates a PR has been approved by a committer. status/LGT3 The PR has already had 3 LGTM. type/bug-fix This PR fixes a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

apply return wrong result in transaction Apply return non empty set in transaction
5 participants
@ichn-hu @XuHuaiyu @lzmhhh123 @ti-srebot @winoros