Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sorter(ticdc): split large txn #7889

Merged
merged 5 commits into from
Dec 14, 2022
Merged

sorter(ticdc): split large txn #7889

merged 5 commits into from
Dec 14, 2022

Conversation

overvenus
Copy link
Member

@overvenus overvenus commented Dec 12, 2022 •
edited
Loading

What problem does this PR solve?

Issue Number: close #7913

close pingcap/tidb#39819

What is changed and how it works?

Split large txn during sorter read.

Check List

Tests

  • Unit test
  • Integration test

Questions

Will it cause performance regression or break compatibility?

Yes, it might cause performance regression.

Do you need to update user documentation, design documentation or monitoring documentation?

No.

Release note 

Fix an OOM issue when TiCDC replicates big transactions

@overvenus
sorter(ticdc): split large txn
167da25 
Signed-off-by: Neil Shen <overvenus@gmail.com>
@overvenus overvenus added type/bugfix This PR fixes a bug. area/ticdc Issues or PRs related to TiCDC. labels Dec 12, 2022
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Dec 12, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • asddongmen
  • sdojjy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-linked-issue do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. labels Dec 12, 2022
@asddongmen
Copy link
Contributor

/run-integration-test

@overvenus
Copy link
Member Author

/run-all-tests

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 13, 2022
@overvenus
add more tests
127a01d 
Signed-off-by: Neil Shen <overvenus@gmail.com>
@ti-chi-bot ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 13, 2022
@overvenus
make fmt
c493252 
Signed-off-by: Neil Shen <overvenus@gmail.com>
@overvenus overvenus added the needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. label Dec 14, 2022
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 14, 2022
@overvenus overvenus marked this pull request as ready for review December 14, 2022 07:44
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 14, 2022
@overvenus
Copy link
Member Author

/run-all-tests

@overvenus
fix compile
69d89c4 
Signed-off-by: Neil Shen <overvenus@gmail.com>
@overvenus
Copy link
Member Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 69d89c4

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 14, 2022
@overvenus
Copy link
Member Author

/run-all-tests pd=release-6.5

@overvenus
Copy link
Member Author

/run-integration-test pd=release-6.5

@overvenus
Copy link
Member Author

/run-check-issue-triage-complete

@ti-chi-bot ti-chi-bot merged commit 917b2f8 into pingcap:master Dec 14, 2022
@ti-chi-bot ti-chi-bot mentioned this pull request Dec 14, 2022
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.1: #7914.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Dec 14, 2022
@overvenus @ti-chi-bot
This is an automated cherry-pick of pingcap#7889
47f5a00 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Dec 14, 2022
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #7915.

ti-chi-bot added a commit that referenced this pull request Dec 15, 2022
@ti-chi-bot
sorter(ticdc): split large txn (#7889) (#7915)
cf14c6d 
close #7913
overvenus added a commit to ti-chi-bot/tiflow that referenced this pull request Dec 21, 2022
@overvenus
sorter(ticdc): split large txn (pingcap#7889)
e07c360 
close pingcap#7913
ti-chi-bot added a commit that referenced this pull request Dec 21, 2022
@ti-chi-bot
sorter(ticdc): split large txn (#7889) (#7914)
861a092 
close #7913
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sdojjy sdojjy sdojjy approved these changes

@3AceShowHand 3AceShowHand 3AceShowHand approved these changes

@asddongmen asddongmen asddongmen approved these changes

Assignees
No one assigned
Labels
area/ticdc Issues or PRs related to TiCDC. needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/bugfix This PR fixes a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

DB sorter does not split large txn and may cause OOM after flashback, ticdc checkpoint don't move forward
5 participants
@overvenus @ti-chi-bot @asddongmen @sdojjy @3AceShowHand