Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PiTR resolver may cause TiKV OOM #15414

Closed
Tracked by #16375
overvenus opened this issue Aug 23, 2023 · 4 comments · Fixed by #16008
Closed
Tracked by #16375

PiTR resolver may cause TiKV OOM #15414

overvenus opened this issue Aug 23, 2023 · 4 comments · Fixed by #16008
Assignees
@overvenus @YuJuncen
Labels
affects-6.5 affects-7.1 affects-7.5 component/backup-restore Component: backup, import, external_storage severity/major type/bug Type: Issue - Confirmed a bug

Comments

@overvenus
Copy link
Member

Bug Report

Similar to #14864, PiTR (backup-stream) has its own resolver, and it may cause OOM too.

What version of TiKV are you using?

> v6.5.0

Steps to reproduce

Run a large txn that is much greater than TiKV memory size.

What did you expect?

No OOM.

What did happened?

OOM.
@overvenus overvenus added the type/bug Type: Issue - Confirmed a bug label Aug 23, 2023
@tonyxuqqi
Copy link
Contributor

Please help to take care of it all together. Thanks. @overvenus

@BornChanger
Copy link
Contributor

Per @overvenus 's suggestion, I assign this issue to @YuJuncen

@BornChanger
Copy link
Contributor

/assign @YuJuncen

@BornChanger
Copy link
Contributor

/component backup-restore

@ti-chi-bot ti-chi-bot bot added the component/backup-restore Component: backup, import, external_storage label Sep 5, 2023
@YuJuncen YuJuncen mentioned this issue Dec 1, 2023
Merged
@overvenus overvenus mentioned this issue Jan 15, 2024
Open
15 tasks
ti-chi-bot bot added a commit that referenced this issue Jan 21, 2024
@YuJuncen @overvenus @ti-chi-bot
log_backup: stop task while memory out of quota (#16008)
6630125 
close #15414

This PR have refactored the subscription manager.
Generally, this:
- Replace the instance itself with a handle. This make it a real reactor(with an real event loop).
- Handle the result of subscripting a region via the message system instead of asynchronously, this will be the basis of making subscription tracker thread safe and (someday, hopefully) merge the basic libraries with TiCDC.

Based on the changes above, this PR also allows a region to be temporarily deregistered while we are about to reach the memory quota.

Signed-off-by: Yu Juncen <yu745514916@live.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Jan 21, 2024
@YuJuncen @ti-chi-bot
This is an automated cherry-pick of tikv#16008
5679c8f 
close tikv#15414

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Jan 21, 2024
Closed
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Jan 21, 2024
@YuJuncen @ti-chi-bot
This is an automated cherry-pick of tikv#16008
97b62bf 
close tikv#15414

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Jan 21, 2024
Open
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Jan 21, 2024
@YuJuncen @ti-chi-bot
This is an automated cherry-pick of tikv#16008
29aeb0a 
close tikv#15414

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Jan 21, 2024
Open
YuJuncen added a commit to ti-chi-bot/tikv that referenced this issue Mar 4, 2024
@YuJuncen
This is an automated cherry-pick of tikv#16008
668b143 
close tikv#15414

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
YuJuncen added a commit to ti-chi-bot/tikv that referenced this issue Mar 4, 2024
@YuJuncen
This is an automated cherry-pick of tikv#16008
7c1bc70 
close tikv#15414

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: hillium <yujuncen@pingcap.com>
dbsid pushed a commit to dbsid/tikv that referenced this issue Mar 24, 2024
@YuJuncen @overvenus
@ti-chi-bot @dbsid
log_backup: stop task while memory out of quota (tikv#16008)
2678ea6 
close tikv#15414

This PR have refactored the subscription manager.
Generally, this:
- Replace the instance itself with a handle. This make it a real reactor(with an real event loop).
- Handle the result of subscripting a region via the message system instead of asynchronously, this will be the basis of making subscription tracker thread safe and (someday, hopefully) merge the basic libraries with TiCDC.

Based on the changes above, this PR also allows a region to be temporarily deregistered while we are about to reach the memory quota.

Signed-off-by: Yu Juncen <yu745514916@live.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: dbsid <chenhuansheng@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@overvenus overvenus

@YuJuncen YuJuncen

Labels
affects-6.5 affects-7.1 affects-7.5 component/backup-restore Component: backup, import, external_storage severity/major type/bug Type: Issue - Confirmed a bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

log_backup: stop task while memory out of quota YuJuncen/tikv
6 participants
@overvenus @jebter @tonyxuqqi @YuJuncen @ti-chi-bot @BornChanger