Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

br restore very slow (like it hangs) when merging files #50613

Closed
Tracked by #50701
fubinzh opened this issue Jan 21, 2024 · 3 comments · Fixed by #50648
Closed
Tracked by #50701

br restore very slow (like it hangs) when merging files #50613

fubinzh opened this issue Jan 21, 2024 · 3 comments · Fixed by #50648
Labels
affects-6.5 affects-7.1 affects-7.5 component/br This issue is related to BR of TiDB. severity/major type/bug This issue is a bug.

Comments

@fubinzh
Copy link

fubinzh commented Jan 21, 2024

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. Run br restore full to restore a 300 TB cluster backup to a cluster with 100 TiKV nodes.

2. What did you expect to see? (Required)

Restore should succeeds.

3. What did you see instead (Required)

Restore is very slow, it looks like it hangs when merging files. br restore doesn't enter "split and scatter phase" after ~40m of restore.

4. What is your TiDB version? (Required)

v7.6.0
@fubinzh fubinzh added the type/bug This issue is a bug. label Jan 21, 2024
@fubinzh
Copy link
Author

fubinzh commented Jan 21, 2024

goroutine.1.txt
goroutine.2.txt

@fubinzh
Copy link
Author

fubinzh commented Jan 21, 2024

/component br

@ti-chi-bot ti-chi-bot bot added the component/br This issue is related to BR of TiDB. label Jan 21, 2024
@Leavrth Leavrth mentioned this issue Jan 23, 2024
Merged
13 tasks
@Leavrth Leavrth mentioned this issue Jan 29, 2024
Open
26 tasks
@3pointer
Copy link
Contributor

To workaround. we need skip merge files during restore

use --merge-region-key-count=1 to make merge skipped.

ti-chi-bot bot pushed a commit that referenced this issue Feb 2, 2024
@Leavrth
br: merge range in lightweight when snapshot restore (#50648)
7cdab19 
close #50613
This was referenced Feb 4, 2024
Merged
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Feb 4, 2024
@ti-chi-bot
br: merge range in lightweight when snapshot restore (#50648) (#50946)
a81c032 
close #50613
ti-chi-bot bot pushed a commit that referenced this issue Feb 27, 2024
@ti-chi-bot
br: merge range in lightweight when snapshot restore (#50648) (#50945)
e7782b3 
close #50613
ti-chi-bot bot pushed a commit that referenced this issue Mar 28, 2024
@ti-chi-bot
br: merge range in lightweight when snapshot restore (#50648) (#50944)
efd2fab 
close #50613
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.5 affects-7.1 affects-7.5 component/br This issue is related to BR of TiDB. severity/major type/bug This issue is a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

br: merge range in lightweight when snapshot restore Leavrth/tidb
4 participants
@jebter @3pointer @fubinzh @Leavrth