Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

backupccl: 8tb online restore link phase is about 4x slower than expected #124346

Open
msbutler opened this issue May 17, 2024 · 3 comments
Open

backupccl: 8tb online restore link phase is about 4x slower than expected #124346

msbutler opened this issue May 17, 2024 · 3 comments
Assignees
@dt
Labels
A-disaster-recovery C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-disaster-recovery
Projects
Disaster Recovery Backlog

Comments

@msbutler
Copy link
Collaborator

msbutler commented May 17, 2024 •
edited by cockroach-jira-scripts

Last time we ran a working 8tb online restore, the link phase took about 15 minutes. If you pull the latest master binary from master and run an 8tb tpcc online restore (fixture in cockroach-fixtures-us-east1/backups/tpc-c/v24.1/db/warehouses=150k), the link phase takes over an hour. several cpu profiles suggest we're computing mvcc stats, which should not be happening. Further investigation is required to get to the bottom of this:
image

Jira issue: CRDB-38840
@msbutler msbutler added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-disaster-recovery labels May 17, 2024
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented May 17, 2024

cc @cockroachdb/disaster-recovery

@blathers-crl blathers-crl bot added this to Backlog in Disaster Recovery Backlog May 17, 2024
@msbutler
Copy link
Collaborator Author

note that this fixture was created on top of #124156

@msbutler
Copy link
Collaborator Author

even after setting "kv.range_merge.skip_external_bytes.enabled" and disabling the consistency checker, the link phase is still very slow and it's because we're materializing external ssts during snapshots for some reason.
image
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dt dt

Labels
A-disaster-recovery C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-disaster-recovery
Projects
Disaster Recovery Backlog
Backlog
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dt @msbutler