Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docdb] Batch yb_backup.py's yb-admin commands to improve backup performance #11564

Open
hulien22 opened this issue Feb 22, 2022 · 1 comment
Assignees
Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue
Projects

Comments

@hulien22
Copy link
Contributor

hulien22 commented Feb 22, 2022

Jira Link: DB-1151
We currently perform this step during a restore (from issue #8892):

 12 min: PHASE 2: Generate list of TSes for every tablet
 19 min: PHASE 3: Find all table/tablet data dirs on all TSes

Phase 2 currently involves sending a yb-admin list_tablet_servers for each tablet individually (29d2c2c added some parallelization to this, but this can be further improved by batching all these requests into a singular yb-admin request).

Additionally, we currently need to perform a second request to all tservers for each table to find their tablet directories for Phase 3.

Ideally, we should combine these two phases to one, with a single batched yb-admin command that can retrieve the tablet servers for the list of tablets, and also return the list of corresponding data directories. This requires a new endpoint for yb-admin, and adding new parsing support for yb_backup.py.

@hulien22 hulien22 added the area/docdb YugabyteDB core features label Feb 22, 2022
@hulien22 hulien22 self-assigned this Feb 22, 2022
@hulien22 hulien22 added this to To do in Backups via automation Feb 22, 2022
@hulien22
Copy link
Contributor Author

One other thing to consider is that the backup script needs to determine if the db has this capability, and if so decide to use this new yb-admin command, else use the old approach

@yugabyte-ci yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue labels Jun 8, 2022
@yugabyte-ci yugabyte-ci added kind/enhancement This is an enhancement of an existing feature and removed kind/bug This issue is a bug labels Jul 30, 2022
@yugabyte-ci yugabyte-ci changed the title [docdb] Batch yb_backup.py's yb-admin commands [docdb] Batch yb_backup.py's yb-admin commands to improve backup performance Aug 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue
Projects
Backups
  
To do
Development

No branches or pull requests

2 participants