Geo-Partitioned Backups go to a Single Object Storage Bucket #8732
Assignees
Labels
area/ysql
Yugabyte SQL (YSQL)
kind/enhancement
This is an enhancement of an existing feature
priority/medium
Medium priority issue
Projects
Comments
OlegLoginov
added
area/ysql
|
Short YB design doc: https://docs.google.com/document/d/1K0d7FNKFVRNdsMjoCnYajtu4OAKeBRBLHnjbnd42IKc |
OlegLoginov
added a commit
that referenced
this issue
Jan 12, 2022
…al Object Storage Buckets Summary: Support geo-partitioned backups. This patch allows clients to specify a list of --region and --region-location pairs to allow backing up tablets in a location specific to a region. List of changes: - `ysql_dumpall` has `--dump-single-database` mode (as `ysql_dump` uses it) to dump a single DB only. - `yb_backup.py` has new arguments: `--use_tablespaces` to store `CREATE TABLESPACE` statements into new metadata file YSQLDump_tablespaces. By default the argument is `False`. - `yb_backup.py` uses `ysql_dumpall` tool to get `CREATE TABLESPACE` statements (with `--use_tablespaces` only). - `yb_backup.py` has new arguments: `--region` and `--region_location` to get a set of pairs: region -> location in the backup-create mode. - `yb_backup.py` now creates third metadata file `Manifest` in the backup folder, which has backup properties including tablet locations (for all tablets). - `yb_backup.py` output now includes JSON with all backup locations (before it included only one location). Note: TS region is loaded via `<ts_ip>:9000/varz` API - value of `--placement_region` TS G-flag. If TS region is unknown - default target location will be used (from `--backup_location` argument). Single region (old) format is also supported in backup-create & backup-restore. Test Plan: ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioning ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioningNoRegions ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioningOneRegion ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioningRestoringIntoExisting ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioningWithTablespaces ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioningNoRegionsWithTablespaces ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioningOneRegionWithTablespaces ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioningRestoringIntoExistingWithTablespaces PG testing (for ysql_dump): ybd --java-test org.yb.pgsql.TestPgRegressMisc Generic backup tests: ybd --cxx-test tools_yb-backup-test_ent ybd --java-test org.yb.pgsql.TestYbBackup --tp 1 ybd --java-test org.yb.cql.TestYbBackup --tp 1 ybd --java-test org.yb.cql.ParameterizedTestYbBackup --tp 1 Reviewers: mihnea, vpatibandla, vkumar, alex, dsrinivasan Reviewed By: alex, dsrinivasan Subscribers: alex, smishra, jenkins-bot, yql Differential Revision: https://phabricator.dev.yugabyte.com/D13741
OlegLoginov
added a commit
that referenced
this issue
Feb 4, 2022
…ckups to Regional Object Storage Buckets Summary: Support geo-partitioned backups. This patch allows clients to specify a list of --region and --region-location pairs to allow backing up tablets in a location specific to a region. List of changes: - `ysql_dumpall` has `--dump-single-database` mode (as `ysql_dump` uses it) to dump a single DB only. - `yb_backup.py` has new arguments: `--use_tablespaces` to store `CREATE TABLESPACE` statements into new metadata file YSQLDump_tablespaces. By default the argument is `False`. - `yb_backup.py` uses `ysql_dumpall` tool to get `CREATE TABLESPACE` statements (with `--use_tablespaces` only). - `yb_backup.py` has new arguments: `--region` and `--region_location` to get a set of pairs: region -> location in the backup-create mode. - `yb_backup.py` now creates third metadata file `Manifest` in the backup folder, which has backup properties including tablet locations (for all tablets). - `yb_backup.py` output now includes JSON with all backup locations (before it included only one location). Note: TS region is loaded via `<ts_ip>:9000/varz` API - value of `--placement_region` TS G-flag. If TS region is unknown - default target location will be used (from `--backup_location` argument). Single region (old) format is also supported in backup-create & backup-restore. Original commit: https://phabricator.dev.yugabyte.com/D13741 Original diff: b0ced11 Test Plan: ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioning ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioningNoRegions ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioningOneRegion ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioningRestoringIntoExisting ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioningWithTablespaces ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioningNoRegionsWithTablespaces ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioningOneRegionWithTablespaces ybd --java-test org.yb.pgsql.TestYbBackup#testGeoPartitioningRestoringIntoExistingWithTablespaces PG testing (for ysql_dump): ybd --java-test org.yb.pgsql.TestPgRegressMisc Generic backup tests: ybd --cxx-test tools_yb-backup-test_ent ybd --java-test org.yb.pgsql.TestYbBackup --tp 1 ybd --java-test org.yb.cql.TestYbBackup --tp 1 ybd --java-test org.yb.cql.ParameterizedTestYbBackup --tp 1 Reviewers: mihnea, vpatibandla, vkumar, alex, achauhan, dsrinivasan Reviewed By: dsrinivasan Subscribers: jenkins-bot, yql Differential Revision: https://phabricator.dev.yugabyte.com/D15177
|
Commit: b0ced11 |
yugabyte-ci
added
kind/enhancement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-605
Currently when we do a backup while using geo-partitioning everything goes to a single object storage bucket. There is no way to allow for geo-partitioned data to go to an object storage bucket within the zone/region the data is pinned to. Could cause data residency issues.
The text was updated successfully, but these errors were encountered: