[DocDB] Packed + B/R + Colocation: Restore should restore all the schema packings of snapshot #14861
Assignees
Labels
Projects
Comments
sanketkedia
added
area/docdb
yugabyte-ci
added
kind/bug
yugabyte-ci
added
priority/high
druzac
added a commit
that referenced
this issue
Mar 20, 2023
…the schema packings of snapshot Summary: With packed row enabled, in the case of an out-of-cluster backup/restore, the table id in the snapshot will be different from the id created in the restored cluster for the same table. So we need a way to find the correct table entry to merge schema packings properly. Assuming the snapshot folder is downloaded correctly, then this pairing is only needed for a colocated tablet (i.e., mapping colocated tables). Since each colocated table entry from the snapshot and its restored counterpart have the same colocation id, this diff uses this colocation id to find the corresponding restored table id thereby pairing the table entry. Test Plan: ``` ybd --cxx-test yb-backup-cross-feature-test --gtest_filter YBBackupTestWithPackedRows.YSQLSchemaPackingWithColocatedDB ./yb_build.sh --java-test 'org.yb.pgsql.TestYsqlPartitionedBackup#testColocatedPartitionedTable' ``` I also ran the manual test case in [[ #15966 | GH15966 ]] to verify that bug is fixed by this diff. I also ran a modified version of the test case to verify we can successfully restore backups made prior to [[ 8acc2a3 | 8acc2a3 ]]: ``` % git checkout 60f81b0 % ybd % ./bin/yb-ctl destroy && ./bin/yb-ctl create && ./bin/yb-ctl start % ysqlsh yugabyte=# CREATE DATABASE qqq colocated = true; yugabyte=# \c qqq # create partitioned table: qqq=# CREATE TABLE htest(k1 int, k2 text, k3 int, v1 int, v2 text) PARTITION BY HASH (k1); # Creating table htest, SQL statement: qqq=# CREATE TABLE htest_1 PARTITION OF htest(k1, k2, k3, v1, v2) FOR VALUES WITH (modulus 2, remainder 0); # Creating table htest, SQL statement: qqq=# CREATE TABLE htest_2 PARTITION OF htest(k1, k2, k3, v1, v2) FOR VALUES WITH (modulus 2, remainder 1); qqq=# CREATE INDEX ON htest_1(k1, k2) INCLUDE (k3, v1); qqq=# INSERT INTO htest VALUES (1, 2, 3, 4, 5); ^D % ./managed/devops/bin/yb_backup.py --mac --masters 127.0.0.1 --remote_yb_admin_binary ./build/latest/bin/yb-admin --remote_ysql_dump_binary ./build/latest/postgres/bin/ysql_dump --remote_ysql_shell_binary ./build/latest/postgres/bin/ysqlsh --no_ssh --storage_type nfs --nfs_storage_path ~/yugabyte-data --backup_location ~/bug_repro/14861/ --no_auto_name --keyspace ysql.qqq --verbose create % git checkout arcpatch-D22774 % ybd % ./managed/devops/bin/yb_backup.py --mac --masters 127.0.0.1 --remote_yb_admin_binary ./build/latest/bin/yb-admin --remote_ysql_dump_binary ./build/latest/postgres/bin/ysql_dump --remote_ysql_shell_binary ./build/latest/postgres/bin/ysqlsh --no_ssh --storage_type nfs --nfs_storage_path ~/yugabyte-data --backup_location ~/bug_repro/14861/ --no_auto_name --keyspace ysql.qqq --verbose restore ``` Reviewers: slingam, skedia Reviewed By: skedia Subscribers: mhaddad, bogdan, slingam, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D22774
premkumr
pushed a commit
to premkumr/yugabyte-db
that referenced
this issue
Apr 10, 2023
…uld restore all the schema packings of snapshot Summary: With packed row enabled, in the case of an out-of-cluster backup/restore, the table id in the snapshot will be different from the id created in the restored cluster for the same table. So we need a way to find the correct table entry to merge schema packings properly. Assuming the snapshot folder is downloaded correctly, then this pairing is only needed for a colocated tablet (i.e., mapping colocated tables). Since each colocated table entry from the snapshot and its restored counterpart have the same colocation id, this diff uses this colocation id to find the corresponding restored table id thereby pairing the table entry. Test Plan: ``` ybd --cxx-test yb-backup-cross-feature-test --gtest_filter YBBackupTestWithPackedRows.YSQLSchemaPackingWithColocatedDB ./yb_build.sh --java-test 'org.yb.pgsql.TestYsqlPartitionedBackup#testColocatedPartitionedTable' ``` I also ran the manual test case in [[ yugabyte#15966 | GH15966 ]] to verify that bug is fixed by this diff. I also ran a modified version of the test case to verify we can successfully restore backups made prior to [[ yugabyte@8acc2a3 | 8acc2a3 ]]: ``` % git checkout 60f81b0 % ybd % ./bin/yb-ctl destroy && ./bin/yb-ctl create && ./bin/yb-ctl start % ysqlsh yugabyte=# CREATE DATABASE qqq colocated = true; yugabyte=# \c qqq # create partitioned table: qqq=# CREATE TABLE htest(k1 int, k2 text, k3 int, v1 int, v2 text) PARTITION BY HASH (k1); # Creating table htest, SQL statement: qqq=# CREATE TABLE htest_1 PARTITION OF htest(k1, k2, k3, v1, v2) FOR VALUES WITH (modulus 2, remainder 0); # Creating table htest, SQL statement: qqq=# CREATE TABLE htest_2 PARTITION OF htest(k1, k2, k3, v1, v2) FOR VALUES WITH (modulus 2, remainder 1); qqq=# CREATE INDEX ON htest_1(k1, k2) INCLUDE (k3, v1); qqq=# INSERT INTO htest VALUES (1, 2, 3, 4, 5); ^D % ./managed/devops/bin/yb_backup.py --mac --masters 127.0.0.1 --remote_yb_admin_binary ./build/latest/bin/yb-admin --remote_ysql_dump_binary ./build/latest/postgres/bin/ysql_dump --remote_ysql_shell_binary ./build/latest/postgres/bin/ysqlsh --no_ssh --storage_type nfs --nfs_storage_path ~/yugabyte-data --backup_location ~/bug_repro/14861/ --no_auto_name --keyspace ysql.qqq --verbose create % git checkout arcpatch-D22774 % ybd % ./managed/devops/bin/yb_backup.py --mac --masters 127.0.0.1 --remote_yb_admin_binary ./build/latest/bin/yb-admin --remote_ysql_dump_binary ./build/latest/postgres/bin/ysql_dump --remote_ysql_shell_binary ./build/latest/postgres/bin/ysqlsh --no_ssh --storage_type nfs --nfs_storage_path ~/yugabyte-data --backup_location ~/bug_repro/14861/ --no_auto_name --keyspace ysql.qqq --verbose restore ``` Reviewers: slingam, skedia Reviewed By: skedia Subscribers: mhaddad, bogdan, slingam, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D22774
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-4154
Description
This is a follow-up to #14296. For colocated tables, the algorithm to merge schema packings is more involved. The table ids involved in the snapshot on the source cluster are different from the table ids that restore creates on the target cluster. For a colocated tablet, there will be >=1 tables whose metadata will be colocated on the same tablet thus while merging schema packings from the snapshot, restore also needs to know about this old_table_id -> new_table_id mapping to merge the relevant packings of a table. We need to have a way for the restore script to somehow pass this information down to the tservers so that they are able to use it at the time of restore.
The text was updated successfully, but these errors were encountered: