diff --git a/docs/content/preview/yugabyte-voyager/known-issues/_index.md b/docs/content/preview/yugabyte-voyager/known-issues/_index.md index b077e273c556..27b0870e6cb1 100644 --- a/docs/content/preview/yugabyte-voyager/known-issues/_index.md +++ b/docs/content/preview/yugabyte-voyager/known-issues/_index.md @@ -25,6 +25,7 @@ Currently, yb-voyager doesn't support the following features: | ALTER VIEW | YugabyteDB does not yet support any schemas containing `ALTER VIEW` statements. | [48](https://github.com/yugabyte/yb-voyager/issues/48) | | USERS/GRANTS | Voyager does not support migrating the USERS and GRANTS from the source database to the target cluster. | | Unsupported data types | Data migration is unsupported for some data types, such as BLOB and XML. For others such as ANY and BFile, both schema and data migration is unsupported. Refer to [datatype mapping](../reference/datatype-mapping-oracle/) for the detailed list of data types. | | +| Unsupported PostgreSQL features | Yugabyte currently doesn't support the PostgreSQL features listed in [PostgreSQL compatibility](../../explore/ysql-language-features/postgresql-compatibility/#unsupported-postgresql-features). If such schema clauses are encountered, Voyager results in an error. | | ## Manual review diff --git a/docs/content/preview/yugabyte-voyager/known-issues/oracle.md b/docs/content/preview/yugabyte-voyager/known-issues/oracle.md index 91cb1a308a9a..1fae446fbaea 100644 --- a/docs/content/preview/yugabyte-voyager/known-issues/oracle.md +++ b/docs/content/preview/yugabyte-voyager/known-issues/oracle.md @@ -31,6 +31,7 @@ Cluster, Domain, Bitmap join, IOT indexes, and reverse indexes are not exported. - [Large-sized CLOB/NCLOB data is not supported](#large-sized-clob-nclob-data-is-not-supported) - [%TYPE syntax is unsupported](#type-syntax-is-unsupported) - [TRANSLATE USING is unsupported](#translate-using-is-unsupported) +- [Issue with importing data from tables with reserved keyword datatypes matching table names](#issue-with-importing-data-from-tables-with-reserved-keyword-datatypes-matching-table-names) ### Some numeric types are not exported @@ -338,3 +339,58 @@ OR **Result**: 'PostgreSQL' (in UTF8 (Unicode, 8-bit) encoding) --- + +### Issue with importing data from tables with reserved keyword datatypes matching table names + +**GitHub**: [Issue #1505](https://github.com/yugabyte/yb-voyager/issues/1505) + +**Description**: If there's a table with a name as a reserved word of datatype, then there is an issue in dumping the incorrect data dump for that table, possibly resulting in syntax errors when importing that exported data in target. + +**Workaround**: Modify the data to correct syntax as per the datatype exported for that table. + +**Example**: + +An example schema on the source database is as follows: + +```sql +create table "number"(id int PRIMARY KEY); +insert into "number" values(0); +insert into "number" values(1); +insert into "number" values(2); +insert into "number" values(3); +insert into "number" values(4); +``` + +The exported schema and data are as follows: + +```sql +create table "number"(id numeric(38) NOT NULL, PRIMARY KEY (id)); + +COPY number (id) FROM STDIN; +f +t +2 +3 +4 +\. +``` + +Error during data import is as follows: + +```sql +ERROR: invalid input syntax for type numeric: "t" (SQLSTATE 22P02), COPY number, line 1, column id: "t" +``` + +Workaround for the example is to modify the data to change the `f/t` to `0/1` respectively: + +```sql +COPY number (id) FROM STDIN; +0 +1 +2 +3 +4 +\. +``` + +--- diff --git a/docs/content/preview/yugabyte-voyager/known-issues/postgresql.md b/docs/content/preview/yugabyte-voyager/known-issues/postgresql.md index 5cd1406208a1..7d523920b49d 100644 --- a/docs/content/preview/yugabyte-voyager/known-issues/postgresql.md +++ b/docs/content/preview/yugabyte-voyager/known-issues/postgresql.md @@ -20,6 +20,7 @@ Review limitations and implement suggested workarounds to successfully migrate d - [Adding primary key to a partitioned table results in an error](#adding-primary-key-to-a-partitioned-table-results-in-an-error) - [Index creation on partitions fail for some YugabyteDB builds](#index-creation-on-partitions-fail-for-some-yugabytedb-builds) - [Creation of certain views in the rule.sql file](#creation-of-certain-views-in-the-rule-sql-file) +- [Indexes on INET type are not supported](#indexes-on-inet-type-are-not-supported) ### Adding primary key to a partitioned table results in an error @@ -137,3 +138,35 @@ CREATE OR REPLACE VIEW public.v1 AS FROM public.foo GROUP BY foo.n1; ``` + +--- + +### Indexes on INET type are not supported + +**GitHub**: [Issue #17017](https://github.com/yugabyte/yb-voyager/issues/17017) + +**Description**: If there is an index on a column of the INET type, it errors out during import. + +**Workaround**: Modify the column to a TEXT type. + +**Example** + +An example schema on the source database is as follows: + +```sql +create table test( id int primary key, f1 inet); +create index test_index on test(f1); +``` + +The import schema error is as follows: + +```sql +INDEXES_table.sql: CREATE INDEX test_index ON public.test USING btree (f1); +ERROR: INDEX on column of type 'INET' not yet supported (SQLSTATE 0A000) +``` + +Suggested workaround is to change the INET column to TEXT for the index creation to succeed as follows: + +```sql +create table test( id int primary key, f1 text); +``` diff --git a/docs/content/preview/yugabyte-voyager/migrate/_index.md b/docs/content/preview/yugabyte-voyager/migrate/_index.md index 65510405602d..6da1d3991888 100644 --- a/docs/content/preview/yugabyte-voyager/migrate/_index.md +++ b/docs/content/preview/yugabyte-voyager/migrate/_index.md @@ -15,63 +15,42 @@ menu: You can perform migration by taking your applications offline to perform the migration, migrate your data while your application is running (currently Oracle only), or add a fall-forward database for your live migration (currently Oracle only). -
+{{}} - + {{}} - + {{}} - - - -
+ {{}} + + {{}} + + {{}} + + {{}} + +{{}} diff --git a/docs/content/preview/yugabyte-voyager/migrate/assess-migration.md b/docs/content/preview/yugabyte-voyager/migrate/assess-migration.md new file mode 100644 index 000000000000..5b4f011a2bab --- /dev/null +++ b/docs/content/preview/yugabyte-voyager/migrate/assess-migration.md @@ -0,0 +1,122 @@ +--- +title: YB Voyager Migration Assessment +linkTitle: Assess migration +headcontent: Steps to create a migration assessment report +description: Steps to create a migration assessment report to ensure successful migration using YugabyteDB Voyager. +menu: + preview_yugabyte-voyager: + identifier: assess-migration + parent: migration-types + weight: 101 +techPreview: /preview/releases/versioning/#feature-availability +type: docs +--- + +The Voyager Migration Assessment feature is specifically designed to optimize the database migration process from various source databases, currently supporting PostgreSQL to YugabyteDB. Voyager conducts a detailed analysis of the source database by capturing essential metadata and metrics. It generates a comprehensive assessment report that recommends effective migration strategies, and provides key insights on ideal cluster configurations for optimal performance with YugabyteDB. + +## Overview + +Voyager collects various metadata or metrics from the source database, such as table columns metadata, sizes of tables/indexes, read/write IOPS for tables/indexes, and so on. With this data, an assessment report is prepared which covers the following key details: + +- **Database compatibility**: Assesses the compatibility of the source database with YugabyteDB, identifying unsupported features and data types. + +- **Cluster size evaluation**: Provides an estimation of the resource requirements for the target environment, helping in the planning and scaling of infrastructure needs. The sizing logic depends on various factors such as the size and number of tables in the source database, as well as the throughput requirements (read/write IOPS). + +- **Schema evaluation**: Reviews the database schema, to suggest effective sharding strategies for tables and indexes. + +- **Performance metrics**: Analyzes performance metrics to understand workload characteristics, and provides recommendations for optimization in YugabyteDB. + +- **Migration time estimation**: Offers a calculated estimate of the time needed to import data into YugabyteDB after export from the source database, helping with better migration planning. These estimates are calculated based on various experiments during data import to YugabyteDB. + +{{< warning title="Caveats" >}} +Recommendations are based on experiments done on [RF3](../../../architecture/docdb-replication/replication/#replication-factor) setup on instance types with 4GiB memory/core against YugabyteDB v2024.1. +{{< /warning >}} + +Note that for cases where it is not possible to provide database access to the client machine running voyager, you can gather the metadata from the source database using plain bash/psql scripts by voyager, and then use voyager to analyze the metadata directly. + +### Sample Migration Assessment Report + +A sample Migration Assessment Report is as follows: + +![Migration report](/images/migrate/assess-migration.jpg) + +## Generate a Migration Assessment Report + +1. [Install yb-voyager](../../install-yb-voyager/). +1. Prepare the source database. + + 1. Create a new user, `ybvoyager` as follows: + + ```sql + CREATE USER ybvoyager PASSWORD 'password'; + ``` + + 1. Grant necessary permissions to the `ybvoyager` user. + + ```sql + /* Switch to the database that you want to migrate.*/ + \c + + /* Grant the USAGE permission to the ybvoyager user on all schemas of the database.*/ + + SELECT 'GRANT USAGE ON SCHEMA ' || schema_name || ' TO ybvoyager;' FROM information_schema.schemata; \gexec + + /* The above SELECT statement generates a list of GRANT USAGE statements which are then executed by psql because of the \gexec switch. The \gexec switch works for PostgreSQL v9.6 and later. For older versions, you'll have to manually execute the GRANT USAGE ON SCHEMA schema_name TO ybvoyager statement, for each schema in the source PostgreSQL database. */ + + /* Grant SELECT permission on all the tables. */ + + SELECT 'GRANT SELECT ON ALL TABLES IN SCHEMA ' || schema_name || ' TO ybvoyager;' FROM information_schema.schemata; \gexec + ``` + +1. Assess migration - Voyager supports two primary modes for conducting migration assessments, depending on your access to the source database as follows: + + 1. **With source database connectivity**: This mode requires direct connectivity to the source database from the client machine where voyager is installed. You initiate the assessment by executing the `assess-migration` command of `yb-voyager`. This command facilitates a live analysis by interacting directly with the source database, to gather metadata required for assessment. A sample command is as follows: + + ```sh + yb-voyager assess-migration --source-db-type postgresql \ + --source-db-host hostname --source-db-user ybvoyager \ + --source-db-password password --source-db-name dbname \ + --source-db-schema schema1,schema2 --export-dir /path/to/export/dir + ``` + + 1. **Without source database connectivity**: In situations where direct access to the source database is restricted, there is an alternative approach. Voyager includes packages with scripts present at `/etc/yb-voyager/gather-assessment-metadata/postgresql`. You can perform the following steps with these scripts. + + 1. Copy the scripts to a machine which has access to the source database. + 1. Run the `yb-voyager-pg-gather-assessment-metadata.sh` script by providing the source connection string, the schema names, path to a directory where metadata will be saved, and an optional argument of an interval to capture the IOPS metadata of the source (in seconds with a default value of 120). For example, + + ```sh + /path/to/yb-voyager-pg-gather-assessment-metadata.sh 'postgresql://ybvoyager@host:port/dbname' 'schema1|schema2' '/path/to/assessment_metadata_dir' '60' + ``` + + 1. Copy the metadata directory to the client machine on which voyager is installed, and run the `assess-migration` command by specifying the path to the metadata directory as follows: + + ```sh + yb-voyager assess-migration --source-db-type postgresql \ + --assessment-metadata-dir /path/to/assessment_metadata_dir --export-dir /path/to/export/dir + ``` + + The output of both the methods is a migration assessment report, and its path is printed on the console. + + {{< warning title="Important" >}} +For the most accurate migration assessment, the source database must be actively handling its typical workloads at the time the metadata is gathered. This ensures that the recommendations for sharding strategies and cluster sizing are well-aligned with the database's real-world performance and operational needs. + {{< /warning >}} + +1. Create a target YugabyteDB cluster as follows: + + 1. Create a cluster based on the sizing recommendations in the assessment report. + 1. Create a database with colocation set to TRUE. + + ```sql + CREATE DATABASE with COLOCATION=TRUE; + ``` + +1. Proceed with migration with one of the migration workflows: + + - [Offline migration](../../migrate/migrate-steps/) + - [Live migration](../../migrate/live-migrate/) + - [Live migration with fall-forward](../../migrate/live-fall-forward/) + - [Live migration with fall-back](../../migrate/live-fall-back/) + +## Learn more + +- [Assess migration CLI](../../reference/assess-migration/) \ No newline at end of file diff --git a/docs/content/preview/yugabyte-voyager/migrate/live-fall-back.md b/docs/content/preview/yugabyte-voyager/migrate/live-fall-back.md index 0059231fddf5..88b9ac956f1e 100644 --- a/docs/content/preview/yugabyte-voyager/migrate/live-fall-back.md +++ b/docs/content/preview/yugabyte-voyager/migrate/live-fall-back.md @@ -75,53 +75,144 @@ Before proceeding with migration, ensure that you have completed the following s Create a new database user, and assign the necessary user permissions. -