Skip to content

Commit

Permalink
made changes to stable as well
Browse files Browse the repository at this point in the history
  • Loading branch information
@aishwarya24
aishwarya24 committed Jul 25, 2022
1 parent a6213df commit 94d9392
Show file tree
Hide file tree
Showing 4 changed files with 152 additions and 221 deletions.
88 changes: 0 additions & 88 deletions docs/content/preview/migrate/manual-import/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,91 +27,3 @@ The steps below cover how to manually migrate PostgreSQL data and applications t
{{< tip title="Migrate using YugabyteDB Voyager" >}}
To automate your migration from PostgreSQL to YugabyteDB, use [YugabyteDB Voyager](../yb-voyager/).
{{< /tip >}}

<!-- <div class="row">
<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="migrate-schema/">
<div class="head">
<div class="icon">
<i class="icon-database-alt2"></i>
</div>
<div class="title">Migrate a DDL schema</div>
</div>
<div class="body">
Migrate your DDL schema from PostgreSQL to YugabyteDB.
</div>
</a>
</div>
<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="migrate-application/">
<div class="head">
<div class="icon">
<i class="icon-database-alt2"></i>
</div>
<div class="title">Migrate a PostgreSQL application</div>
</div>
<div class="body">
Migrate a PostgreSQL application to YugabyteDB.
</div>
</a>
</div>
<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="export-data/">
<div class="head">
<div class="icon">
<i class="icon-database-alt2"></i>
</div>
<div class="title">Export PostgreSQL data</div>
</div>
<div class="body">
Export data from PostgreSQL for importing into YugabyteDB.
</div>
</a>
</div>
<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="prepare-cluster/">
<div class="head">
<div class="icon">
<i class="icon-database-alt2"></i>
</div>
<div class="title">Prepare a cluster</div>
</div>
<div class="body">
Prepare your YugabyteDB cluster for data import.
</div>
</a>
</div>
<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="import-data/">
<div class="head">
<div class="icon">
<i class="icon-database-alt2"></i>
</div>
<div class="title">Import PostgreSQL data</div>
</div>
<div class="body">
Import PostgreSQL data into a YugabyteDB cluster.
</div>
</a>
</div>
<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="verify-migration/">
<div class="head">
<div class="icon">
<i class="icon-database-alt2"></i>
</div>
<div class="title">Verify the migration</div>
</div>
<div class="body">
Verify the migration to YugabyteDB was successful.
</div>
</a>
</div>
</div> -->
99 changes: 9 additions & 90 deletions docs/content/stable/migrate/manual-import/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
title: Manual import
headerTitle: Manual import
linkTitle: Manual import
description: Migrate PostgreSQL data to YugabyteDB.
description: Manual PostgreSQL import to YugabyteDB.
image: /images/section_icons/develop/learn.png
headcontent: Migrate PostgreSQL data to YugabyteDB using ysql_dump.
headcontent: Manual PostgreSQL import to YugabyteDB.
menu:
stable:
identifier: manual-import
Expand All @@ -15,94 +15,13 @@ type: indexpage

The steps below cover how to manually migrate PostgreSQL data and applications to YugabyteDB.

- [Convert a PostgreSQL schema](migrate-schema/)
- [Migrate a PostgreSQL application](migrate-application/)
- [Export PostgreSQL data](export-data/)
- [Prepare a cluster](prepare-cluster/)
- [Import PostgreSQL data](import-data/)
- [Verify a migration](verify-migration/)

{{< tip title="Migrate using YugabyteDB Voyager" >}}
To automate your migration from PostgreSQL to YugabyteDB, use [YugabyteDB Voyager](../yb-voyager/).
{{< /tip >}}

<div class="row">

<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="migrate-schema/">
<div class="head">
<div class="icon">
<i class="icon-database-alt2"></i>
</div>
<div class="title">Migrate a DDL schema</div>
</div>
<div class="body">
Migrate your DDL schema from PostgreSQL to YugabyteDB.
</div>
</a>
</div>

<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="migrate-application/">
<div class="head">
<div class="icon">
<i class="icon-database-alt2"></i>
</div>
<div class="title">Migrate a PostgreSQL application</div>
</div>
<div class="body">
Migrate a PostgreSQL application to YugabyteDB.
</div>
</a>
</div>

<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="export-data/">
<div class="head">
<div class="icon">
<i class="icon-database-alt2"></i>
</div>
<div class="title">Export PostgreSQL data</div>
</div>
<div class="body">
Export data from PostgreSQL for importing into YugabyteDB.
</div>
</a>
</div>

<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="prepare-cluster/">
<div class="head">
<div class="icon">
<i class="icon-database-alt2"></i>
</div>
<div class="title">Prepare a cluster</div>
</div>
<div class="body">
Prepare your YugabyteDB cluster for data import.
</div>
</a>
</div>

<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="import-data/">
<div class="head">
<div class="icon">
<i class="icon-database-alt2"></i>
</div>
<div class="title">Import PostgreSQL data</div>
</div>
<div class="body">
Import PostgreSQL data into a YugabyteDB cluster.
</div>
</a>
</div>

<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="verify-migration/">
<div class="head">
<div class="icon">
<i class="icon-database-alt2"></i>
</div>
<div class="title">Verify the migration</div>
</div>
<div class="body">
Verify the migration to YugabyteDB was successful.
</div>
</a>
</div>

</div>
85 changes: 56 additions & 29 deletions docs/content/stable/migrate/manual-import/export-data.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,57 +11,84 @@ menu:
type: docs
---

The recommended way to export data from PostgreSQL for purposes of importing it to YugabyteDB is using the CSV format.
The recommended way to export data from PostgreSQL for purposes of importing it to YugabyteDB is via CSV files using the COPY command.
However, for exporting an entire database that consists of smaller datasets, you can use the YugabyteDB [`ysql_dump`](../../../admin/ysql-dump/) utility.

## Exporting an entire database
{{< tip title="Migrate using YugabyteDB Voyager" >}}
To automate your migration from PostgreSQL to YugabyteDB, use [YugabyteDB Voyager](../../yb-voyager/). To learn more, refer to the [export schema](../../yb-voyager/migrate-steps/#export-and-analyze-schema) and [export data](../../yb-voyager/migrate-steps/#export-data) steps.
{{< /tip >}}

The recommended way to dump an entire database from PostgreSQL is to use the YugabyteDB [`ysql_dump`](../../../admin/ysql-dump/) backup utility, which is in turn derived from PostgreSQL pg_dump.
## Export data into CSV files using the COPY command

```sh
$ ysql_dump -d mydatabase > mydatabase-dump.sql
To export the data, connect to the source PostgreSQL database using the psql tool, and execute the COPY TO command as follows:

```sql
COPY <table_name>
TO '<table_name>.csv'
WITH (FORMAT CSV DELIMITER ',' HEADER);
```

{{< note title="Note" >}}
The `ysql_dump` approach has been tested on PostgreSQL v11.2, and may not work on very new versions of PostgreSQL. To export an entire database in these cases, use the pg_dump tool, which is documented in detail in the [PostgreSQL documentation on pg_dump](https://www.postgresql.org/docs/12/app-pgdump.html).
{{< /note >}}

## Export using COPY

This is an alternative to using ysql_dump in order to export a single table from the source PostgreSQL database into CSV files. This tool allows extracting a subset of rows and/or columns from a table. This can be achieved by connecting to the source DB using psql and using the `COPY TO` command, as shown below.
The COPY TO command exports a single table, so you should execute it for every table that you want to export.

```sql
COPY mytable TO 'export-1.csv' DELIMITER ',' CSV HEADER;
```
{{< /note >}}

To extract a subset of rows from a table, it is possible to output the result of an SQL command.
It is also possible to export a subset of rows based on a condition:

```sql
COPY (
SELECT * FROM mytable
WHERE <where condition>
) TO 'export-1.csv' DELIMITER ',' CSV HEADER;
SELECT * FROM <table_name>
WHERE <condition>
)
TO '<table_name>.csv'
WITH (FORMAT CSV DELIMITER ',' HEADER);
```

The various options here are described in detail in the [PostgreSQL documentation for the COPY command](https://www.postgresql.org/docs/12/sql-copy.html).

## Run large table exports in parallel
For all available options provided by the COPY TO command, refer to the [PostgreSQL documentation](https://www.postgresql.org/docs/current/sql-copy.html).

Exporting large data sets from PostgreSQL can be made efficient by running multiple COPY processes in parallel for a subset of data. This will result in multiple csv files being produced, which can subsequently be imported in parallel.
### Parallelize large table export

An example of running multiple exports in parallel is shown below. Remember to use a suitable value for *num_rows_per_export*, for example 1 million rows.
For large tables, it might be beneficial to parallelize the process by exporting data in chunks as follows:

```sql
COPY (
SELECT * FROM mytable
ORDER BY primary_key_col
SELECT * FROM <table_name>
ORDER BY <primary_key_col>
LIMIT num_rows_per_export OFFSET 0
) TO 'export-1.csv' DELIMITER ',' CSV HEADER;
)
TO '<table_name>_1.csv'
WITH (FORMAT CSV DELIMITER ',' HEADER);
```

```sql
COPY (
SELECT * FROM mytable
ORDER BY primary_key_col
SELECT * FROM <table_name>
ORDER BY <primary_key_col>
LIMIT num_rows_per_export OFFSET num_rows_per_export
) TO 'export-2.csv' WITH CSV;
)
TO '<table_name>_2.csv'
WITH (FORMAT CSV DELIMITER ',' HEADER);
```

```sql
COPY (
SELECT * FROM <table_name>
ORDER BY <primary_key_col>
LIMIT num_rows_per_export OFFSET num_rows_per_export * 2
)
TO '<table_name>_3.csv'
WITH (FORMAT CSV DELIMITER ',' HEADER);
```

...
You can run the above commands in parallel to speed up the process. This approach will also produce multiple CSV files, allowing for parallel import on the YugabyteDB side.

## Export data into SQL script using ysql_dump

An alternative way to export the data is using the YugabyteDB [`ysql_dump`](../../../admin/ysql-dump/) backup utility, which is derived from PostgreSQL pg_dump.

```sh
$ ysql_dump -d <database_name> > <database_name>.sql
```

`ysql_dump` is the ideal option for smaller datasets, because it allows you to export a whole database by running a single command. However, the COPY command is recommended for large databases, because it significantly enhances the performance.

0 comments on commit 94d9392

Please sign in to comment.