Skip to content

Commit b5ec9c6

Browse files
moved the splitting up of Fetch into a separate PR, fixed links for this PR
1 parent 91df8d5 commit b5ec9c6

16 files changed

+24
-1086
lines changed

src/current/_includes/molt/molt-docker.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
MOLT Fetch, Verify, and Replicator are likely to run more slowly in a Docker container than on a local machine. To improve performance, increase the memory or compute resources, or both, on your Docker container.
44

5-
{% if page.name == "molt-fetch.md" or page.name == "molt-fetch-install.md" %}
5+
{% if page.name == "molt-fetch.md" %}
66
#### Authentication
77

88
When using MOLT Fetch with [cloud storage](#bucket-path), it is necessary to specify volumes and environment variables, as described in the following sections for [Google Cloud Storage](#google-cloud-storage) and [Amazon S3](#amazon-s3).

src/current/_includes/molt/molt-drop-constraints-indexes.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
To optimize data load performance, drop all non-`PRIMARY KEY` [constraints]({% link {{ site.current_cloud_version }}/alter-table.md %}#drop-constraint) and [indexes]({% link {{site.current_cloud_version}}/drop-index.md %}) on the target CockroachDB database before migrating:
2-
{% if page.name == "molt-fetch.md" or page.name == "molt-fetch-best-practices.md" %}
2+
{% if page.name == "molt-fetch.md" %}
33
- [`FOREIGN KEY`]({% link {{ site.current_cloud_version }}/foreign-key.md %})
44
- [`UNIQUE`]({% link {{ site.current_cloud_version }}/unique.md %})
55
- [Secondary indexes]({% link {{ site.current_cloud_version }}/schema-design-indexes.md %})

src/current/_includes/molt/molt-install.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ To display the current version of each binary, run `molt --version` and `replica
2222

2323
For previous binaries, refer to the [MOLT version manifest](https://molt.cockroachdb.com/molt/cli/versions.html). {% if page.name != "molt.md" %}For release details, refer to the [MOLT changelog]({% link releases/molt.md %}).{% endif %}
2424

25-
{% if page.name == "molt-fetch.md" or page.name == "molt.md" or page.name == "molt-fetch-install" %}
25+
{% if page.name == "molt-fetch.md" or page.name == "molt.md" %}
2626
{{site.data.alerts.callout_info}}
2727
MOLT Fetch is supported on Red Hat Enterprise Linux (RHEL) 9 and above.
2828
{{site.data.alerts.end}}
@@ -55,7 +55,7 @@ docker pull cockroachdb/molt:oracle-latest
5555
~~~
5656
{% endif %}
5757

58-
{% if page.name != "molt-fetch.md" and page.name != "molt-fetch-install.md" %}
58+
{% if page.name != "molt-fetch.md" %}
5959
#### MOLT Replicator
6060

6161
[Docker images for MOLT Replicator](https://hub.docker.com/r/cockroachdb/replicator/tags) are also available as a standalone binary:

src/current/molt/migrate-bulk-load.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ Perform a one-time bulk load of source data into CockroachDB.
2121

2222
Perform the bulk load of the source data.
2323

24-
1. Run the [MOLT Fetch]({% link molt/molt-fetch-overview.md %}) command to move the source data into CockroachDB. This example command passes the source and target connection strings [as environment variables](#secure-connections), writes [intermediate files](#intermediate-file-storage) to S3 storage, and uses the `truncate-if-exists` [table handling mode](#table-handling-mode) to truncate the target tables before loading data. It limits the migration to a single schema and filters for three specific tables. The [data load mode](#data-load-mode) defaults to `IMPORT INTO`. Include the `--ignore-replication-check` flag to skip replication checkpoint queries, which eliminates the need to configure the source database for logical replication.
24+
1. Run the [MOLT Fetch]({% link molt/molt-fetch.md %}) command to move the source data into CockroachDB. This example command passes the source and target connection strings [as environment variables](#secure-connections), writes [intermediate files](#intermediate-file-storage) to S3 storage, and uses the `truncate-if-exists` [table handling mode](#table-handling-mode) to truncate the target tables before loading data. It limits the migration to a single schema and filters for three specific tables. The [data load mode](#data-load-mode) defaults to `IMPORT INTO`. Include the `--ignore-replication-check` flag to skip replication checkpoint queries, which eliminates the need to configure the source database for logical replication.
2525

2626
<section class="filter-content" markdown="1" data-scope="postgres">
2727
{% include_cached copy-clipboard.html %}
@@ -91,6 +91,6 @@ Perform a cutover by resuming application traffic, now to CockroachDB.
9191
- [Migration Overview]({% link molt/migration-overview.md %})
9292
- [Migration Strategy]({% link molt/migration-strategy.md %})
9393
- [MOLT Schema Conversion Tool]({% link cockroachcloud/migrations-page.md %})
94-
- [MOLT Fetch]({% link molt/molt-fetch-overview.md %})
94+
- [MOLT Fetch]({% link molt/molt-fetch.md %})
9595
- [MOLT Verify]({% link molt/molt-verify.md %})
9696
- [Migration Failback]({% link molt/migrate-failback.md %})

src/current/molt/migrate-to-cockroachdb.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,8 @@ MOLT Fetch supports various migration flows using [MOLT Fetch modes]({% link mol
1111

1212
| Migration flow | Mode | Description | Best for |
1313
|---------------------------------------------------------------------|------------------------------|---------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
14-
| [Bulk load]({% link molt/migrate-bulk-load.md %}) | `--mode data-load` | Perform a one-time bulk load of source data into CockroachDB. | Testing, migrations with [planned downtime]({% link molt/migration-strategy.md %}#approach-to-downtime) |
15-
| [Load and replicate]({% link molt/migrate-load-replicate.md %}) | MOLT Fetch + MOLT Replicator | Load source data using MOLT Fetch, then replicate subsequent changes using MOLT Replicator. | [Minimal downtime]({% link molt/migration-strategy.md %}#approach-to-downtime) migrations |
14+
| [Bulk load]({% link molt/migrate-bulk-load.md %}) | `--mode data-load` | Perform a one-time bulk load of source data into CockroachDB. | Testing, migrations with [planned downtime]({% link molt/migration-considerations.md %}#permissible-downtime) |
15+
| [Load and replicate]({% link molt/migrate-load-replicate.md %}) | MOLT Fetch + MOLT Replicator | Load source data using MOLT Fetch, then replicate subsequent changes using MOLT Replicator. | [Minimal downtime]({% link molt/migration-considerations.md %}#permissible-downtime) migrations |
1616
| [Resume replication]({% link molt/migrate-resume-replication.md %}) | `--mode replication-only` | Resume replication from a checkpoint after interruption. | Resuming interrupted migrations, post-load sync |
1717
| [Failback]({% link molt/migrate-failback.md %}) | `--mode failback` | Replicate changes from CockroachDB back to the source database. | [Rollback]({% link molt/migrate-failback.md %}) scenarios |
1818

src/current/molt/migration-considerations-phases.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -48,11 +48,11 @@ Tips for picking slices:
4848

4949
Phased and unphased migrations are both supported natively by MOLT.
5050

51-
By default, [MOLT Fetch]() moves all data from the source database to CockroachDB. However, you can use the `--schema-filter`, `--table-filter`, and `--filter-path` flags to selective migrate data from the source to the target. Learn more about [schema and table selection]({% link molt/molt-fetch-usage.md %}#schema-and-table-selection) and [selective data movement]({% link molt/molt-fetch-usage.md %}#selective-data-movement), both of which can enable a phased migration.
51+
By default, [MOLT Fetch]({% link molt/molt-fetch.md %}) moves all data from the source database to CockroachDB. However, you can use the `--schema-filter`, `--table-filter`, and `--filter-path` flags to selective migrate data from the source to the target. Learn more about [schema and table selection]({% link molt/molt-fetch.md %}#schema-and-table-selection) and [selective data movement]({% link molt/molt-fetch.md %}#selective-data-movement), both of which can enable a phased migration.
5252

53-
Similarly, you can use [MOLT Verify]()'s `--schema-filter` and `--table-filter` flags to run validation checks on subsets of the data in your source and target databases. In a phased migration, you will likely want to verify data at the end of each migration phase, rather than at the end of the entire migration.
53+
Similarly, you can use [MOLT Verify]({% link molt/molt-verify.md %})'s `--schema-filter` and `--table-filter` flags to run validation checks on subsets of the data in your source and target databases. In a phased migration, you will likely want to verify data at the end of each migration phase, rather than at the end of the entire migration.
5454

55-
[MOLT Replicator]() replicates full tables by default. If you choose to combine phased migration with [continuous replication]({% link molt/migration-considerations-replication.md %}), you will either need to select phases that include whole tables, or else use [userscripts]({% link molt/molt-replicator.md %}#flags) to select rows to replicate.
55+
[MOLT Replicator]({% link molt/molt-replicator.md %}) replicates full tables by default. If you choose to combine phased migration with [continuous replication]({% link molt/migration-considerations-replication.md %}), you will either need to select phases that include whole tables, or else use [userscripts]({% link molt/molt-replicator.md %}#flags) to select rows to replicate.
5656

5757
## Example sequences
5858

@@ -73,4 +73,4 @@ Similarly, you can use [MOLT Verify]()'s `--schema-filter` and `--table-filter`
7373
- [Migration Overview]({% link molt/migration-overview.md %})
7474
- [Migration Considerations]({% link molt/migration-considerations.md %})
7575
- [Continuous Replication]({% link molt/migration-considerations-replication.md %})
76-
- [MOLT Fetch]({% link molt/molt-fetch-overview.md %})
76+
- [MOLT Fetch]({% link molt/molt-fetch.md %})

src/current/molt/migration-considerations.md

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ When deciding on the options for each migration variable, consider the following
3030

3131
### Permissible downtime
3232

33-
How much downtime can your application tolerate during the migration? This is one of the most critical factors in determining your migration approach, and it may influence your choices for [migration granularity](), [continuous replication](), and [cutover strategy]().
33+
How much downtime can your application tolerate during the migration? This is one of the most critical factors in determining your migration approach, and it may influence your choices for [migration granularity]({% link molt/migration-considerations-phases.md %}), [continuous replication]({% link molt/migration-considerations-replication.md %}), and [cutover strategy]({% link molt/migration-considerations-cutover.md %}).
3434

3535
- **Planned downtime** is made known to your users in advance. It involves taking the application offline, conducting the migration, and bginging the application back online on CockroachDB.
3636

@@ -40,29 +40,29 @@ How much downtime can your application tolerate during the migration? This is on
4040

4141
- **Minimal downtime** impacts as few customers as possible, ideally without impacting their regular usage. If your application is intentionally offline at certain times (e.g., outside business hours), you can migrate the data without users noticing. Alternatively, if your application's functionality is not time-sensitive (e.g., it sends batched messages or emails), you can queue requests while the system is offline and process them after completing the migration to CockroachDB.
4242

43-
- **Near-zero downtime** is necessary for mission-critical applications. For these migrations, consider cutover strategies that keep applications online for as long as possible, and which utilize [continuous replication]().
43+
- **Near-zero downtime** is necessary for mission-critical applications. For these migrations, consider cutover strategies that keep applications online for as long as possible, and which utilize [continuous replication]({% link molt/migration-considerations-replication.md %}).
4444

4545
In addition to downtime duration, consider whether your application could support windows of **reduced functionality** in which some, but not all, application functionality is brought offline. For example, you can disable writes but not reads while you migrate the application data, and queue data to be written after completing the migration.
4646

4747
### Migration timeframe and allowable complexity
4848

49-
When do you need to complete the migration? How many team members can be allocated for this effort? How much complex orchestration can your team manage? These factors may influence your choices for [migration granularity](), [continuous replication](), and [cutover strategy]().
49+
When do you need to complete the migration? How many team members can be allocated for this effort? How much complex orchestration can your team manage? These factors may influence your choices for [migration granularity]({% link molt/migration-considerations-phases.md %}), [continuous replication]({% link molt/migration-considerations-replication.md %}), and [cutover strategy]({% link molt/migration-considerations-cutover.md %}).
5050

51-
- Migrations with a short timeline, or which cannot accommodate high complexity, may want to migrate data [all at once](), without utilizing [continuous replication](), and requiring [manual reconciliation]() in the event of migration failure.
51+
- Migrations with a short timeline, or which cannot accommodate high complexity, may want to migrate data [all at once]({% link molt/migration-considerations-phases.md %}), without utilizing [continuous replication]({% link molt/migration-considerations-replication.md %}), and requiring [manual reconciliation]({% link molt/migration-considerations-rollback.md %}) in the event of migration failure.
5252

53-
- Migrations with a long timeline, or which can accomodate complexity, may want to migrate data [in phases](). If the migration requires minimal downtime, these migrations may also want to utilize [continuous replication](). If the migration is low in risk-tolerance, these migrations may also want to enable [failback]().
53+
- Migrations with a long timeline, or which can accomodate complexity, may want to migrate data [in phases]({% link molt/migration-considerations-phases.md %}). If the migration requires minimal downtime, these migrations may also want to utilize [continuous replication]({% link molt/migration-considerations-replication.md %}). If the migration is low in risk-tolerance, these migrations may also want to enable [failback]({% link molt/migration-considerations-rollback.md %}).
5454

5555
### Risk tolerance
5656

57-
How much risk is your organization willing to accept during the migration? This may influence your choices for [migration granularity](), [validation strategy](), and [rollback plan]().
57+
How much risk is your organization willing to accept during the migration? This may influence your choices for [migration granularity]({% link molt/migration-considerations-phases.md %}), [validation strategy]({% link molt/migration-considerations-validation.md %}), and [rollback plan]({% link molt/migration-considerations-rollback.md %}).
5858

59-
- Risk-averse migrations should prefer [phased migrations]() that limit the blast radius of any issues. Start with low-risk slices (e.g., a small cohort of tenants or a non-critical service), [validate thoroughly](), and progressively expand to higher-value workloads. These migrations may also prefer [rollback plans]() that enable quick recovery in the event of migration issues.
59+
- Risk-averse migrations should prefer [phased migrations]({% link molt/migration-considerations-phases.md %}) that limit the blast radius of any issues. Start with low-risk slices (e.g., a small cohort of tenants or a non-critical service), [validate thoroughly]({% link molt/migration-considerations-validation.md %}), and progressively expand to higher-value workloads. These migrations may also prefer [rollback plans]({% link molt/migration-considerations-rollback.md %}) that enable quick recovery in the event of migration issues.
6060

61-
- For risk-tolerant migrations, it may be acceptable to migrate [all of your data at once](). Less stringent [validation strategies]() and [manual reconciliation]() in the event of a migration failure may also be acceptable.
61+
- For risk-tolerant migrations, it may be acceptable to migrate [all of your data at once]({% link molt/migration-considerations-phases.md %}). Less stringent [validation strategies]({% link molt/migration-considerations-validation.md %}) and [manual reconciliation]({% link molt/migration-considerations-rollback.md %}) in the event of a migration failure may also be acceptable.
6262

6363
___
6464

65-
These above factors are only a subset of all of what you'll want to consider in the decision-making about your CockroachDB migration, along with your specific business requirements and technical constraints. It's recommended that you document these decisions and the reasoning behind them as part of your [migration plan]().
65+
These above factors are only a subset of all of what you'll want to consider in the decision-making about your CockroachDB migration, along with your specific business requirements and technical constraints. It's recommended that you document these decisions and the reasoning behind them as part of your [migration plan]({% link molt/migration-strategy.md %}#develop-a-migration-plan).
6666

6767
## See also
6868

src/current/molt/migration-overview.md

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,6 @@ This page provides an overview of the following:
1313

1414
- Overall [migration sequence](#migration-sequence)
1515
- [MOLT tools](#molt-tools)
16-
- Supported [migration flows](#migration-flows)
1716

1817
## Migration sequence
1918

@@ -48,8 +47,6 @@ The MOLT (Migrate Off Legacy Technology) toolkit enables safe, minimal-downtime
4847
<img src="{{ 'images/molt/molt_flows_1.svg' | relative_url }}" alt="MOLT tooling overview" style="max-width:100%" />
4948
</div>
5049

51-
For more details, refer to [Migration flows](#migration-flows).
52-
5350
## MOLT tools
5451

5552
[MOLT (Migrate Off Legacy Technology)]({% link releases/molt.md %}) is a set of tools for schema conversion, data load, replication, and validation. Migrations with MOLT are resilient, restartable, and scalable to large data sets.
@@ -103,7 +100,7 @@ The [MOLT Schema Conversion Tool]({% link cockroachcloud/migrations-page.md %})
103100

104101
[MOLT Fetch]({% link molt/molt-fetch.md %}) performs the initial data load to CockroachDB. It supports:
105102

106-
- [Multiple migration flows](#migration-flows) via `IMPORT INTO` or `COPY FROM`.
103+
- Multiple migration flows via `IMPORT INTO` or `COPY FROM`.
107104
- Data movement via [cloud storage, local file servers, or direct copy]({% link molt/molt-fetch.md %}#data-path).
108105
- [Concurrent data export]({% link molt/molt-fetch.md %}#best-practices) from multiple source tables and shards.
109106
- [Schema transformation rules]({% link molt/molt-fetch.md %}#transformations).

0 commit comments

Comments
 (0)