Recovery and I/O
+Recovery and I/O
Change data capture (Changefeeds)
diff --git a/src/current/_includes/releases/v23.1/v23.1.21.md b/src/current/_includes/releases/v23.1/v23.1.21.md index fc4d6200ddf..4c60598830f 100644 --- a/src/current/_includes/releases/v23.1/v23.1.21.md +++ b/src/current/_includes/releases/v23.1/v23.1.21.md @@ -8,7 +8,7 @@ Release Date: May 7, 2024 - The `FORCE_INVERTED_INDEX` hint causes the [optimizer]({% link v23.1/cost-based-optimizer.md %}) to prefer a query plan scan over any [inverted index]({% link v23.1/inverted-indexes.md %}) of the hinted table. An error is emitted if no such query plan can be generated. [#122301][#122301] - Introduced three new [cluster settings]({% link v23.1/cluster-settings.md %}) for controlling [table statistics]({% link v23.1/cost-based-optimizer.md %}#table-statistics) forecasting: - - [`sql.stats.forecasts.min_observations`]({% link v23.1/cluster-settings.md %}#setting-sql-stats-forecasts-min-observations) is the minimum number of observed statistics required to produce a forecast. + - [`sql.stats.forecasts.min_observations`]({% link v23.1/cluster-settings.md %}) is the minimum number of observed statistics required to produce a forecast. - [`sql.stats.forecasts.min_goodness_of_fit`]({% link v23.1/cluster-settings.md %}#setting-sql-stats-forecasts-min-goodness-of-fit) is the minimum R² (goodness of fit) measurement required from all predictive models to use a forecast. - [`sql.stats.forecasts.max_decrease`]({% link v23.1/cluster-settings.md %}#setting-sql-stats-forecasts-max-decrease) is the most a prediction can decrease, expressed as the minimum ratio of the prediction to the lowest prior observation. [#122990][#122990] - Added a [session variable]({% link v23.1/set-vars.md %}) `optimizer_use_improved_multi_column_selectivity_estimate`, which if enabled, causes the [optimizer]({% link v23.1/cost-based-optimizer.md %}) to use an improved selectivity estimate for multi-column predicates. This setting will default to `true` on v24.2 and later, and `false` on prior versions. [#123068][#123068] diff --git a/src/current/_includes/releases/v23.2/v23.2.0-alpha.1.md b/src/current/_includes/releases/v23.2/v23.2.0-alpha.1.md index ee50d8cf863..63ba4aaf471 100644 --- a/src/current/_includes/releases/v23.2/v23.2.0-alpha.1.md +++ b/src/current/_includes/releases/v23.2/v23.2.0-alpha.1.md @@ -11,6 +11,7 @@ Release Date: September 26, 2023 - When customizing the [SQL shell's interactive prompt]({% link v23.2/cockroach-sql.md %}), the special sequence `%M` now expands to the full host name instead of the combination of host name and port number. To include the port number explicitly, use `%>`. The special sequence `%m` now expands to the host name up to the first period. [#105137][#105137] - The [`cockroach debug zip`]({% link v23.2/cockroach-debug-zip.md %}) command stores data retrieved from SQL tables in the remote cluster using the TSV format by default. [#107474][#107474] - The [`changefeed.protect_timestamp.max_age` cluster setting]({% link v23.2/protect-changefeed-data.md %}) will only apply to newly created changefeeds in v23.2. For existing changefeeds, you can set the [`protect_data_from_gc_on_pause`]({% link v23.2/create-changefeed.md %}#protect-pause) option so that changefeeds do not experience infinite retries and accumulate protected change data. You can use the [`ALTER CHANGEFEED`]({% link v23.2/alter-changefeed.md %}) statement to add `protect_data_from_gc_on_pause` to existing changefeeds. [#103539][#103539] +- The `changefeed.new_pubsub_sink_enabled` cluster setting is now enabled by default, which improves changefeed throughput. With this setting enabled, the top-level fields in JSON-encoded messages are capitalized: `{Key: ..., Value: ..., Topic: ...}`. After upgrading to CockroachDB v23.2, you may need to reconfigure downstream systems to parse the new message format. If you disable this setting, changefeeds emitting to Pub/Sub sinks with JSON-encoded events have the top-level message fields all lowercase: `{key: ..., value: ..., topic: ...}`.Security updates
diff --git a/src/current/_includes/releases/v24.1/v24.1.0-alpha.1.md b/src/current/_includes/releases/v24.1/v24.1.0-alpha.1.md index eeed1ab4764..02332ef2759 100644 --- a/src/current/_includes/releases/v24.1/v24.1.0-alpha.1.md +++ b/src/current/_includes/releases/v24.1/v24.1.0-alpha.1.md @@ -6,7 +6,7 @@ Release Date: March 7, 2024Backward-incompatible changes
-- [`AS OF SYSTEM TIME`]({% link v23.2/as-of-system-time.md %}) queries can no longer use a timestamp followed by a question mark to signifiy a future-time value. This was an undocumented syntax. [#116830][#116830] +- [`AS OF SYSTEM TIME`]({% link v23.2/as-of-system-time.md %}) queries can no longer use a timestamp followed by a question mark to signify a future-time value. This was an undocumented syntax. [#116830][#116830]{{ site.data.products.enterprise }} edition changes
diff --git a/src/current/_includes/releases/v24.1/v24.1.0-beta.3.md b/src/current/_includes/releases/v24.1/v24.1.0-beta.3.md index 0852831e401..c4bd4dc4f69 100644 --- a/src/current/_includes/releases/v24.1/v24.1.0-beta.3.md +++ b/src/current/_includes/releases/v24.1/v24.1.0-beta.3.md @@ -9,7 +9,7 @@ Release Date: April 30, 2024 - Updated the [`SHOW GRANTS`]({% link v24.1/show-grants.md %}) responses to display the `object_type` and `object_name`, which has replaced the `relation_name` column. [#122823][#122823] - Added [external connection]({% link v24.1/create-external-connection.md %}) granted privileges to the [`SHOW GRANTS`]({% link v24.1/show-grants.md %}) command. [#122823][#122823] - Introduced three new [cluster settings]({% link v24.1/cluster-settings.md %}) for controlling table statistics forecasting: - - [`sql.stats.forecasts.min_observations`]({% link v24.1/cluster-settings.md %}#setting-sql-stats-forecasts-min-observations) is the minimum number of observed statistics required to produce a forecast. + - [`sql.stats.forecasts.min_observations`]({% link v24.1/cluster-settings.md %}) is the minimum number of observed statistics required to produce a forecast. - [`sql.stats.forecasts.min_goodness_of_fit`]({% link v24.1/cluster-settings.md %}#setting-sql-stats-forecasts-min-goodness-of-fit) is the minimum R² (goodness of fit) measurement required from all predictive models to use a forecast. - [`sql.stats.forecasts.max_decrease`]({% link v24.1/cluster-settings.md %}#setting-sql-stats-forecasts-max-decrease) is the most a prediction can decrease, expressed as the minimum ratio of the prediction to the lowest prior observation. [#122459][#122459] diff --git a/src/current/_includes/releases/v24.1/v24.1.0.md b/src/current/_includes/releases/v24.1/v24.1.0.md index 5c9e947b332..b7931e5593d 100644 --- a/src/current/_includes/releases/v24.1/v24.1.0.md +++ b/src/current/_includes/releases/v24.1/v24.1.0.md @@ -1,6 +1,6 @@ ## v24.1.0 -Release Date: TBD TBD, 2024 +Release Date: May 20, 2024 With the release of CockroachDB v24.1, we've added new capabilities to help you migrate, build, and operate more efficiently. Refer to our summary of the most significant user-facing changes under [Feature Highlights](#v24-1-0-feature-highlights). @@ -11,12 +11,14 @@ With the release of CockroachDB v24.1, we've added new capabilities to help you This section summarizes the most significant user-facing changes in v24.1.0 and other features recently made available to CockroachDB users across versions. For a complete list of features and changes in v24.1, including bug fixes and performance improvements, refer to the [release notes]({% link releases/index.md %}#testing-releases) for previous v24.1 testing releases. You can also search the docs for sections labeled [New in v24.1](https://www.cockroachlabs.com/docs/search?query=new+in+v24.1). - **Feature categories** - - [Observability](#v24-1-0-observability) + - [Performance](#v24-1-0-performance) + - [Change data capture](#v24-1-0-change-data-capture) + - [Recovery](#v24-1-0-recovery) + - [Security](#v24-1-0-security) - [Migrations](#v24-1-0-migrations) - - [Security and compliance](#v24-1-0-security-and-compliance) - - [Disaster recovery](#v24-1-0-disaster-recovery) - - [Deployment and operations](#v24-1-0-deployment-and-operations) - [SQL](#v24-1-0-sql) + - [Operations](#v24-1-0-operations) + - [Observability](#v24-1-0-observability) - **Additional information** - [Backward-incompatible changes](#v24-1-0-backward-incompatible-changes) - [Deprecations](#v24-1-0-deprecations) @@ -29,7 +31,7 @@ In CockroachDB Self-Hosted, all available features are free to use unless their
-
-
-
-
-
+
+
+
+
+
+
-
@@ -234,26 +468,45 @@ In CockroachDB Self-Hosted, all available features are free to use unless their
-
Observability
+Performance
|
- TBD -TBD - |
- TBD | -{% include icon-yes.html %} | -{% include icon-yes.html %} | -{% include icon-no.html %} | -
|
+ WAL (Write-Ahead Logging) Failover +When a CockroachDB node is configured to run with multiple stores, you can mitigate some effects of disk stalls by configuring the node to failover the store's write-ahead log (WAL) to another store's data directory. +This feature is in preview. + |
+ 24.1 | +{% include icon-yes.html %} | +{% include icon-no.html %} | +{% include icon-no.html %} | +
|
+ Autocommit DDL +In 24.1 we have added the |
+ 24.1 | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +
|
+ Statistics for virtual computed columns +CockroachDB now collects statistics for virtual computed columns, making it possible to optimize query plans for tables that have these columns. Customers will see better query performance for queries that involve virtual computed columns. +For more information, you can also view a video demo. + |
+ 24.1 | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +
|
+ Write Amplification Improvements +A number of Pebble improvements are now on by default, leading to a reduction in write amplification. Of particular note, a vast majority of range snapshots previously ingested into L0 are now ingested directly into L6 in the LSM, reducing write and read amplification with positive effects to node stability and SQL performance. + |
+ 24.1 | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +
|
+ Increased SLA for multi-region Dedicated clusters +CockroachDB Dedicated multi-region clusters now offer a five-nines (99.999%) availability target. + |
+ All* | +{% include icon-no.html %} | +{% include icon-yes.html %} | +{% include icon-no.html %} | +
Migrations
- +Change Data Capture
|
- TBD -TBD - |
- TBD | -{% include icon-yes.html %} | -{% include icon-yes.html %} | -{% include icon-yes.html %} | -
|
+ Improved performance for changefeeds at scale +Mux rangefeed is a subsystem that improves the performance of rangefeeds with scale. In v24.1, Mux rangefeeds are enabled by default. + |
+ 24.1 | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +
|
+ Emit changefeed messages to an Azure Event Hub +Changefeeds can emit messages to an Azure Event Hub, which is compatible with Apache Kafka. You can use the Kafka changefeed and sink configuration options. + |
+ 24.1 | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +
|
+ Google Cloud Pub/Sub sink for changefeeds is now generally available +The changefeed sink for Google Cloud Pub/Sub is GA. + |
+ 24.1 | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +
|
+ Filter changefeed messages based on cluster, SQL session, or row-level TTL jobs +Use the |
+ 24.1 | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +
|
+ Set Kafka quotas per changefeed +For Kafka sinks, implement a resource usage limit per changefeed by setting a client ID and quota in your Kafka server's configuration. + |
+ 24.1 | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +
Disaster recovery
- +Recovery
|
- TBD -TBD. +Physical Cluster Replication is now generally available +Physical Cluster Replication is an asynchonous replication feature that allows your cluster to recover from full-cluster failure with a low RPO and RTO. In 24.1, it is a GA feature, requiring an Enterprise license, and only available for self-hosted CockroachDB deployments. For more information, refer to the Physical Cluster Replication overview and a blog post about Physical Cluster Replication. |
- TBD | +24.1 | {% include icon-yes.html %} | {% include icon-no.html %} | {% include icon-no.html %} | -
Security and compliance
- +Security
|
- TBD -TBD +Private connectivity in CockroachDB Dedicated with Google Cloud Private Service Connect +Private connectivity using Private Service Connect is now available in preview for CockroachDB Dedicated clusters on GCP. For more information, refer to Establish Private Connectivity and a video about how to set it up. |
- TBD | +All* | {% include icon-no.html %} | {% include icon-yes.html %} | {% include icon-no.html %} | -
|
+ Support for matching against SUBJECT +When using certificate based authentication, CockroachDB now supports mapping of SQL user roles to values in the Subject field of the X.509 certificate used for TLS authentication. +Subject mapping is useful if: +
|
+ 24.1 | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +{% include icon-no.html %} | +
Deployment and operations
+Migrations
+| Feature | +Availability | +|||
|---|---|---|---|---|
| Ver. | +Self-Hosted | +Dedicated | +Serverless | +|
|
+
+ MOLT Fetch is GA. MOLT Fetch automatically imports data from a PostgreSQL, MySQL, or CockroachDB source to a target CockroachDB database. If the fetch process encounters an error, data import can be continued from the point where it was interrupted. MOLT Fetch can be configured to replicate ongoing changes on the source database to CockroachDB following the initial data load. + |
+ All* | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +
SQL
|
+ Stored Procedures and UDF Improvements +Support for PL/pgSQL, user-defined functions, and stored procedures is now expanded: +
For more information, you can also view a video demo. + |
+ 24.1 | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +
|
+ Read Committed Isolation in GA +Read Committed isolation is now GA. Under For more information, you can also view a video about using Read Committed isolation and the blog post Isolation levels without the anomaly table. + |
+ 24.1 | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +
|
+ SELECT FOR SHARE updates +A shared lock that is acquired explicitly using |
+ 24.1 | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +
|
+ SHOW DEFAULT SESSION VARIABLES FOR ROLE +Previously, a database administrator could discover which session variables were set for a role only by switching into to that user or querying a system table. The new command |
+ 24.1 | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +
Operations
+| Feature | +Availability | +||||
|---|---|---|---|---|---|
| Ver. | +Self-Hosted | +Dedicated | +Serverless | +||
|
- TBD -TBD +New Regions for CockroachDB Dedicated +Cockroach Cloud is now available in additional Azure regions. This expansion enhances our global reach and offers more options for deploying resilient, distributed databases with low latency and high availability. +With these new regions, you can now select a geographical location that best fits your needs, reducing data latency and ensuring compliance with regional data residency regulations. Deploying in a region close to your service or customers means improved performance and a better user experience. |
- TBD | +All* | {% include icon-no.html %} | {% include icon-yes.html %} | {% include icon-no.html %} | -
|
+ Organize Cloud clusters using Folders +Folders help users organize and manage access for Clusters by projects, teams or business units. This feature is now in public preview and available to all Cloud organizations. For additional details, refer to the release note. + |
+ All* | +{% include icon-no.html %} | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +|
|
+ CockroachDB Cloud Terraform provider IAM enhancements +The CockroachDB Cloud Terraform provider has new resources for User Role Grant and Service Accounts, and a new data source for Folders. These new features improve customers’ ability to manage secure access to data, API, and infrastructure on CockroachDB Cloud. + |
+ All* | +{% include icon-no.html %} | +{% include icon-yes.html %} | +{% include icon-yes.html %} | +|
SQL
- +Observability
|
- TBD -TBD +CockroachDB Dedicated supports a Prometheus-compatibile metric scrape endpoint. +The Prometheus-compatibile metric scrape endpoint for CockroachDB Dedicated is generally available. |
- TBD | +All* | {% include icon-no.html %} | {% include icon-yes.html %} | {% include icon-no.html %} | -
|
+ Export CockroachDB Dedicated metrics to Azure Monitor +CockroachDB Dedicated supports exporting metrics to Azure Monitor in limited access. Contact your Cockroach Labs account team for access. + |
+ All* | +{% include icon-no.html %} | +{% include icon-yes.html %} | +{% include icon-no.html %} | +|
|
+ Export CockroachDB Dedicated logs to Azure Monitor +CockroachDB Dedicated supports exporting logs to Azure Monitor in limited access. Contact your Cockroach Labs account team for access. + |
+ All* | +{% include icon-no.html %} | +{% include icon-yes.html %} | +{% include icon-no.html %} | +
Backward-incompatible changes
Before [upgrading to CockroachDB v24.1]({% link v24.1/upgrade-cockroach-version.md %}), be sure to review the following backward-incompatible changes, as well as [key cluster setting changes](#v24-1-0-cluster-settings), and adjust your deployment as necessary. -- TBD +- [`AS OF SYSTEM TIME`]({% link v24.1/as-of-system-time.md %}) queries can no longer use a timestamp followed by a question mark to signify a future-time value. This was an undocumented syntax. [#116830](https://github.com/cockroachdb/cockroach/pull/116830) +- The [`READ COMMITTED`]({% link v24.1/read-committed.md %}) isolation level now requires the cluster to have a valid [enterprise license](https://cockroachlabs.com/docs/v24.1/licensing-faqs#obtain-a-license). Otherwise, transactions which are configured to run as `READ COMMITTED` will be upgraded to [`SERIALIZABLE`]({% link v24.1/demo-serializable.md %}), as described in the next note. [#120154](https://github.com/cockroachdb/cockroach/pull/120154) +- The `sql.txn.read_committed_isolation.enabled` [cluster setting]({% link v24.1/cluster-settings.md %}) is now `true` by default. As a result for enterprise users, [`READ COMMITTED`]({% link v24.1/read-committed.md %}) transactions are **not** automatically upgraded to [`SERIALIZABLE`]({% link v24.1/demo-serializable.md %}), and will run as `READ COMMITTED` by default. On v23.2, refer to the [Upgrades of SQL Transaction Isolation Level](https://www.cockroachlabs.com/docs/v24.1/ui-sql-dashboard#upgrades-of-sql-transaction-isolation-level) graph in the DB Console to check whether any transaction is being upgraded from a weaker isolation level to `SERIALIZABLE`, and could therefore run differently on v24.1. [#118479](https://github.com/cockroachdb/cockroach/pull/118479)Key Cluster Setting Changes
The following changes should be reviewed prior to upgrading. Default cluster settings will be used unless you have manually set a value for a setting. This can be confirmed by checking the `system.settings` table (`select * from system.settings`) to view the non-default settings. -- TBD +- `sql.txn.read_committed_isolation.enabled` is now `true` by default. When set to `true`, transactions use the `READ COMMITTED` isolation level if specified by `BEGIN`/`SET` commands. + - If the cluster setting is `false`, as was the default in v23.2, such `READ COMMITTED` transactions will instead run as `SERIALIZABLE`. + - To check whether any transactions are being upgraded to `SERIALIZABLE`, see the [**Upgrades of SQL Transaction Isolation Level**](https://www.cockroachlabs.com/docs/v24.1/ui-sql-dashboard#upgrades-of-sql-transaction-isolation-level) graph in the DB Console." +- The `changefeed.balance_range_distribution.enable` [cluster setting]({% link v24.1/cluster-settings.md %}) is now deprecated. Instead, use the new cluster setting `changefeed.default_range_distribution_strategy`. `changefeed.default_range_distribution_strategy='balanced_simple'` has the same effect as setting `changefeed.balance_range_distribution.enable=true`. It does not require `initial_scan='only'`, which was required by the old setting. [#115166][#115166] +- Added the [cluster setting]({% link v24.1/cluster-settings.md %}) `security.client_cert.subject_required.enabled` which enforces a mandatory requirement for the client certificate's role subject to be set. The subject can be defined through either the subject role option or by specifying the `root-cert-distinguished-name` and `node-cert-distinguished-name` properties. This setting applies to both RPC access and login via authCert. [#122368][#122368] +- The [cluster setting]({% link v24.1/cluster-settings.md %}) `sql.contention.record_serialization_conflicts.enabled` is now `on` by default. As a result, any [`40001` error]({% link v24.1/transaction-retry-error-reference.md %}) that contains conflicting transaction information will be recorded by the contention registry, improving the ability to troubleshoot. For more information, refer to the [Insights page]({% link v24.1/ui-insights-page.md %}) documentation. [#116664][#116664] +- The new [cluster setting]({% link v24.1/cluster-settings.md %}) `storage.sstable.compression_algorithm` configures the compression algorithm used when compressing sstable blocks. Supported values are: "snappy" and "zstd" [snappy = `1`, zstd = `2`]. Changing the default of snappy to zstd can result in substantial performance improvement, however, the effects this change may be highly dependent on the workload and data, so experimentation is recommended before enabling zstd in production environments. +- The new setting `storage.wal_failover.unhealthy_op_threshold` allows you to set the latency threshold at which a WAL (Write-Ahead Logging) write is considered unhealthy. When exceeded, the node will attempt to write WAL entries to a secondary store's volume. For more information, refer toDeprecations
-{% comment %}TODO: Intro para? Each sibling section has one.{% endcomment %} - -- TBD +- `changefeed.balance_range_distribution.enable` is now deprecated. Instead, use the new [cluster setting]({% link v24.1/cluster-settings.md %}) `changefeed.default_range_distribution_strategy`. `changefeed.default_range_distribution_strategy='balanced_simple'` has the same effect as setting `changefeed.balance_range_distribution.enable=true`. It does not require `initial_scan='only'`, which was required by the old setting. [#115166][#115166] +- The `cockroach connect` command has been removed. This command was [deprecated]({% link releases/v23.2.md %}#v23-2-0-deprecations) in CockroachDB v23.2. [#113893][#113893]Known limitations
@@ -263,7 +516,22 @@ For information about new and unresolved limitations in CockroachDB v24.1, with Resource | Topic | Description ---------------------+--------------------------------------------+------------- -Cockroach University | [Example link](https://example.com/course1)| Summary here -Cockroach University | [Example link](https://example.com/course2)| Summary here -Docs | [Example link](https://example.com/doc1) | Summary here -Docs | [Example link](https://example.com/doc2) | Summary here +Cockroach University | [Introduction to Distributed SQL and CockroachDB](https://university.cockroachlabs.com/courses/course-v1:crl+intro-to-distributed-sql-and-cockroachdb+self-paced/about) | This course introduces the core concepts behind distributed SQL databases and describes how CockroachDB fits into this landscape. You will learn what differentiates CockroachDB from both legacy SQL and NoSQL databases and how CockroachDB ensures consistent transactions without sacrificing scale and resiliency. You'll learn about CockroachDB's seamless horizontal scalability, distributed transactions with strict ACID guarantees, and high availability and resilience. +Cockroach University | [Practical First Steps with CockroachDB](https://university.cockroachlabs.com/courses/course-v1:crl+practical-first-steps-with-crdb+self-paced/about) | This course will give you the tools you need to get started with CockroachDB. During the course, you will learn how to spin up a cluster, use the Admin UI to monitor cluster activity, and use SQL shell to solve a set of hands-on exercises. +Cockroach University | [Enterprise Application Development with CockroachDB](https://university.cockroachlabs.com/courses/course-v1:crl+client-side-txn-handling+self-paced/about) | This course is the first in a series designed to equip you with best practices for mastering application-level (client-side) transaction management in CockroachDB. We'll dive deep on common differences between CockroachDB and legacy SQL databases and help you sidestep challenges you might encounter when migrating to CockroachDB from Oracle, PostgreSQL, and MySQL. +Cockroach University | [Building a Highly Resilient Multi-region Database using CockroachDB](https://university.cockroachlabs.com/courses/course-v1:crl+intro-to-resilience-in-multi-region+self-paced/about) | This course is part of a series introducing solutions to running low-latency, highly resilient applications for data-intensive workloads on CockroachDB. In this course we focus on surviving large-scale infrastructure failures like losing an entire cloud region without losing data during recovery. We’ll show you how to use CockroachDB survival goals in a multi-region cluster to implement a highly resilient database that survives node or network failures across multiple regions with zero data loss. +Docs | [Migration Overview](https://www.cockroachlabs.com/docs/v24.1/migration-overview) | This page summarizes the steps of migrating a database to CockroachDB, which include testing and updating your schema to work with CockroachDB, moving your data into CockroachDB, and testing and updating your application. +Docs | [Architecture Overview](https://www.cockroachlabs.com/docs/v24.1/architecture/overview) | This page provides a starting point for understanding the architecture and design choices that enable CockroachDB's scalability and consistency capabilities. +Docs | [SQL Feature Support](https://www.cockroachlabs.com/docs/v24.1/sql-feature-support) | The page summarizes the standard SQL features CockroachDB supports as well as common extensions to the standard. +Docs | [Change Data Capture Overview](https://www.cockroachlabs.com/docs/v24.1/change-data-capture-overview) | This page summarizes CockroachDB's data streaming capabilities. Change data capture (CDC) provides efficient, distributed, row-level changefeeds into a configurable sink for downstream processing such as reporting, caching, or full-text indexing. +Docs | [Backup Architecture](https://www.cockroachlabs.com/docs/v24.1/backup-architecture) | This page describes the backup job workflow with a high-level overview, diagrams, and more details on each phase of the job. + +[#115166]: https://github.com/cockroachdb/cockroach/pull/115166 +[#113893]: https://github.com/cockroachdb/cockroach/pull/113893 +[#115166]: https://github.com/cockroachdb/cockroach/pull/115166 +[#122368]: https://github.com/cockroachdb/cockroach/pull/122368 +[#116664]: https://github.com/cockroachdb/cockroach/pull/116664 +[#120509]: https://github.com/cockroachdb/cockroach/pull/120509 +[#118781]: https://github.com/cockroachdb/cockroach/pull/118781 +[#117729]: https://github.com/cockroachdb/cockroach/pull/117729 +[#120255]: https://github.com/cockroachdb/cockroach/pull/120255 diff --git a/src/current/_includes/sidebar-data-v23.2.json b/src/current/_includes/sidebar-data-v23.2.json index f2aedcaf392..85e0a95ba8a 100644 --- a/src/current/_includes/sidebar-data-v23.2.json +++ b/src/current/_includes/sidebar-data-v23.2.json @@ -22,5 +22,6 @@ {% include_cached v23.2/sidebar-data/sql.json %}, {% include_cached v23.2/sidebar-data/reference.json %}, {% include_cached v23.2/sidebar-data/faqs.json %}, + {% include_cached v23.2/sidebar-data/releases.json %}, {% include_cached sidebar-data-cockroach-university.json %} ] diff --git a/src/current/_includes/sidebar-releases.json b/src/current/_includes/sidebar-releases.json index 26a6c95da8a..351e4192850 100644 --- a/src/current/_includes/sidebar-releases.json +++ b/src/current/_includes/sidebar-releases.json @@ -7,7 +7,7 @@ {% assign v_test = site.data.versions | where_exp: "v_test", "v_test.release_date == 'N/A'" | sort: "release_date" | last | map: "major_version" %} {% comment %} v_test iterates through the list of major versions in _data/versions.csv and returns the single latest testing version (if the release date is in the future or not otherwise specified). It's possible there is no testing release (in between GA of a version and the first alpha of the following version). {% endcomment %} { - "title": "CockroachDB Releases", + "title": "CockroachDB", "items": [ { "title": "Latest Production Release", @@ -20,7 +20,7 @@ "urls": [ "/releases/{{ site.versions["dev"] }}.html" ] - },{% endunless %} + }{% endunless %}, { "title": "All Releases", "urls": [ @@ -35,12 +35,15 @@ ] }, {% endfor %} + {% if v_test[0] %} {% comment %} check if a testing version is available {% endcomment %} { - "title": "Staged Release Process", + "title": "Latest Testing Release", "urls": [ - "/releases/staged-release-process.html" + {% comment %} check if a testing version is available and pull the latest testing version {% endcomment %} + "/releases/{{ v_test[0] }}.html" ] }, + {% endif %} { "title": "Release Support Policy", "urls": [ @@ -55,25 +58,25 @@ } ] }, -{% assign advisories = site.pages | where_exp: "advisories", "advisories.path contains 'advisories'" | where_exp: "advisories", "advisories.index != 'true'" %} -{ - "title": "Technical Advisories", - "urls": [ - "/advisories/index.html" - {% for x in advisories %} - ,"{{ x.url }}" - {% endfor %} - ] -}, { - "title": "Cloud Releases", + "title": "CockroachDB Cloud", "urls": [ "/releases/cloud.html" ] }, { - "title": "Kubernetes Operator", + "title": "CockroachDB Kubernetes Operator", "urls": [ "/releases/kubernetes-operator.html" ] +}, +{% assign advisories = site.pages | where_exp: "advisories", "advisories.path contains 'advisories'" | where_exp: "advisories", "advisories.index != 'true'" %} +{ + "title": "Technical Advisories", + "urls": [ + "/advisories/index.html" + {% for x in advisories %} + ,"{{ x.url }}" + {% endfor %} + ] } diff --git a/src/current/_includes/v23.1/cdc/pubsub-performance-setting.md b/src/current/_includes/v23.1/cdc/pubsub-performance-setting.md index f6b55402846..86ebed03a76 100644 --- a/src/current/_includes/v23.1/cdc/pubsub-performance-setting.md +++ b/src/current/_includes/v23.1/cdc/pubsub-performance-setting.md @@ -1,3 +1,3 @@ {{site.data.alerts.callout_info}} -{% include_cached new-in.html version="v23.1" %} Enable the `changefeed.new_pubsub_sink_enabled` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) to improve the throughput of changefeeds emitting to {% if page.name == "changefeed-sinks.md" %} Pub/Sub sinks. {% else %} [Pub/Sub sinks]({% link {{ page.version.version }}/changefeed-sinks.md %}#google-cloud-pub-sub). {% endif %} +{% include_cached new-in.html version="v23.1" %} Enable the `changefeed.new_pubsub_sink_enabled` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) to improve the throughput of changefeeds emitting to [Pub/Sub sinks]({% link {{ page.version.version }}/changefeed-sinks.md %}#google-cloud-pub-sub). Enabling this setting also alters the message format to use capitalized top-level fields in changefeeds emitting JSON-encoded messages to a Pub/Sub sink. For more details, refer to the [Pub/Sub sink messages]({% link {{ page.version.version }}/changefeed-sinks.md %}#pub-sub-sink-messages) section. {{site.data.alerts.end}} \ No newline at end of file diff --git a/src/current/_includes/v23.1/sidebar-data/migrate.json b/src/current/_includes/v23.1/sidebar-data/migrate.json index 949b2754d81..92a72d90371 100644 --- a/src/current/_includes/v23.1/sidebar-data/migrate.json +++ b/src/current/_includes/v23.1/sidebar-data/migrate.json @@ -9,7 +9,7 @@ ] }, { - "title": "Migration Tools", + "title": "MOLT Tools", "items": [ { "title": "Schema Conversion Tool", @@ -18,7 +18,7 @@ ] }, { - "title": "MOLT Verify", + "title": "Verify", "urls": [ "/${VERSION}/molt-verify.html" ] @@ -28,7 +28,12 @@ "urls": [ "/${VERSION}/live-migration-service.html" ] - }, + } + ] + }, + { + "title": "Third-Party Migration Tools", + "items": [ { "title": "AWS DMS", "urls": [ @@ -120,6 +125,12 @@ "/${VERSION}/migrate-from-mysql.html" ] }, + { + "title": "Migrate from Oracle", + "urls": [ + "/${VERSION}/migrate-from-oracle.html" + ] + }, { "title": "Migration Strategy: Lift and Shift", "urls": [ diff --git a/src/current/_includes/v23.2/sidebar-data/migrate.json b/src/current/_includes/v23.2/sidebar-data/migrate.json index 9b87e0c2f31..5b6338a628b 100644 --- a/src/current/_includes/v23.2/sidebar-data/migrate.json +++ b/src/current/_includes/v23.2/sidebar-data/migrate.json @@ -9,7 +9,7 @@ ] }, { - "title": "Migration Tools", + "title": "MOLT Tools", "items": [ { "title": "Schema Conversion Tool", @@ -18,13 +18,13 @@ ] }, { - "title": "MOLT Fetch", + "title": "Fetch", "urls": [ "/${VERSION}/molt-fetch.html" ] }, { - "title": "MOLT Verify", + "title": "Verify", "urls": [ "/${VERSION}/molt-verify.html" ] @@ -34,7 +34,12 @@ "urls": [ "/${VERSION}/live-migration-service.html" ] - }, + } + ] + }, + { + "title": "Third-Party Migration Tools", + "items": [ { "title": "AWS DMS", "urls": [ @@ -126,6 +131,12 @@ "/${VERSION}/migrate-from-mysql.html" ] }, + { + "title": "Migrate from Oracle", + "urls": [ + "/${VERSION}/migrate-from-oracle.html" + ] + }, { "title": "Migration Strategy: Lift and Shift", "urls": [ diff --git a/src/current/_includes/v23.2/sidebar-data/releases.json b/src/current/_includes/v23.2/sidebar-data/releases.json index 8ee555fdc9a..fb49f7c9acc 100644 --- a/src/current/_includes/v23.2/sidebar-data/releases.json +++ b/src/current/_includes/v23.2/sidebar-data/releases.json @@ -1,5 +1,5 @@ { - "title": "Releases", + "title": "CockroachDB Releases", "is_top_level": true, "items": [ {% include_cached sidebar-all-releases.json %} diff --git a/src/current/_includes/v24.1/known-limitations/plpgsql-datatype-limitations.md b/src/current/_includes/v24.1/known-limitations/plpgsql-datatype-limitations.md deleted file mode 100644 index 229120607e2..00000000000 --- a/src/current/_includes/v24.1/known-limitations/plpgsql-datatype-limitations.md +++ /dev/null @@ -1,7 +0,0 @@ -- `RECORD` parameters and variables are not supported in [user-defined functions]({% link {{ page.version.version }}/user-defined-functions.md %}). [#105713](https://github.com/cockroachdb/cockroach/issues/105713) -- Variable shadowing (e.g., declaring a variable with the same name in an inner block) is not supported in PL/pgSQL. [#117508](https://github.com/cockroachdb/cockroach/issues/117508) -- Syntax for accessing members of composite types without parentheses is not supported. [#114687](https://github.com/cockroachdb/cockroach/issues/114687) -- `NOT NULL` variable declarations are not supported. [#105243](https://github.com/cockroachdb/cockroach/issues/105243) -- Cursors opened in PL/pgSQL execute their queries on opening, affecting performance and resource usage. [#111479](https://github.com/cockroachdb/cockroach/issues/111479) -- Cursors in PL/pgSQL cannot be declared with arguments. [#117746](https://github.com/cockroachdb/cockroach/issues/117746) -- `OPEN FOR EXECUTE` is not supported for opening cursors. [#117744](https://github.com/cockroachdb/cockroach/issues/117744) \ No newline at end of file diff --git a/src/current/_includes/v24.1/known-limitations/plpgsql-feature-limitations.md b/src/current/_includes/v24.1/known-limitations/plpgsql-feature-limitations.md deleted file mode 100644 index 953bb475587..00000000000 --- a/src/current/_includes/v24.1/known-limitations/plpgsql-feature-limitations.md +++ /dev/null @@ -1,9 +0,0 @@ -- PL/pgSQL arguments cannot be referenced with ordinals (e.g., `$1`, `$2`). [#114701](https://github.com/cockroachdb/cockroach/issues/114701) -- The following statements are not supported: - - `FOR` loops, including `FOR` cursor loops, `FOR` query loops, and `FOREACH` loops. [#105246](https://github.com/cockroachdb/cockroach/issues/105246) - - `RETURN NEXT` and `RETURN QUERY`. [#117744](https://github.com/cockroachdb/cockroach/issues/117744) - - `PERFORM`, `EXECUTE`, `GET DIAGNOSTICS`, and `CASE`. [#117744](https://github.com/cockroachdb/cockroach/issues/117744) -- PL/pgSQL exception blocks cannot catch [transaction retry errors]({% link {{ page.version.version }}/transaction-retry-error-reference.md %}). [#111446](https://github.com/cockroachdb/cockroach/issues/111446) -- `RAISE` statements cannot be annotated with names of schema objects related to the error (i.e., using `COLUMN`, `CONSTRAINT`, `DATATYPE`, `TABLE`, or `SCHEMA`). [#106237](https://github.com/cockroachdb/cockroach/issues/106237) -- `RAISE` statements message the client directly, and do not produce log output. [#117750](https://github.com/cockroachdb/cockroach/issues/117750) -- `ASSERT` debugging checks are not supported. [#117744](https://github.com/cockroachdb/cockroach/issues/117744) \ No newline at end of file diff --git a/src/current/_includes/v24.1/known-limitations/plpgsql-limitations.md b/src/current/_includes/v24.1/known-limitations/plpgsql-limitations.md new file mode 100644 index 00000000000..70b66dbd6a1 --- /dev/null +++ b/src/current/_includes/v24.1/known-limitations/plpgsql-limitations.md @@ -0,0 +1,20 @@ +{% if page.name != "known-limitations.md" # New limitations in v24.1 %} +- It is not possible to use a variable as a target more than once in the same `INTO` clause. For example, `SELECT 1, 2 INTO x, x;`. [#121605](https://github.com/cockroachdb/cockroach/issues/121605) +- PLpgSQL variable declarations cannot inherit the type of a table row or column using `%TYPE` or `%ROWTYPE` syntax. [#114676](https://github.com/cockroachdb/cockroach/issues/114676) +{% endif %} +- PL/pgSQL arguments cannot be referenced with ordinals (e.g., `$1`, `$2`). [#114701](https://github.com/cockroachdb/cockroach/issues/114701) +- The following statements are not supported: + - `FOR` loops, including `FOR` cursor loops, `FOR` query loops, and `FOREACH` loops. [#105246](https://github.com/cockroachdb/cockroach/issues/105246) + - `RETURN NEXT` and `RETURN QUERY`. [#117744](https://github.com/cockroachdb/cockroach/issues/117744) + - `PERFORM`, `EXECUTE`, `GET DIAGNOSTICS`, and `CASE`. [#117744](https://github.com/cockroachdb/cockroach/issues/117744) +- PL/pgSQL exception blocks cannot catch [transaction retry errors]({% link {{ page.version.version }}/transaction-retry-error-reference.md %}). [#111446](https://github.com/cockroachdb/cockroach/issues/111446) +- `RAISE` statements cannot be annotated with names of schema objects related to the error (i.e., using `COLUMN`, `CONSTRAINT`, `DATATYPE`, `TABLE`, or `SCHEMA`). [#106237](https://github.com/cockroachdb/cockroach/issues/106237) +- `RAISE` statements message the client directly, and do not produce log output. [#117750](https://github.com/cockroachdb/cockroach/issues/117750) +- `ASSERT` debugging checks are not supported. [#117744](https://github.com/cockroachdb/cockroach/issues/117744) +- `RECORD` parameters and variables are not supported in [user-defined functions]({% link {{ page.version.version }}/user-defined-functions.md %}). [#105713](https://github.com/cockroachdb/cockroach/issues/105713) +- Variable shadowing (e.g., declaring a variable with the same name in an inner block) is not supported in PL/pgSQL. [#117508](https://github.com/cockroachdb/cockroach/issues/117508) +- Syntax for accessing members of composite types without parentheses is not supported. [#114687](https://github.com/cockroachdb/cockroach/issues/114687) +- `NOT NULL` variable declarations are not supported. [#105243](https://github.com/cockroachdb/cockroach/issues/105243) +- Cursors opened in PL/pgSQL execute their queries on opening, affecting performance and resource usage. [#111479](https://github.com/cockroachdb/cockroach/issues/111479) +- Cursors in PL/pgSQL cannot be declared with arguments. [#117746](https://github.com/cockroachdb/cockroach/issues/117746) +- `OPEN FOR EXECUTE` is not supported for opening cursors. [#117744](https://github.com/cockroachdb/cockroach/issues/117744) \ No newline at end of file diff --git a/src/current/_includes/v24.1/known-limitations/read-committed-limitations.md b/src/current/_includes/v24.1/known-limitations/read-committed-limitations.md index e9d4c899293..5087e29ac00 100644 --- a/src/current/_includes/v24.1/known-limitations/read-committed-limitations.md +++ b/src/current/_includes/v24.1/known-limitations/read-committed-limitations.md @@ -3,4 +3,5 @@ - Multi-column-family checks during updates are not supported under `READ COMMITTED` isolation. [#112488](https://github.com/cockroachdb/cockroach/issues/112488) - Because locks acquired by [foreign key]({% link {{ page.version.version }}/foreign-key.md %}) checks, [`SELECT FOR UPDATE`]({% link {{ page.version.version }}/select-for-update.md %}), and [`SELECT FOR SHARE`]({% link {{ page.version.version }}/select-for-update.md %}) are fully replicated under `READ COMMITTED` isolation, some queries experience a delay for Raft replication. - [Foreign key]({% link {{ page.version.version }}/foreign-key.md %}) checks are not performed in parallel under `READ COMMITTED` isolation. -- [`SELECT FOR UPDATE` and `SELECT FOR SHARE`]({% link {{ page.version.version }}/select-for-update.md %}) statements are less optimized under `READ COMMITTED` isolation than under `SERIALIZABLE` isolation. Under `READ COMMITTED` isolation, `SELECT FOR UPDATE` and `SELECT FOR SHARE` usually perform an extra lookup join for every locked table when compared to the same queries under `SERIALIZABLE`. In addition, some optimization steps (such as de-correlation of correlated [subqueries]({% link {{ page.version.version }}/subqueries.md %})) are not currently performed on these queries. \ No newline at end of file +- [`SELECT FOR UPDATE` and `SELECT FOR SHARE`]({% link {{ page.version.version }}/select-for-update.md %}) statements are less optimized under `READ COMMITTED` isolation than under `SERIALIZABLE` isolation. Under `READ COMMITTED` isolation, `SELECT FOR UPDATE` and `SELECT FOR SHARE` usually perform an extra lookup join for every locked table when compared to the same queries under `SERIALIZABLE`. In addition, some optimization steps (such as de-correlation of correlated [subqueries]({% link {{ page.version.version }}/subqueries.md %})) are not currently performed on these queries. +- Regardless of isolation level, [`SELECT FOR UPDATE` and `SELECT FOR SHARE`]({% link {{ page.version.version }}/select-for-update.md %}) statements in CockroachDB do not prevent insertion of new rows matching the search condition (i.e., [phantom reads]({% link {{ page.version.version }}/read-committed.md %}#non-repeatable-reads-and-phantom-reads)). This matches PostgreSQL behavior at all isolation levels. [#120673](https://github.com/cockroachdb/cockroach/issues/120673) \ No newline at end of file diff --git a/src/current/_includes/v24.1/known-limitations/routine-limitations.md b/src/current/_includes/v24.1/known-limitations/routine-limitations.md new file mode 100644 index 00000000000..f122a76f25f --- /dev/null +++ b/src/current/_includes/v24.1/known-limitations/routine-limitations.md @@ -0,0 +1,7 @@ +{% if page.name != "known-limitations.md" # New limitations in v24.1 %} +- Routines cannot be invoked with named arguments, e.g., `SELECT foo(a => 1, b => 2);` or `SELECT foo(b := 1, a := 2);`. [#122264](https://github.com/cockroachdb/cockroach/issues/122264) +- Routines cannot be created if they reference temporary tables. [#121375](https://github.com/cockroachdb/cockroach/issues/121375) +- Routines cannot be created with unnamed `INOUT` parameters. For example, `CREATE PROCEDURE p(INOUT INT) AS $$ BEGIN NULL; END; $$ LANGUAGE PLpgSQL;`. [#121251](https://github.com/cockroachdb/cockroach/issues/121251) +- Routines cannot be created if they return fewer columns than declared. For example, `CREATE FUNCTION f(OUT sum INT, INOUT a INT, INOUT b INT) LANGUAGE SQL AS $$ SELECT (a + b, b); $$;`. [#121247](https://github.com/cockroachdb/cockroach/issues/121247) +{% endif %} +- DDL statements (e.g., `CREATE TABLE`, `CREATE INDEX`) are not allowed within UDFs or stored procedures. [#110080](https://github.com/cockroachdb/cockroach/issues/110080) \ No newline at end of file diff --git a/src/current/_includes/v24.1/known-limitations/stored-proc-limitations.md b/src/current/_includes/v24.1/known-limitations/stored-proc-limitations.md new file mode 100644 index 00000000000..ec0fa31c9b8 --- /dev/null +++ b/src/current/_includes/v24.1/known-limitations/stored-proc-limitations.md @@ -0,0 +1,2 @@ +{% if page.name != "known-limitations.md" # New limitations in v24.1 %} +{% endif %} \ No newline at end of file diff --git a/src/current/_includes/v24.1/known-limitations/udf-stored-proc-limitations.md b/src/current/_includes/v24.1/known-limitations/udf-limitations.md similarity index 75% rename from src/current/_includes/v24.1/known-limitations/udf-stored-proc-limitations.md rename to src/current/_includes/v24.1/known-limitations/udf-limitations.md index 0ff2a43cd90..7064acb8775 100644 --- a/src/current/_includes/v24.1/known-limitations/udf-stored-proc-limitations.md +++ b/src/current/_includes/v24.1/known-limitations/udf-limitations.md @@ -1,13 +1,9 @@ -{% if page.name != "stored-procedures.md" %} +{% if page.name != "known-limitations.md" # New limitations in v24.1 %} +- A `RECORD`-returning UDF cannot be created without a `RETURN` statement in the root block, which would restrict the wildcard type to a concrete one. [#122945](https://github.com/cockroachdb/cockroach/issues/122945) +{% endif %} - User-defined functions are not currently supported in: - Expressions (column, index, constraint) in tables. [#87699](https://github.com/cockroachdb/cockroach/issues/87699) - Views. [#87699](https://github.com/cockroachdb/cockroach/issues/87699) - User-defined functions cannot call themselves recursively. [#93049](https://github.com/cockroachdb/cockroach/issues/93049) - [Common table expressions]({% link {{ page.version.version }}/common-table-expressions.md %}) (CTE), recursive or non-recursive, are not supported in [user-defined functions]({% link {{ page.version.version }}/user-defined-functions.md %}) (UDF). That is, you cannot use a `WITH` clause in the body of a UDF. [#92961](https://github.com/cockroachdb/cockroach/issues/92961) -- The `setval` function cannot be resolved when used inside UDF bodies. [#110860](https://github.com/cockroachdb/cockroach/issues/110860) -{% endif %} - -{% if page.name != "user-defined-functions.md" %} -{% endif %} - -- DDL statements (e.g., `CREATE TABLE`, `CREATE INDEX`) are not allowed within UDFs or stored procedures. [#110080](https://github.com/cockroachdb/cockroach/issues/110080) +- The `setval` function cannot be resolved when used inside UDF bodies. [#110860](https://github.com/cockroachdb/cockroach/issues/110860) \ No newline at end of file diff --git a/src/current/_includes/v24.1/misc/force-index-selection.md b/src/current/_includes/v24.1/misc/force-index-selection.md index beeb1bcceac..a2f70c98ee6 100644 --- a/src/current/_includes/v24.1/misc/force-index-selection.md +++ b/src/current/_includes/v24.1/misc/force-index-selection.md @@ -6,34 +6,34 @@ Index selection can impact [performance]({% link {{ page.version.version }}/perf ##### Force index scan -The syntax to force a scan of a specific index is: +To force a scan of a specific index: {% include_cached copy-clipboard.html %} ~~~ sql -> SELECT * FROM table@my_idx; +SELECT * FROM table@my_idx; ~~~ This is equivalent to the longer expression: {% include_cached copy-clipboard.html %} ~~~ sql -> SELECT * FROM table@{FORCE_INDEX=my_idx}; +SELECT * FROM table@{FORCE_INDEX=my_idx}; ~~~ ##### Force reverse scan -The syntax to force a reverse scan of a specific index is: +To force a reverse scan of a specific index: {% include_cached copy-clipboard.html %} ~~~ sql -> SELECT * FROM table@{FORCE_INDEX=my_idx,DESC}; +SELECT * FROM table@{FORCE_INDEX=my_idx,DESC}; ~~~ -Forcing a reverse scan is sometimes useful during [performance tuning]({% link {{ page.version.version }}/performance-best-practices-overview.md %}). For reference, the full syntax for choosing an index and its scan direction is +Forcing a reverse scan can help with [performance tuning]({% link {{ page.version.version }}/performance-best-practices-overview.md %}). To choose an index and its scan direction: {% include_cached copy-clipboard.html %} ~~~ sql -SELECT * FROM table@{FORCE_INDEX=idx[,DIRECTION]} +SELECT * FROM table@{FORCE_INDEX=idx[,DIRECTION]}; ~~~ where the optional `DIRECTION` is either `ASC` (ascending) or `DESC` (descending). @@ -44,14 +44,14 @@ You can verify that the optimizer is choosing your desired scan direction using {% include_cached copy-clipboard.html %} ~~~ sql -> CREATE TABLE kv (K INT PRIMARY KEY, v INT); +CREATE TABLE kv (K INT PRIMARY KEY, v INT); ~~~ you can check the scan direction with: {% include_cached copy-clipboard.html %} ~~~ sql -> EXPLAIN (opt) SELECT * FROM users@{FORCE_INDEX=primary,DESC}; +EXPLAIN (opt) SELECT * FROM users@{FORCE_INDEX=primary,DESC}; ~~~ ~~~ @@ -62,6 +62,21 @@ you can check the scan direction with: (2 rows) ~~~ +#### Force inverted index scan + +To force a scan of any [inverted index]({% link {{ page.version.version }}/inverted-indexes.md %}) of the hinted table: + +{% include_cached copy-clipboard.html %} +~~~ sql +SELECT * FROM table@{FORCE_INVERTED_INDEX}; +~~~ + +The `FORCE_INVERTED_INDEX` hint does not allow specifying an inverted index. If no query plan can be generated, the query will result in the error: + +~~~ +ERROR: could not produce a query plan conforming to the FORCE_INVERTED_INDEX hint +~~~ + ##### Force partial index scan To force a [partial index scan]({% link {{ page.version.version }}/partial-indexes.md %}), your statement must have a `WHERE` clause that implies the partial index filter. diff --git a/src/current/_includes/v24.1/misc/session-vars.md b/src/current/_includes/v24.1/misc/session-vars.md index a8be2430bcd..7b0e8441a9d 100644 --- a/src/current/_includes/v24.1/misc/session-vars.md +++ b/src/current/_includes/v24.1/misc/session-vars.md @@ -51,6 +51,7 @@ | `optimizer_use_lock_op_for_serializable` | If `on`, the optimizer uses a `Lock` operator to construct query plans for `SELECT` statements using the [`FOR UPDATE` and `FOR SHARE`]({% link {{ page.version.version }}/select-for-update.md %}) clauses. This setting only affects `SERIALIZABLE` transactions. `READ COMMITTED` transactions are evaluated with the `Lock` operator regardless of the setting. | `off` | Yes | Yes | | `optimizer_use_multicol_stats` | If `on`, the optimizer uses collected multi-column statistics for cardinality estimation. | `on` | No | Yes | | `optimizer_use_not_visible_indexes` | If `on`, the optimizer uses not visible indexes for planning. | `off` | No | Yes | +| `optimizer_use_virtual_computed_column_stats` | If `on`, the optimizer uses table statistics on [virtual computed columns]({% link {{ page.version.version }}/computed-columns.md %}#virtual-computed-columns). | `on` | Yes | Yes | `plpgsql_use_strict_into` | If `on`, PL/pgSQL [`SELECT ... INTO` and `RETURNING ... INTO` statements]({% link {{ page.version.version }}/plpgsql.md %}#assign-a-result-to-a-variable) behave as though the `STRICT` option is specified. This causes the SQL statement to error if it does not return exactly one row. | `off` | Yes | Yes | | `pg_trgm.similarity_threshold` | The threshold above which a [`%`]({% link {{ page.version.version }}/functions-and-operators.md %}#operators) string comparison returns `true`. The value must be between `0` and `1`. For more information, see [Trigram Indexes]({% link {{ page.version.version }}/trigram-indexes.md %}). | `0.3` | Yes | Yes | | `prefer_lookup_joins_for_fks` | If `on`, the optimizer prefers [`lookup joins`]({% link {{ page.version.version }}/joins.md %}#lookup-joins) to [`merge joins`]({% link {{ page.version.version }}/joins.md %}#merge-joins) when performing [`foreign key`]({% link {{ page.version.version }}/foreign-key.md %}) checks. | `off` | Yes | Yes | diff --git a/src/current/_includes/v24.1/physical-replication/fast-cutback-syntax.md b/src/current/_includes/v24.1/physical-replication/fast-cutback-syntax.md index 1c217be5af0..4b468429d92 100644 --- a/src/current/_includes/v24.1/physical-replication/fast-cutback-syntax.md +++ b/src/current/_includes/v24.1/physical-replication/fast-cutback-syntax.md @@ -2,7 +2,7 @@ {% include_cached copy-clipboard.html %} ~~~ sql -ALTER VIRTUAL CLUSTER {original_primary_vc} START REPLICATION FROM {promoted_standby_vc} ON {connection_string_standby}; +ALTER VIRTUAL CLUSTER {original_primary_vc} START REPLICATION OF {promoted_standby_vc} ON {connection_string_standby}; ~~~ The original primary virtual cluster may be almost up to date with the promoted standby's virtual cluster. The difference in data between the two virtual clusters will include only the writes that have been applied to the promoted standby after cutover from the primary cluster. \ No newline at end of file diff --git a/src/current/_includes/v24.1/sidebar-data/latest-releases.json b/src/current/_includes/v24.1/sidebar-data/latest-releases.json index e69de29bb2d..e25f8c0fa7a 100644 --- a/src/current/_includes/v24.1/sidebar-data/latest-releases.json +++ b/src/current/_includes/v24.1/sidebar-data/latest-releases.json @@ -0,0 +1,7 @@ +{ + "title": "Latest Releases", + "is_top_level": true, + "items": [ + {% include_cached sidebar-latest-releases.json %} + ] + } diff --git a/src/current/_includes/v24.1/sidebar-data/migrate.json b/src/current/_includes/v24.1/sidebar-data/migrate.json index 9b87e0c2f31..5b6338a628b 100644 --- a/src/current/_includes/v24.1/sidebar-data/migrate.json +++ b/src/current/_includes/v24.1/sidebar-data/migrate.json @@ -9,7 +9,7 @@ ] }, { - "title": "Migration Tools", + "title": "MOLT Tools", "items": [ { "title": "Schema Conversion Tool", @@ -18,13 +18,13 @@ ] }, { - "title": "MOLT Fetch", + "title": "Fetch", "urls": [ "/${VERSION}/molt-fetch.html" ] }, { - "title": "MOLT Verify", + "title": "Verify", "urls": [ "/${VERSION}/molt-verify.html" ] @@ -34,7 +34,12 @@ "urls": [ "/${VERSION}/live-migration-service.html" ] - }, + } + ] + }, + { + "title": "Third-Party Migration Tools", + "items": [ { "title": "AWS DMS", "urls": [ @@ -126,6 +131,12 @@ "/${VERSION}/migrate-from-mysql.html" ] }, + { + "title": "Migrate from Oracle", + "urls": [ + "/${VERSION}/migrate-from-oracle.html" + ] + }, { "title": "Migration Strategy: Lift and Shift", "urls": [ diff --git a/src/current/_includes/v24.1/sidebar-data/releases.json b/src/current/_includes/v24.1/sidebar-data/releases.json index 18f2a1b7c6a..fb49f7c9acc 100644 --- a/src/current/_includes/v24.1/sidebar-data/releases.json +++ b/src/current/_includes/v24.1/sidebar-data/releases.json @@ -1,7 +1,7 @@ { - "title": "Releases", + "title": "CockroachDB Releases", "is_top_level": true, "items": [ - {% include_cached sidebar-releases.json %} + {% include_cached sidebar-all-releases.json %} ] } diff --git a/src/current/_includes/v24.1/ui/jobs.md b/src/current/_includes/v24.1/ui/jobs.md index 95b709a0b33..b7796f14bea 100644 --- a/src/current/_includes/v24.1/ui/jobs.md +++ b/src/current/_includes/v24.1/ui/jobs.md @@ -8,9 +8,7 @@ The Jobs list is designed for you to manage pending work. It is not intended to Use the **Jobs** table to see recently created and completed jobs. -The following screenshot shows `BACKUP` jobs: - -
+
### Filter jobs
@@ -34,10 +32,7 @@ Job ID | Unique job ID. This value is used to [pause]({{ link_prefix }}pause-job
User Name | User that created the job.
Creation Time (UTC) | Date and time the job was created.
Last Modified Time (UTC) | Date and time the job was last modified.
-Last Execution Time (UTC) | Date and time the job was last executed.
-High-water Timestamp | A checkpoint for a [changefeed job's progress]({{ link_prefix }}monitor-and-debug-changefeeds.html#monitor-a-changefeed) that guarantees that all changes before (or at) the timestamp have been emitted. Hover over the high-water timestamp to view the [system time]({{ link_prefix }}as-of-system-time.html).
-Execution Count | Number of times the job was executed.
-Coordinator Node | ID of the coordinating node.
+Completed Time (UTC) | Date and time the job was completed.
To view [job details](#job-details), click the job description.
@@ -56,7 +51,7 @@ Status | Description
`REVERTING`| Job failed or was canceled and its changes are being reverted.
`REVERT-FAILED` | Job encountered a non-retryable error when reverting the changes. It is necessary to manually clean up a job with this status.
`RETRYING` | Job is retrying another job that failed.
-
+
## Job details
The details show:
diff --git a/src/current/cockroachcloud/aws-privatelink.md b/src/current/cockroachcloud/aws-privatelink.md
index 97159f551b3..c093ce2808f 100644
--- a/src/current/cockroachcloud/aws-privatelink.md
+++ b/src/current/cockroachcloud/aws-privatelink.md
@@ -23,7 +23,7 @@ AWS PrivateLink for CockroachDB {{ site.data.products.serverless }} is in **[lim
+
-
- To view published messages from your terminal, run the following command:
-
- {% include_cached copy-clipboard.html %}
- ~~~ shell
- gcloud pubsub subscriptions pull movr-users-sub --auto-ack --limit=10
- ~~~
-
- This command will **only** pull these messages once per subscription. For example, if you ran this command again you would receive 10 different messages in your output. To receive more than one message at a time, pass the `--limit` flag. For more details, see the [gcloud pubsub subscriptions pull](https://cloud.google.com/sdk/gcloud/reference/pubsub/subscriptions/pull) documentation.
-
- ~~~
- ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬──────────────────┬─────────────────────────────────────────────────────────┬────────────┬──────────────────┐
- │ DATA │ MESSAGE_ID │ ORDERING_KEY │ ATTRIBUTES │ DELIVERY_ATTEMPT │
- ├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────┼─────────────────────────────────────────────────────────┼────────────┼──────────────────┤
- │ {"key":["boston","40ef7cfa-5e16-4bd3-9e14-2f23407a66df"],"value":{"after":{"address":"14980 Gentry Plains Apt. 64","city":"boston","credit_card":"2466765790","id":"40ef7cfa-5e16-4bd3-9e14-2f23407a66df","name":"Vickie Fitzpatrick"}},"topic":"movr-users"} │ 4466153049158588 │ ["boston", "40ef7cfa-5e16-4bd3-9e14-2f23407a66df"] │ │ │
- │ {"key":["los angeles","947ae147-ae14-4800-8000-00000000001d"],"value":{"after":{"address":"35627 Chelsey Tunnel Suite 94","city":"los angeles","credit_card":"2099932769","id":"947ae147-ae14-4800-8000-00000000001d","name":"Kenneth Barnes"}},"topic":"movr-users"} │ 4466144577818136 │ ["los angeles", "947ae147-ae14-4800-8000-00000000001d"] │ │ │
- │ {"key":["amsterdam","c28f5c28-f5c2-4000-8000-000000000026"],"value":{"after":{"address":"14729 Karen Radial","city":"amsterdam","credit_card":"5844236997","id":"c28f5c28-f5c2-4000-8000-000000000026","name":"Maria Weber"}},"topic":"movr-users"} │ 4466151194002912 │ ["amsterdam", "c28f5c28-f5c2-4000-8000-000000000026"] │ │ │
- │ {"key":["new york","6c8ab772-584a-439d-b7b4-fda37767c74c"],"value":{"after":{"address":"34196 Roger Row Suite 6","city":"new york","credit_card":"3117945420","id":"6c8ab772-584a-439d-b7b4-fda37767c74c","name":"James Lang"}},"topic":"movr-users"} │ 4466147099992681 │ ["new york", "6c8ab772-584a-439d-b7b4-fda37767c74c"] │ │ │
- │ {"key":["boston","c56dab0a-63e7-4fbb-a9af-54362c481c41"],"value":{"after":{"address":"83781 Ross Overpass","city":"boston","credit_card":"7044597874","id":"c56dab0a-63e7-4fbb-a9af-54362c481c41","name":"Mark Butler"}},"topic":"movr-users"} │ 4466150752442731 │ ["boston", "c56dab0a-63e7-4fbb-a9af-54362c481c41"] │ │ │
- │ {"key":["amsterdam","f27e09d5-d7cd-4f88-8b65-abb910036f45"],"value":{"after":{"address":"77153 Donald Road Apt. 62","city":"amsterdam","credit_card":"7531160744","id":"f27e09d5-d7cd-4f88-8b65-abb910036f45","name":"Lisa Sandoval"}},"topic":"movr-users"} │ 4466147182359256 │ ["amsterdam", "f27e09d5-d7cd-4f88-8b65-abb910036f45"] │ │ │
- │ {"key":["new york","46d200c0-6924-4cc7-b3c9-3398997acb84"],"value":{"after":{"address":"92843 Carlos Grove","city":"new york","credit_card":"8822366402","id":"46d200c0-6924-4cc7-b3c9-3398997acb84","name":"Mackenzie Malone"}},"topic":"movr-users"} │ 4466142864542016 │ ["new york", "46d200c0-6924-4cc7-b3c9-3398997acb84"] │ │ │
- │ {"key":["boston","52ecbb26-0eab-4e0b-a160-90caa6a7d350"],"value":{"after":{"address":"95044 Eric Corner Suite 33","city":"boston","credit_card":"3982363300","id":"52ecbb26-0eab-4e0b-a160-90caa6a7d350","name":"Brett Porter"}},"topic":"movr-users"} │ 4466152539161631 │ ["boston", "52ecbb26-0eab-4e0b-a160-90caa6a7d350"] │ │ │
- │ {"key":["amsterdam","ae147ae1-47ae-4800-8000-000000000022"],"value":{"after":{"address":"88194 Angela Gardens Suite 94","city":"amsterdam","credit_card":"4443538758","id":"ae147ae1-47ae-4800-8000-000000000022","name":"Tyler Dalton"}},"topic":"movr-users"} │ 4466151398997150 │ ["amsterdam", "ae147ae1-47ae-4800-8000-000000000022"] │ │ │
- │ {"key":["paris","dc28f5c2-8f5c-4800-8000-00000000002b"],"value":{"after":{"address":"2058 Rodriguez Stream","city":"paris","credit_card":"9584502537","id":"dc28f5c2-8f5c-4800-8000-00000000002b","name":"Tony Ortiz"}},"topic":"movr-users"} │ 4466146372222914 │ ["paris", "dc28f5c2-8f5c-4800-8000-00000000002b"] │ │ │
- └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────┴─────────────────────────────────────────────────────────┴────────────┴──────────────────┘
- ~~~
+ To view all the messages delivered to your topic, you can use:
+ - The Google Cloud Console. From the Pub/Sub menu, select **Subscriptions** in the left-hand navigation and then select the subscription ID from your list of subscriptions. On the subscription's overview, click **Messages**, and then **Pull** to view messages.
+ - The `gcloud` CLI. From your terminal, run the following command:
+
+ {% include_cached copy-clipboard.html %}
+ ~~~ shell
+ gcloud pubsub subscriptions pull movr-users-sub --auto-ack --limit=10
+ ~~~
+
+ This command will **only** pull these messages once per subscription. For example, if you ran this command again you would receive 10 different messages in your output. To receive more than one message at a time, pass the `--limit` flag. For more details, refer to the [gcloud pubsub subscriptions pull](https://cloud.google.com/sdk/gcloud/reference/pubsub/subscriptions/pull) documentation.
+
+ If you have enabled the `changefeed.new_pubsub_sink_enabled` cluster setting, the output will contain capitalized top-level fields:
+
+ ~~~
+ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────────┬──────────────┬────────────┬──────────────────┬────────────┐
+ │ DATA │ MESSAGE_ID │ ORDERING_KEY │ ATTRIBUTES │ DELIVERY_ATTEMPT │ ACK_STATUS │
+ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────┼──────────────┼────────────┼──────────────────┼────────────┤
+ │ {"Key":["amsterdam", "09ee2856-5856-40c4-85d3-7d65bed978f0"],"Value":{"after": {"address": "84579 Peter Divide Apt. 47", "city": "amsterdam", "credit_card": "0100007510", "id": "09ee2856-5856-40c4-85d3-7d65bed978f0", "name": "Timothy Jackson"}},"Topic":"movr-users"} │ 11249015757941393 │ │ │ │ SUCCESS │
+ │ {"Key":["new york", "8803ab9e-5001-4994-a2e6-68d587f95f1d"],"Value":{"after": {"address": "37546 Andrew Roads Apt. 68", "city": "new york", "credit_card": "4731676650", "id": "8803ab9e-5001-4994-a2e6-68d587f95f1d", "name": "Susan Harrington"}},"Topic":"movr-users"} │ 11249015757941394 │ │ │ │ SUCCESS │
+ │ {"Key":["seattle", "32e27201-ca0d-4a0c-ada2-fbf47f6a4711"],"Value":{"after": {"address": "86725 Stephen Gardens", "city": "seattle", "credit_card": "3639690115", "id": "32e27201-ca0d-4a0c-ada2-fbf47f6a4711", "name": "Brad Hill"}},"Topic":"movr-users"} │ 11249015757941395 │ │ │ │ SUCCESS │
+ ...
+ ~~~
+
+ If you have **not** enabled `changefeed.new_pubsub_sink_enabled`, the output will contain lowercase top-level fields:
+
+ ~~~
+ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬──────────────────┬─────────────────────────────────────────────────────────┬────────────┬──────────────────┐
+ │ DATA │ MESSAGE_ID │ ORDERING_KEY │ ATTRIBUTES │ DELIVERY_ATTEMPT │
+ ├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────┼─────────────────────────────────────────────────────────┼────────────┼──────────────────┤
+ │ {"key":["boston","40ef7cfa-5e16-4bd3-9e14-2f23407a66df"],"value":{"after":{"address":"14980 Gentry Plains Apt. 64","city":"boston","credit_card":"2466765790","id":"40ef7cfa-5e16-4bd3-9e14-2f23407a66df","name":"Vickie Fitzpatrick"}},"topic":"movr-users"} │ 4466153049158588 │ ["boston", "40ef7cfa-5e16-4bd3-9e14-2f23407a66df"] │ │ │
+ │ {"key":["los angeles","947ae147-ae14-4800-8000-00000000001d"],"value":{"after":{"address":"35627 Chelsey Tunnel Suite 94","city":"los angeles","credit_card":"2099932769","id":"947ae147-ae14-4800-8000-00000000001d","name":"Kenneth Barnes"}},"topic":"movr-users"} │ 4466144577818136 │ ["los angeles", "947ae147-ae14-4800-8000-00000000001d"] │ │ │
+ │ {"key":["amsterdam","c28f5c28-f5c2-4000-8000-000000000026"],"value":{"after":{"address":"14729 Karen Radial","city":"amsterdam","credit_card":"5844236997","id":"c28f5c28-f5c2-4000-8000-000000000026","name":"Maria Weber"}},"topic":"movr-users"} │ 4466151194002912 │ ["amsterdam", "c28f5c28-f5c2-4000-8000-000000000026"] │ │ │
+ ...
+ ~~~
+
+ For more detail on the `changefeed.new_pubsub_sink_enabled` cluster setting, refer to [Pub/Sub sink messages]({% link {{ page.version.version }}/changefeed-sinks.md %}#pub-sub-sink-messages).
## Create a changefeed connected to a cloud storage sink
diff --git a/src/current/v23.1/changefeed-sinks.md b/src/current/v23.1/changefeed-sinks.md
index a7fc405b4c6..5f3e17f5ed1 100644
--- a/src/current/v23.1/changefeed-sinks.md
+++ b/src/current/v23.1/changefeed-sinks.md
@@ -198,10 +198,17 @@ See the [Changefeed Examples]({% link {{ page.version.version }}/changefeed-exam
{% include feature-phases/preview.md %}
{{site.data.alerts.end}}
-{% include {{ page.version.version }}/cdc/pubsub-performance-setting.md %}
-
Changefeeds can deliver messages to a Google Cloud Pub/Sub sink, which is integrated with Google Cloud Platform.
+{% include_cached new-in.html version="v23.1" %} Enable the `changefeed.new_pubsub_sink_enabled` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) to improve the throughput of changefeeds emitting to Pub/Sub sinks. Enabling this setting also alters the message format to use capitalized top-level fields in changefeeds emitting JSON-encoded messages. Therefore, you may need to reconfigure downstream systems to parse the new message format before enabling this setting:
+
+{% include_cached copy-clipboard.html %}
+~~~ sql
+SET CLUSTER SETTING changefeed.new_pubsub_sink_enabled = true;
+~~~
+
+For more details, refer to the [Pub/Sub sink messages](#pub-sub-sink-messages) section.
+
A Pub/Sub sink URI follows this example:
~~~
@@ -276,7 +283,29 @@ pubsub_sink_config = '{ "Flush": {"Messages": 100, "Frequency": "5s"}, "Retry":
### Pub/Sub sink messages
-The following shows the default JSON messages for a changefeed emitting to Pub/Sub. These changefeed messages were emitted as part of the [Create a changefeed connected to a Google Cloud Pub/Sub sink]({% link {{ page.version.version }}/changefeed-examples.md %}#create-a-changefeed-connected-to-a-google-cloud-pub-sub-sink) example:
+{{site.data.alerts.callout_info}}
+In CockroachDB v23.2 and later, changefeeds will have `changefeed.new_pubsub_sink_enabled` enabled by default.
+{{site.data.alerts.end}}
+
+When the `changefeed.new_pubsub_sink_enabled` cluster setting is enabled, changefeeds will have improved throughput and the changefeed JSON-encoded message format has top-level fields that are capitalized:
+
+~~~
+{Key: ..., Value: ..., Topic: ...}
+~~~
+
+{{site.data.alerts.callout_danger}}
+Before enabling `changefeed.new_pubsub_sink_enabled`, you may need to reconfigure downstream systems to parse the new message format.
+{{site.data.alerts.end}}
+
+With `changefeed.new_pubsub_sink_enabled` set to `false`, changefeeds emit JSON messages with the top-level fields all lowercase:
+
+~~~
+{key: ..., value: ..., topic: ...}
+~~~
+
+If `changefeed.new_pubsub_sink_enabled` is set to `false`, changefeeds will not benefit from the improved throughput performance that this setting enables.
+
+The following shows the default JSON messages for a changefeed created with `changefeed.new_pubsub_sink_enabled` set to `false`. These changefeed messages were emitted as part of the [Create a changefeed connected to a Google Cloud Pub/Sub sink]({% link {{ page.version.version }}/changefeed-examples.md %}#create-a-changefeed-connected-to-a-google-cloud-pub-sub-sink) example:
~~~
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬──────────────────┬─────────────────────────────────────────────────────────┬────────────┬──────────────────┐
diff --git a/src/current/v23.1/live-migration-service.md b/src/current/v23.1/live-migration-service.md
index 15f1ec55e75..dec3c51062d 100644
--- a/src/current/v23.1/live-migration-service.md
+++ b/src/current/v23.1/live-migration-service.md
@@ -9,7 +9,7 @@ docs_area: migrate
{% include feature-phases/preview.md %}
{{site.data.alerts.end}}
-MOLT LMS (Live Migration Service) is used to perform a [live migration]({% link {{ page.version.version }}/migration-overview.md %}#minimal-downtime) to CockroachDB.
+MOLT LMS (Live Migration Service) is used during a [live migration]({% link {{ page.version.version }}/migration-overview.md %}#minimal-downtime) to CockroachDB.
The LMS is a self-hosted, horizontally scalable proxy that routes traffic between an application, a source database, and a target CockroachDB database. You use the LMS to control which database, as the "source of truth", is serving reads and writes to an application. You can optionally configure the LMS to [shadow production traffic](#shadowing-modes) from the source database and validate the query results on CockroachDB. When you have sufficiently tested your application and are confident with its consistency and performance on CockroachDB, you use the LMS to [perform the cutover](#perform-a-cutover) to CockroachDB.
diff --git a/src/current/v23.1/migrate-from-oracle.md b/src/current/v23.1/migrate-from-oracle.md
index 7606c9d9521..0709925396c 100644
--- a/src/current/v23.1/migrate-from-oracle.md
+++ b/src/current/v23.1/migrate-from-oracle.md
@@ -6,10 +6,10 @@ docs_area: migrate
---
{{site.data.alerts.callout_danger}}
-The instructions on this page require updates. We currently recommend [using AWS Database Migration Service (DMS) to migrate data]({% link {{ page.version.version }}/aws-dms.md %}) from Oracle to CockroachDB. You can also [migrate from CSV]({% link {{ page.version.version }}/migrate-from-csv.md %}).
-{{site.data.alerts.end}}
+The instructions on this page are outdated. Use the [Schema Conversion Tool]({% link cockroachcloud/migrations-page.md %}?filters=oracle) to convert an Oracle schema into a compatible CockroachDB schema, and a tool such as [AWS Database Migration Service (DMS)]({% link {{ page.version.version }}/aws-dms.md %}) or [Qlik]({% link {{ page.version.version }}/qlik.md %}) to migrate data from Oracle to CockroachDB.
-This page has instructions for migrating data from Oracle into CockroachDB by [importing]({% link {{ page.version.version }}/import.md %}) CSV files. Note that `IMPORT` only works for creating new tables. For information on how to add CSV data to existing tables, see [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}).
+`IMPORT` is deprecated. To move data into CockroachDB, use [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}) or [`COPY FROM`]({% link {{ page.version.version }}/copy-from.md %}).
+{{site.data.alerts.end}}
To illustrate this process, we use the following sample data and tools:
diff --git a/src/current/v23.1/molt-verify.md b/src/current/v23.1/molt-verify.md
index 405c304308b..7bf7b9ab489 100644
--- a/src/current/v23.1/molt-verify.md
+++ b/src/current/v23.1/molt-verify.md
@@ -40,7 +40,7 @@ To install MOLT Verify, download the binary that matches your system. To downloa
| Mac | [Download](https://molt.cockroachdb.com/molt/cli/molt-latest.darwin-amd64.tgz) | [Download](https://molt.cockroachdb.com/molt/cli/molt-latest.darwin-arm64.tgz) |
{{site.data.alerts.callout_success}}
-For previous binaries, see the [MOLT version manifest](https://molt.cockroachdb.com/molt/cli/versions.html). For releases v0.0.6 and earlier, see the [MOLT repository](https://github.com/cockroachdb/molt/releases).
+For previous binaries, refer to the [MOLT version manifest](https://molt.cockroachdb.com/molt/cli/versions.html).
{{site.data.alerts.end}}
To set up MOLT Verify:
diff --git a/src/current/v23.1/restore.md b/src/current/v23.1/restore.md
index 56127599cc5..8690ae760c0 100644
--- a/src/current/v23.1/restore.md
+++ b/src/current/v23.1/restore.md
@@ -117,20 +117,21 @@ You can control `RESTORE` behavior using any of the following in the `restore_op
Option | 
-
- To view published messages from your terminal, run the following command:
-
- {% include_cached copy-clipboard.html %}
- ~~~ shell
- gcloud pubsub subscriptions pull movr-users-sub --auto-ack --limit=10
- ~~~
+ {% include_cached copy-clipboard.html %}
+ ~~~ shell
+ gcloud pubsub subscriptions pull movr-users-sub --auto-ack --limit=10
+ ~~~
- This command will **only** pull these messages once per subscription. For example, if you ran this command again you would receive 10 different messages in your output. To receive more than one message at a time, pass the `--limit` flag. For more details, see the [gcloud pubsub subscriptions pull](https://cloud.google.com/sdk/gcloud/reference/pubsub/subscriptions/pull) documentation.
+ This command will **only** pull these messages once per subscription. For example, if you ran this command again you would receive 10 different messages in your output. To receive more than one message at a time, pass the `--limit` flag. For more details, refer to the [gcloud pubsub subscriptions pull](https://cloud.google.com/sdk/gcloud/reference/pubsub/subscriptions/pull) documentation.
- ~~~
- ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬──────────────────┬─────────────────────────────────────────────────────────┬────────────┬──────────────────┐
- │ DATA │ MESSAGE_ID │ ORDERING_KEY │ ATTRIBUTES │ DELIVERY_ATTEMPT │
- ├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────┼─────────────────────────────────────────────────────────┼────────────┼──────────────────┤
- │ {"key":["boston","40ef7cfa-5e16-4bd3-9e14-2f23407a66df"],"value":{"after":{"address":"14980 Gentry Plains Apt. 64","city":"boston","credit_card":"2466765790","id":"40ef7cfa-5e16-4bd3-9e14-2f23407a66df","name":"Vickie Fitzpatrick"}},"topic":"movr-users"} │ 4466153049158588 │ ["boston", "40ef7cfa-5e16-4bd3-9e14-2f23407a66df"] │ │ │
- │ {"key":["los angeles","947ae147-ae14-4800-8000-00000000001d"],"value":{"after":{"address":"35627 Chelsey Tunnel Suite 94","city":"los angeles","credit_card":"2099932769","id":"947ae147-ae14-4800-8000-00000000001d","name":"Kenneth Barnes"}},"topic":"movr-users"} │ 4466144577818136 │ ["los angeles", "947ae147-ae14-4800-8000-00000000001d"] │ │ │
- │ {"key":["amsterdam","c28f5c28-f5c2-4000-8000-000000000026"],"value":{"after":{"address":"14729 Karen Radial","city":"amsterdam","credit_card":"5844236997","id":"c28f5c28-f5c2-4000-8000-000000000026","name":"Maria Weber"}},"topic":"movr-users"} │ 4466151194002912 │ ["amsterdam", "c28f5c28-f5c2-4000-8000-000000000026"] │ │ │
- │ {"key":["new york","6c8ab772-584a-439d-b7b4-fda37767c74c"],"value":{"after":{"address":"34196 Roger Row Suite 6","city":"new york","credit_card":"3117945420","id":"6c8ab772-584a-439d-b7b4-fda37767c74c","name":"James Lang"}},"topic":"movr-users"} │ 4466147099992681 │ ["new york", "6c8ab772-584a-439d-b7b4-fda37767c74c"] │ │ │
- │ {"key":["boston","c56dab0a-63e7-4fbb-a9af-54362c481c41"],"value":{"after":{"address":"83781 Ross Overpass","city":"boston","credit_card":"7044597874","id":"c56dab0a-63e7-4fbb-a9af-54362c481c41","name":"Mark Butler"}},"topic":"movr-users"} │ 4466150752442731 │ ["boston", "c56dab0a-63e7-4fbb-a9af-54362c481c41"] │ │ │
- │ {"key":["amsterdam","f27e09d5-d7cd-4f88-8b65-abb910036f45"],"value":{"after":{"address":"77153 Donald Road Apt. 62","city":"amsterdam","credit_card":"7531160744","id":"f27e09d5-d7cd-4f88-8b65-abb910036f45","name":"Lisa Sandoval"}},"topic":"movr-users"} │ 4466147182359256 │ ["amsterdam", "f27e09d5-d7cd-4f88-8b65-abb910036f45"] │ │ │
- │ {"key":["new york","46d200c0-6924-4cc7-b3c9-3398997acb84"],"value":{"after":{"address":"92843 Carlos Grove","city":"new york","credit_card":"8822366402","id":"46d200c0-6924-4cc7-b3c9-3398997acb84","name":"Mackenzie Malone"}},"topic":"movr-users"} │ 4466142864542016 │ ["new york", "46d200c0-6924-4cc7-b3c9-3398997acb84"] │ │ │
- │ {"key":["boston","52ecbb26-0eab-4e0b-a160-90caa6a7d350"],"value":{"after":{"address":"95044 Eric Corner Suite 33","city":"boston","credit_card":"3982363300","id":"52ecbb26-0eab-4e0b-a160-90caa6a7d350","name":"Brett Porter"}},"topic":"movr-users"} │ 4466152539161631 │ ["boston", "52ecbb26-0eab-4e0b-a160-90caa6a7d350"] │ │ │
- │ {"key":["amsterdam","ae147ae1-47ae-4800-8000-000000000022"],"value":{"after":{"address":"88194 Angela Gardens Suite 94","city":"amsterdam","credit_card":"4443538758","id":"ae147ae1-47ae-4800-8000-000000000022","name":"Tyler Dalton"}},"topic":"movr-users"} │ 4466151398997150 │ ["amsterdam", "ae147ae1-47ae-4800-8000-000000000022"] │ │ │
- │ {"key":["paris","dc28f5c2-8f5c-4800-8000-00000000002b"],"value":{"after":{"address":"2058 Rodriguez Stream","city":"paris","credit_card":"9584502537","id":"dc28f5c2-8f5c-4800-8000-00000000002b","name":"Tony Ortiz"}},"topic":"movr-users"} │ 4466146372222914 │ ["paris", "dc28f5c2-8f5c-4800-8000-00000000002b"] │ │ │
- └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────┴─────────────────────────────────────────────────────────┴────────────┴──────────────────┘
- ~~~
+ ~~~
+ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────────┬──────────────┬────────────┬──────────────────┬────────────┐
+ │ DATA │ MESSAGE_ID │ ORDERING_KEY │ ATTRIBUTES │ DELIVERY_ATTEMPT │ ACK_STATUS │
+ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────┼──────────────┼────────────┼──────────────────┼────────────┤
+ │ {"Key":["amsterdam", "09ee2856-5856-40c4-85d3-7d65bed978f0"],"Value":{"after": {"address": "84579 Peter Divide Apt. 47", "city": "amsterdam", "credit_card": "0100007510", "id": "09ee2856-5856-40c4-85d3-7d65bed978f0", "name": "Timothy Jackson"}},"Topic":"movr-users"} │ 11249015757941393 │ │ │ │ SUCCESS │
+ │ {"Key":["new york", "8803ab9e-5001-4994-a2e6-68d587f95f1d"],"Value":{"after": {"address": "37546 Andrew Roads Apt. 68", "city": "new york", "credit_card": "4731676650", "id": "8803ab9e-5001-4994-a2e6-68d587f95f1d", "name": "Susan Harrington"}},"Topic":"movr-users"} │ 11249015757941394 │ │ │ │ SUCCESS │
+ │ {"Key":["seattle", "32e27201-ca0d-4a0c-ada2-fbf47f6a4711"],"Value":{"after": {"address": "86725 Stephen Gardens", "city": "seattle", "credit_card": "3639690115", "id": "32e27201-ca0d-4a0c-ada2-fbf47f6a4711", "name": "Brad Hill"}},"Topic":"movr-users"} │ 11249015757941395 │ │ │ │ SUCCESS │
+ │ {"Key":["san francisco", "27b03637-ef9f-49a0-9b58-b16d7a9e34f4"],"Value":{"after": {"address": "85467 Tiffany Field", "city": "san francisco", "credit_card": "0016125921", "id": "27b03637-ef9f-49a0-9b58-b16d7a9e34f4", "name": "Mark Garcia"}},"Topic":"movr-users"} │ 11249015757941396 │ │ │ │ SUCCESS │
+ │ {"Key":["rome", "982e1863-88d4-49cb-adee-0a35baae7e0b"],"Value":{"after": {"address": "54918 Sutton Isle Suite 74", "city": "rome", "credit_card": "6015706174", "id": "982e1863-88d4-49cb-adee-0a35baae7e0b", "name": "Kimberly Nichols"}},"Topic":"movr-users"} │ 11249015757941397 │ │ │ │ SUCCESS │
+ │ {"Key":["washington dc", "7b298994-7b12-414c-90ef-353c7105f012"],"Value":{"after": {"address": "45205 Romero Ford Apt. 86", "city": "washington dc", "credit_card": "3519400314", "id": "7b298994-7b12-414c-90ef-353c7105f012", "name": "Taylor Bullock"}},"Topic":"movr-users"} │ 11249015757941398 │ │ │ │ SUCCESS │
+ │ {"Key":["boston", "4f012f57-577b-4853-b5ab-0d79d0df1369"],"Value":{"after": {"address": "15765 Vang Ramp", "city": "boston", "credit_card": "6747715133", "id": "4f012f57-577b-4853-b5ab-0d79d0df1369", "name": "Ryan Garcia"}},"Topic":"movr-users"} │ 11249015757941399 │ │ │ │ SUCCESS │
+ │ {"Key":["seattle", "9ba85917-5545-4674-8ab2-497fa47ac00f"],"Value":{"after": {"address": "24354 Whitney Lodge", "city": "seattle", "credit_card": "8642661685", "id": "9ba85917-5545-4674-8ab2-497fa47ac00f", "name": "Donald Walsh"}},"Topic":"movr-users"} │ 11249015757941400 │ │ │ │ SUCCESS │
+ │ {"Key":["seattle", "98312fb3-230e-412d-9b22-074ec97329ff"],"Value":{"after": {"address": "72777 Carol Shoal", "city": "seattle", "credit_card": "7789799678", "id": "98312fb3-230e-412d-9b22-074ec97329ff", "name": "Christopher Davis"}},"Topic":"movr-users"} │ 11249015757941401 │ │ │ │ SUCCESS │
+ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────────┴──────────────┴────────────┴──────────────────┴────────────┘
+ ~~~
## Create a changefeed connected to a cloud storage sink
diff --git a/src/current/v23.2/changefeed-sinks.md b/src/current/v23.2/changefeed-sinks.md
index 1fa087b77ec..f8cf997c883 100644
--- a/src/current/v23.2/changefeed-sinks.md
+++ b/src/current/v23.2/changefeed-sinks.md
@@ -230,6 +230,10 @@ The following parameters are also needed, but are **set by default** in Cockroac
Changefeeds can deliver messages to a Google Cloud Pub/Sub sink, which is integrated with Google Cloud Platform.
+{{site.data.alerts.callout_info}}
+{% include_cached new-in.html version="v23.2" %} The `changefeed.new_pubsub_sink_enabled` cluster setting is enabled by default, which provides improved throughput. Without this cluster setting enabled, changefeeds emit JSON-encoded events with the top-level message fields all lowercase. With `changefeed.new_pubsub_sink_enabled`, the top-level fields are capitalized. For more details, refer to the [Pub/Sub sink messages](#pub-sub-sink-messages) section.
+{{site.data.alerts.end}}
+
A Pub/Sub sink URI follows this example:
~~~
@@ -304,23 +308,40 @@ pubsub_sink_config = '{ "Flush": {"Messages": 100, "Frequency": "5s"}, "Retry":
### Pub/Sub sink messages
+{% include_cached new-in.html version="v23.2" %} The `changefeed.new_pubsub_sink_enabled` cluster setting is enabled by default, which provides improved changefeed throughput peformance. With `changefeed.new_pubsub_sink_enabled` enabled, the changefeed JSON-encoded message format have top-level fields that are capitalized:
+
+~~~
+{Key: ..., Value: ..., Topic: ...}
+~~~
+
+{{site.data.alerts.callout_danger}}
+By default in v23.2, the capitalization of top-level fields in the message has changed. Before upgrading to CockroachDB v23.2 and later, you may need to reconfigure downstream systems to parse the new message format.
+{{site.data.alerts.end}}
+
+With `changefeed.new_pubsub_sink_enabled` set to `false`, changefeeds emit JSON messages with the top-level fields all lowercase:
+
+~~~
+{key: ..., value: ..., topic: ...}
+~~~
+
+If `changefeed.new_pubsub_sink_enabled` is set to `false`, changefeeds will not benefit from the improved throughput performance that this setting enables.
+
The following shows the default JSON messages for a changefeed emitting to Pub/Sub. These changefeed messages were emitted as part of the [Create a changefeed connected to a Google Cloud Pub/Sub sink]({% link {{ page.version.version }}/changefeed-examples.md %}#create-a-changefeed-connected-to-a-google-cloud-pub-sub-sink) example:
~~~
-┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬──────────────────┬─────────────────────────────────────────────────────────┬────────────┬──────────────────┐
-│ DATA │ MESSAGE_ID │ ORDERING_KEY │ ATTRIBUTES │ DELIVERY_ATTEMPT │
-├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────┼─────────────────────────────────────────────────────────┼────────────┼──────────────────┤
-│ {"key":["boston","40ef7cfa-5e16-4bd3-9e14-2f23407a66df"],"value":{"after":{"address":"14980 Gentry Plains Apt. 64","city":"boston","credit_card":"2466765790","id":"40ef7cfa-5e16-4bd3-9e14-2f23407a66df","name":"Vickie Fitzpatrick"}},"topic":"movr-users"} │ 4466153049158588 │ ["boston", "40ef7cfa-5e16-4bd3-9e14-2f23407a66df"] │ │ │
-│ {"key":["los angeles","947ae147-ae14-4800-8000-00000000001d"],"value":{"after":{"address":"35627 Chelsey Tunnel Suite 94","city":"los angeles","credit_card":"2099932769","id":"947ae147-ae14-4800-8000-00000000001d","name":"Kenneth Barnes"}},"topic":"movr-users"} │ 4466144577818136 │ ["los angeles", "947ae147-ae14-4800-8000-00000000001d"] │ │ │
-│ {"key":["amsterdam","c28f5c28-f5c2-4000-8000-000000000026"],"value":{"after":{"address":"14729 Karen Radial","city":"amsterdam","credit_card":"5844236997","id":"c28f5c28-f5c2-4000-8000-000000000026","name":"Maria Weber"}},"topic":"movr-users"} │ 4466151194002912 │ ["amsterdam", "c28f5c28-f5c2-4000-8000-000000000026"] │ │ │
-│ {"key":["new york","6c8ab772-584a-439d-b7b4-fda37767c74c"],"value":{"after":{"address":"34196 Roger Row Suite 6","city":"new york","credit_card":"3117945420","id":"6c8ab772-584a-439d-b7b4-fda37767c74c","name":"James Lang"}},"topic":"movr-users"} │ 4466147099992681 │ ["new york", "6c8ab772-584a-439d-b7b4-fda37767c74c"] │ │ │
-│ {"key":["boston","c56dab0a-63e7-4fbb-a9af-54362c481c41"],"value":{"after":{"address":"83781 Ross Overpass","city":"boston","credit_card":"7044597874","id":"c56dab0a-63e7-4fbb-a9af-54362c481c41","name":"Mark Butler"}},"topic":"movr-users"} │ 4466150752442731 │ ["boston", "c56dab0a-63e7-4fbb-a9af-54362c481c41"] │ │ │
-│ {"key":["amsterdam","f27e09d5-d7cd-4f88-8b65-abb910036f45"],"value":{"after":{"address":"77153 Donald Road Apt. 62","city":"amsterdam","credit_card":"7531160744","id":"f27e09d5-d7cd-4f88-8b65-abb910036f45","name":"Lisa Sandoval"}},"topic":"movr-users"} │ 4466147182359256 │ ["amsterdam", "f27e09d5-d7cd-4f88-8b65-abb910036f45"] │ │ │
-│ {"key":["new york","46d200c0-6924-4cc7-b3c9-3398997acb84"],"value":{"after":{"address":"92843 Carlos Grove","city":"new york","credit_card":"8822366402","id":"46d200c0-6924-4cc7-b3c9-3398997acb84","name":"Mackenzie Malone"}},"topic":"movr-users"} │ 4466142864542016 │ ["new york", "46d200c0-6924-4cc7-b3c9-3398997acb84"] │ │ │
-│ {"key":["boston","52ecbb26-0eab-4e0b-a160-90caa6a7d350"],"value":{"after":{"address":"95044 Eric Corner Suite 33","city":"boston","credit_card":"3982363300","id":"52ecbb26-0eab-4e0b-a160-90caa6a7d350","name":"Brett Porter"}},"topic":"movr-users"} │ 4466152539161631 │ ["boston", "52ecbb26-0eab-4e0b-a160-90caa6a7d350"] │ │ │
-│ {"key":["amsterdam","ae147ae1-47ae-4800-8000-000000000022"],"value":{"after":{"address":"88194 Angela Gardens Suite 94","city":"amsterdam","credit_card":"4443538758","id":"ae147ae1-47ae-4800-8000-000000000022","name":"Tyler Dalton"}},"topic":"movr-users"} │ 4466151398997150 │ ["amsterdam", "ae147ae1-47ae-4800-8000-000000000022"] │ │ │
-│ {"key":["paris","dc28f5c2-8f5c-4800-8000-00000000002b"],"value":{"after":{"address":"2058 Rodriguez Stream","city":"paris","credit_card":"9584502537","id":"dc28f5c2-8f5c-4800-8000-00000000002b","name":"Tony Ortiz"}},"topic":"movr-users"} │ 4466146372222914 │ ["paris", "dc28f5c2-8f5c-4800-8000-00000000002b"] │ │ │
-└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────┴─────────────────────────────────────────────────────────┴────────────┴──────────────────┘
+┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────────┬──────────────┬────────────┬──────────────────┬────────────┐
+│ DATA │ MESSAGE_ID │ ORDERING_KEY │ ATTRIBUTES │ DELIVERY_ATTEMPT │ ACK_STATUS │
+├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────┼──────────────┼────────────┼──────────────────┼────────────┤
+│ {"Key":["amsterdam", "09ee2856-5856-40c4-85d3-7d65bed978f0"],"Value":{"after": {"address": "84579 Peter Divide Apt. 47", "city": "amsterdam", "credit_card": "0100007510", "id": "09ee2856-5856-40c4-85d3-7d65bed978f0", "name": "Timothy Jackson"}},"Topic":"users"} │ 11249015757941393 │ │ │ │ SUCCESS │
+│ {"Key":["new york", "8803ab9e-5001-4994-a2e6-68d587f95f1d"],"Value":{"after": {"address": "37546 Andrew Roads Apt. 68", "city": "new york", "credit_card": "4731676650", "id": "8803ab9e-5001-4994-a2e6-68d587f95f1d", "name": "Susan Harrington"}},"Topic":"users"} │ 11249015757941394 │ │ │ │ SUCCESS │
+│ {"Key":["seattle", "32e27201-ca0d-4a0c-ada2-fbf47f6a4711"],"Value":{"after": {"address": "86725 Stephen Gardens", "city": "seattle", "credit_card": "3639690115", "id": "32e27201-ca0d-4a0c-ada2-fbf47f6a4711", "name": "Brad Hill"}},"Topic":"users"} │ 11249015757941395 │ │ │ │ SUCCESS │
+│ {"Key":["san francisco", "27b03637-ef9f-49a0-9b58-b16d7a9e34f4"],"Value":{"after": {"address": "85467 Tiffany Field", "city": "san francisco", "credit_card": "0016125921", "id": "27b03637-ef9f-49a0-9b58-b16d7a9e34f4", "name": "Mark Garcia"}},"Topic":"users"} │ 11249015757941396 │ │ │ │ SUCCESS │
+│ {"Key":["rome", "982e1863-88d4-49cb-adee-0a35baae7e0b"],"Value":{"after": {"address": "54918 Sutton Isle Suite 74", "city": "rome", "credit_card": "6015706174", "id": "982e1863-88d4-49cb-adee-0a35baae7e0b", "name": "Kimberly Nichols"}},"Topic":"users"} │ 11249015757941397 │ │ │ │ SUCCESS │
+│ {"Key":["washington dc", "7b298994-7b12-414c-90ef-353c7105f012"],"Value":{"after": {"address": "45205 Romero Ford Apt. 86", "city": "washington dc", "credit_card": "3519400314", "id": "7b298994-7b12-414c-90ef-353c7105f012", "name": "Taylor Bullock"}},"Topic":"users"} │ 11249015757941398 │ │ │ │ SUCCESS │
+│ {"Key":["boston", "4f012f57-577b-4853-b5ab-0d79d0df1369"],"Value":{"after": {"address": "15765 Vang Ramp", "city": "boston", "credit_card": "6747715133", "id": "4f012f57-577b-4853-b5ab-0d79d0df1369", "name": "Ryan Garcia"}},"Topic":"users"} │ 11249015757941399 │ │ │ │ SUCCESS │
+│ {"Key":["seattle", "9ba85917-5545-4674-8ab2-497fa47ac00f"],"Value":{"after": {"address": "24354 Whitney Lodge", "city": "seattle", "credit_card": "8642661685", "id": "9ba85917-5545-4674-8ab2-497fa47ac00f", "name": "Donald Walsh"}},"Topic":"users"} │ 11249015757941400 │ │ │ │ SUCCESS │
+│ {"Key":["seattle", "98312fb3-230e-412d-9b22-074ec97329ff"],"Value":{"after": {"address": "72777 Carol Shoal", "city": "seattle", "credit_card": "7789799678", "id": "98312fb3-230e-412d-9b22-074ec97329ff", "name": "Christopher Davis"}},"Topic":"users"} │ 11249015757941401 │ │ │ │ SUCCESS │
+└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────────┴──────────────┴────────────┴──────────────────┴────────────┘
~~~
{% include {{ page.version.version }}/cdc/note-changefeed-message-page.md %}
diff --git a/src/current/v23.2/create-changefeed.md b/src/current/v23.2/create-changefeed.md
index e823f9fac61..54326575fdd 100644
--- a/src/current/v23.2/create-changefeed.md
+++ b/src/current/v23.2/create-changefeed.md
@@ -104,6 +104,8 @@ Example of a Google Cloud Pub/Sub sink URI:
'gcpubsub://{project name}?region={region}&topic_name={topic name}&AUTH=specified&CREDENTIALS={base64-encoded key}'
~~~
+In CockroachDB v23.2 and later, the `changefeed.new_pubsub_sink_enabled` cluster setting is enabled by default, which provides improved throughput. For details on the changes to the message format, refer to [Pub/Sub sink messages]({% link {{ page.version.version }}/changefeed-sinks.md %}#pub-sub-sink-messages).
+
[Use Cloud Storage for Bulk Operations]({% link {{ page.version.version }}/cloud-storage-authentication.md %}) explains the requirements for the authentication parameter with `specified` or `implicit`. Refer to [Changefeed Sinks]({% link {{ page.version.version }}/changefeed-sinks.md %}#google-cloud-pub-sub) for further consideration.
#### Cloud Storage
diff --git a/src/current/v23.2/live-migration-service.md b/src/current/v23.2/live-migration-service.md
index e106034bd30..7f619fd3da0 100644
--- a/src/current/v23.2/live-migration-service.md
+++ b/src/current/v23.2/live-migration-service.md
@@ -9,7 +9,7 @@ docs_area: migrate
{% include feature-phases/preview.md %}
{{site.data.alerts.end}}
-MOLT LMS (Live Migration Service) is used to perform a [live migration]({% link {{ page.version.version }}/migration-overview.md %}#minimal-downtime) to CockroachDB.
+MOLT LMS (Live Migration Service) is used during a [live migration]({% link {{ page.version.version }}/migration-overview.md %}#minimal-downtime) to CockroachDB.
The LMS is a self-hosted, horizontally scalable proxy that routes traffic between an application, a source database, and a target CockroachDB database. You use the LMS to control which database, as the "source of truth", is serving reads and writes to an application. You can optionally configure the LMS to [shadow production traffic](#shadowing-modes) from the source database and validate the query results on CockroachDB. When you have sufficiently tested your application and are confident with its consistency and performance on CockroachDB, you use the LMS to [perform the cutover](#perform-a-cutover) to CockroachDB.
diff --git a/src/current/v23.2/migrate-from-oracle.md b/src/current/v23.2/migrate-from-oracle.md
index 7606c9d9521..ed8e3f901d9 100644
--- a/src/current/v23.2/migrate-from-oracle.md
+++ b/src/current/v23.2/migrate-from-oracle.md
@@ -6,10 +6,10 @@ docs_area: migrate
---
{{site.data.alerts.callout_danger}}
-The instructions on this page require updates. We currently recommend [using AWS Database Migration Service (DMS) to migrate data]({% link {{ page.version.version }}/aws-dms.md %}) from Oracle to CockroachDB. You can also [migrate from CSV]({% link {{ page.version.version }}/migrate-from-csv.md %}).
-{{site.data.alerts.end}}
+The instructions on this page are outdated. Use the [Schema Conversion Tool]({% link cockroachcloud/migrations-page.md %}?filters=oracle) to convert an Oracle schema into a compatible CockroachDB schema, and a tool such as [AWS Database Migration Service (DMS)]({% link {{ page.version.version }}/aws-dms.md %}) or [Qlik]({% link {{ page.version.version }}/qlik.md %}) to migrate data from Oracle to CockroachDB.
-This page has instructions for migrating data from Oracle into CockroachDB by [importing]({% link {{ page.version.version }}/import.md %}) CSV files. Note that `IMPORT` only works for creating new tables. For information on how to add CSV data to existing tables, see [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}).
+Note that `IMPORT` is deprecated. To move data into CockroachDB, use [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}) or [`COPY FROM`]({% link {{ page.version.version }}/copy-from.md %}).
+{{site.data.alerts.end}}
To illustrate this process, we use the following sample data and tools:
diff --git a/src/current/v23.2/migration-overview.md b/src/current/v23.2/migration-overview.md
index 3912952a489..3e1372af46a 100644
--- a/src/current/v23.2/migration-overview.md
+++ b/src/current/v23.2/migration-overview.md
@@ -62,17 +62,17 @@ A lift-and-shift approach is the most straightforward. However, it's important t
- *Reduced functionality* takes some, but not all, application functionality offline. For example, you can disable writes but not reads while you migrate the application data, and queue data to be written after completing the migration.
-For an overview of lift-and-shift migrations to CockroachDB, see [Lift and Shift](#lift-and-shift). For considerations and details about the pros and cons of this approach, see [Migration Strategy: Lift and Shift]({% link {{ page.version.version }}/migration-strategy-lift-and-shift.md %}).
+For an overview of lift-and-shift migrations to CockroachDB, see [Lift and Shift](#lift-and-shift).
#### Minimal downtime
-If your application cannot tolerate downtime, then you should aim for a "zero-downtime" approach. "Zero" means that downtime is reduced to either an absolute minimum or zero, such that users do not notice the migration.
+If your application cannot tolerate downtime, then you should aim for a "zero-downtime" approach. This reduces downtime to an absolute minimum, such that users do not notice the migration.
The minimum possible downtime depends on whether you can tolerate inconsistency in the migrated data:
-- *Consistent* migrations reduce downtime to an absolute minimum (i.e., from 30 seconds to sub-seconds) while keeping data synchronized between the source database and CockroachDB. **Consistency requires downtime.** In this approach, downtime occurs right before [cutover](#cutover-strategy), as you drain the remaining transactions from the source database to CockroachDB.
+- Migrations performed using *consistent cutover* reduce downtime to an absolute minimum (i.e., seconds or sub-seconds) while keeping data synchronized between the source database and CockroachDB. **Consistency requires downtime.** In this approach, downtime occurs right before [cutover](#cutover-strategy), as you drain the remaining transactions from the source database to CockroachDB.
-- *Inconsistent* migrations can reduce downtime to zero. These require the most preparation, and typically allow read/write traffic to both databases for at least a small amount of time, thereby sacrificing consistency for availability. {% comment %}You can use the CockroachDB Live Migration Service (MOLT LMS) to run application queries simultaneously on your source database and CockroachDB.{% endcomment %} Without stopping application traffic, you perform an immediate [cutover](#cutover-strategy), while assuming that some writes will not be replicated to CockroachDB. You may want to manually reconcile these data inconsistencies after switching over.
+- Migrations performed using *immediate cutover* can reduce downtime to zero. These require the most preparation, and typically allow read/write traffic to both databases for at least a short period of time, sacrificing consistency for availability. {% comment %}You can use the CockroachDB Live Migration Service (MOLT LMS) to run application queries simultaneously on your source database and CockroachDB.{% endcomment %} Without stopping application traffic, you perform an **immediate** [cutover](#cutover-strategy), while assuming that some writes will not be replicated to CockroachDB. You may want to manually reconcile these data inconsistencies after switching over.
For an overview of zero-downtime migrations to CockroachDB, see [Zero Downtime](#zero-downtime). {% comment %}For details, see [Migration Strategy: Zero Downtime](migration-strategy-zero-downtime).{% endcomment %}
@@ -245,9 +245,11 @@ Then import the converted schema to a CockroachDB cluster:
Before moving data, Cockroach Labs recommends [dropping any indexes]({% link {{ page.version.version }}/drop-index.md %}) on the CockroachDB database. The indexes can be [recreated]({% link {{ page.version.version }}/create-index.md %}) after the data is loaded. Doing so will optimize performance.
{{site.data.alerts.end}}
-After [converting the schema](#convert-the-schema), load your data into CockroachDB so that you can [test your application queries](#validate-queries). Then use one of the following methods to migrate the data (you may need to use additional tooling to extract and/or convert the data to an appropriate file format):
+After [converting the schema](#convert-the-schema), load your data into CockroachDB so that you can [test your application queries](#validate-queries). Then use [MOLT Fetch]({% link {{ page.version.version }}/molt-fetch.md %}) to move the source data to CockroachDB.
-- {% include {{ page.version.version }}/migration/load-data-import-into.md %} Typically, initial data loading during a database migration will not be running concurrently with application traffic, so the fact that `IMPORT INTO` takes the table offline may not have any observable availability impact.
+Alternatively, you can use one of the following methods to migrate the data. Additional tooling may be required to extract or convert the data to a supported file format.
+
+- {% include {{ page.version.version }}/migration/load-data-import-into.md %} Typically during a migration, data is initially loaded before foreground application traffic begins to be served, so the impact of taking the table offline when running `IMPORT INTO` may be minimal.
- {% include {{ page.version.version }}/migration/load-data-third-party.md %} Within the tool, you can select the database tables to migrate to the test cluster.
- {% include {{ page.version.version }}/migration/load-data-copy-from.md %}
@@ -259,9 +261,9 @@ Note that CockroachDB defaults to the [`SERIALIZABLE`]({% link {{ page.version.v
##### Shadowing
-You can "shadow" your production workload by executing your source SQL statements on CockroachDB in parallel. The [CockroachDB Live Migration Service (LMS)]({% link {{ page.version.version }}/live-migration-service.md %}) can perform shadowing. You can then [test the queries](#test-query-results-and-performance) on CockroachDB for consistency, performance, and potential issues with the migration.
+You can "shadow" your production workload by executing your source SQL statements on CockroachDB in parallel. You can then [validate the queries](#test-query-results-and-performance) on CockroachDB for consistency, performance, and potential issues with the migration.
-Shadowing may not be necessary or practical for your workload. For example, because transactions are serialized on CockroachDB, this will limit your ability to validate the performance of high-throughput workloads.
+The [CockroachDB Live Migration Service (MOLT LMS)]({% link {{ page.version.version }}/live-migration-service.md %}) can [perform shadowing]({% link {{ page.version.version }}/live-migration-service.md %}#shadowing-modes). This is intended only for [testing](#test-query-results-and-performance) or [performing a dry run](#perform-a-dry-run). Shadowing should **not** be used in production when performing a [live migration](#zero-downtime).
##### Test query results and performance
@@ -310,35 +312,32 @@ Using this method, consistency is achieved by only performing the cutover once a
The following is a high-level overview of the migration steps. For considerations and details about the pros and cons of this approach, see [Migration Strategy: Lift and Shift]({% link {{ page.version.version }}/migration-strategy-lift-and-shift.md %}).
1. Stop application traffic to your source database. **This begins downtime.**
-1. Move data in one of the following ways:
- - {% include {{ page.version.version }}/migration/load-data-import-into.md %}
- - {% include {{ page.version.version }}/migration/load-data-third-party.md %}
- - {% include {{ page.version.version }}/migration/load-data-copy-from.md %}
-1. After the data is migrated, you can use [MOLT Verify]({% link {{ page.version.version }}/molt-verify.md %}) to validate the consistency of the data between the source database and CockroachDB.
+1. Use [MOLT Fetch]({% link {{ page.version.version }}/molt-fetch.md %}) to move the source data to CockroachDB.
+1. After the data is migrated, use [MOLT Verify]({% link {{ page.version.version }}/molt-verify.md %}) to validate the consistency of the data between the source database and CockroachDB.
1. Perform a [cutover](#cutover-strategy) by resuming application traffic, now to CockroachDB.
{% comment %}1. If you want the ability to [roll back](#all-at-once-rollback) the migration, replicate data back to the source database.{% endcomment %}
### Zero Downtime
-Using this method, downtime is minimized by performing the cutover while writes are still being replicated from the source database to CockroachDB. Inconsistencies are resolved through manual reconciliation.
-
-The following is a high-level overview of the migration steps. {% comment %}For details on this migration strategy, see [Migration Strategy: Zero Downtime]({% link {{ page.version.version }}/migration-strategy-lift-and-shift.md %}).{% endcomment %}
+During a "live migration", downtime is minimized by performing the cutover while writes are still being replicated from the source database to CockroachDB. Inconsistencies are resolved through manual reconciliation.
-{% comment %}You can use the CockroachDB Live Migration Service (MOLT LMS) to run application queries simultaneously on your source database and CockroachDB.{% endcomment %}
+The following is a high-level overview of the migration steps. The two approaches are mutually exclusive, and each has [tradeoffs](#minimal-downtime). {% comment %}For details on this migration strategy, see [Migration Strategy: Zero Downtime]({% link {{ page.version.version }}/migration-strategy-lift-and-shift.md %}).{% endcomment %}
To prioritize consistency and minimize downtime:
-1. {% include {{ page.version.version }}/migration/load-data-third-party.md %} Select the tool's option to **replicate ongoing changes** after performing the initial load of data into CockroachDB.
-1. As the data is migrating, you can use [MOLT Verify]({% link {{ page.version.version }}/molt-verify.md %}) to validate the consistency of the data between the source database and CockroachDB.
-1. Once nearly all data from your source database has been moved to CockroachDB (for example, with a <1 second delay or <1000 rows), stop application traffic to your source database. **This begins downtime.**
-1. Wait for replication to CockroachDB to complete.
-1. Perform a [cutover](#cutover-strategy) by resuming application traffic, now to CockroachDB.
+1. Set up the [CockroachDB Live Migration Service (MOLT LMS)]({% link {{ page.version.version }}/live-migration-service.md %}) to proxy for application traffic between your source database and CockroachDB. Do **not** shadow the application traffic.
+1. Use [MOLT Fetch]({% link {{ page.version.version }}/molt-fetch.md %}) to move the source data to CockroachDB. Use the tool to [**replicate ongoing changes**]({% link {{ page.version.version }}/molt-fetch.md %}#replication) after it performs the initial load of data into CockroachDB.
+1. As the data is migrating, use [MOLT Verify]({% link {{ page.version.version }}/molt-verify.md %}) to validate the consistency of the data between the source database and CockroachDB.
+1. After nearly all data from your source database has been moved to CockroachDB (for example, with a <1-second delay or <1000 rows), use MOLT LMS to begin a [*consistent cutover*]({% link {{ page.version.version }}/live-migration-service.md %}#consistent-cutover) and stop application traffic to your source database. **This begins downtime.**
+1. Wait for MOLT Fetch to finish replicating changes to CockroachDB.
+1. Use MOLT LMS to commit the [consistent cutover]({% link {{ page.version.version }}/live-migration-service.md %}#consistent-cutover). This resumes application traffic, now to CockroachDB.
To achieve zero downtime with inconsistency:
-1. {% include {{ page.version.version }}/migration/load-data-third-party.md %} Select the tool's option to replicate ongoing changes after performing the initial load of data into CockroachDB.
+1. Set up the [CockroachDB Live Migration Service (MOLT LMS)]({% link {{ page.version.version }}/live-migration-service.md %}) to proxy for application traffic between your source database and CockroachDB. Use a [shadowing mode]({% link {{ page.version.version }}/live-migration-service.md %}#shadowing-modes) to run application queries simultaneously on your source database and CockroachDB.
+1. Use [MOLT Fetch]({% link {{ page.version.version }}/molt-fetch.md %}) to move the source data to CockroachDB. Use the tool to **replicate ongoing changes** after performing the initial load of data into CockroachDB.
1. As the data is migrating, you can use [MOLT Verify]({% link {{ page.version.version }}/molt-verify.md %}) to validate the consistency of the data between the source database and CockroachDB.
-1. Once nearly all data from your source database has been moved to CockroachDB (for example, with a <1 second delay or <1000 rows), perform a [cutover](#cutover-strategy) by pointing application traffic to CockroachDB.
+1. After nearly all data from your source database has been moved to CockroachDB (for example, with a <1 second delay or <1000 rows), perform an [*immediate cutover*](#cutover-strategy) by pointing application traffic to CockroachDB.
1. Manually reconcile any inconsistencies caused by writes that were not replicated during the cutover.
1. Close the connection to the source database when you are ready to finish the migration.
diff --git a/src/current/v23.2/molt-fetch.md b/src/current/v23.2/molt-fetch.md
index c4879ad5e27..7d4098b7d23 100644
--- a/src/current/v23.2/molt-fetch.md
+++ b/src/current/v23.2/molt-fetch.md
@@ -5,13 +5,9 @@ toc: true
docs_area: migrate
---
-{{site.data.alerts.callout_info}}
-{% include feature-phases/preview.md %}
-{{site.data.alerts.end}}
-
MOLT Fetch moves data from a source database into CockroachDB as part of a [database migration]({% link {{ page.version.version }}/migration-overview.md %}).
-MOLT Fetch can use `IMPORT INTO` or `COPY FROM` to move the source data to CockroachDB via cloud storage (Google Cloud Storage or Amazon S3), a local file server, or directly without an intermediate store. For details, see [Usage](#usage).
+MOLT Fetch uses [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}) or [`COPY FROM`]({% link {{ page.version.version }}/copy-from.md %}) to move the source data to cloud storage (Google Cloud Storage or Amazon S3), a local file server, or local memory. Once the data is exported, MOLT Fetch loads the data onto a target CockroachDB database. For details, see [Usage](#usage).
## Supported databases
@@ -31,21 +27,27 @@ To install MOLT Fetch, download the binary that matches your system. To download
| Linux | [Download](https://molt.cockroachdb.com/molt/cli/molt-latest.linux-amd64.tgz) | [Download](https://molt.cockroachdb.com/molt/cli/molt-latest.linux-arm64.tgz) |
| Mac | [Download](https://molt.cockroachdb.com/molt/cli/molt-latest.darwin-amd64.tgz) | [Download](https://molt.cockroachdb.com/molt/cli/molt-latest.darwin-arm64.tgz) |
-For previous binaries, see the [MOLT version manifest](https://molt.cockroachdb.com/molt/cli/versions.html). For releases v0.0.6 and earlier, see the [MOLT repository](https://github.com/cockroachdb/molt/releases).
+For previous binaries, refer to the [MOLT version manifest](https://molt.cockroachdb.com/molt/cli/versions.html).
+
+{{site.data.alerts.callout_info}}
+MOLT Fetch is supported on Red Hat Enterprise Linux (RHEL) 9 and above.
+{{site.data.alerts.end}}
## Setup
Complete the following items before using MOLT Fetch:
-- Ensure that the source and target schemas are identical. Tables with mismatching columns may only be partially migrated.
+- Follow the recommendations in [Best practices](#best-practices) and [Security recommendations](#security-recommendations).
-- Ensure that the SQL user running MOLT Fetch has the required privileges to run [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}#required-privileges) or [`COPY FROM`]({% link {{ page.version.version }}/copy-from.md %}#required-privileges) statements, depending on your intended [mode](#fetch-mode).
+- Ensure that the source and target schemas are identical, unless you enable automatic schema creation with the [`'drop-on-target-and-recreate'`](#target-table-handling) option. If you are creating the target schema manually, review the behaviors in [Mismatch handling](#mismatch-handling).
-- To enable the [CDC cursor](#cdc-cursor) for ongoing replication:
+- Ensure that the SQL user running MOLT Fetch has [`SELECT` privileges]({% link {{ page.version.version }}/grant.md %}#supported-privileges) on the source and target CockroachDB databases, along with the required privileges to run [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}#required-privileges) or [`COPY FROM`]({% link {{ page.version.version }}/copy-from.md %}#required-privileges) (depending on the [fetch mode](#fetch-mode)) on CockroachDB, as described on their respective pages.
- - If you are migrating from PostgreSQL, enable logical replication. Set [wal_level](https://www.postgresql.org/docs/current/runtime-config-wal.html) to `logical` in `postgresql.conf` or in the SQL shell.
+- If you plan to use continuous replication (using either [`--ongoing-replication`](#replication) or the [CDC cursor](#cdc-cursor)):
- - If you are migrating from MySQL, enable [GTID](https://dev.mysql.com/doc/refman/8.0/en/replication-options-gtids.html) consistency. Set `gtid-mode` and `enforce-gtid-consistency` to `ON` in `mysql.cnf`, in the SQL shell, or as flags in the `mysql` start command.
+ - If you are migrating from PostgreSQL, enable logical replication. In `postgresql.conf` or in the SQL shell, set [`wal_level`](https://www.postgresql.org/docs/current/runtime-config-wal.html) to `logical`.
+
+ - If you are migrating from MySQL, enable [GTID](https://dev.mysql.com/doc/refman/8.0/en/replication-options-gtids.html) consistency. In `mysql.cnf`, in the SQL shell, or as flags in the `mysql` start command, set `gtid-mode` and `enforce-gtid-consistency` to `ON` and set `binlog_row_metadata` to `full`.
- Percent-encode the connection strings for the source database and [CockroachDB]({% link {{ page.version.version }}/connect-to-the-database.md %}). This ensures that the MOLT tools can parse special characters in your password.
@@ -67,64 +69,155 @@ Complete the following items before using MOLT Fetch:
postgres://postgres:a%2452%26@localhost:5432/replicationload
~~~
-- If you are using Amazon S3 for [cloud storage](#cloud-storage):
+ - If you are using Amazon S3 for [cloud storage](#cloud-storage):
- - Ensure that the environment variable and access tokens are set appropriately in the terminal running `molt fetch`. For example:
+ - Ensure that the environment variable and access tokens are set appropriately in the terminal running `molt fetch`. For example:
- {% include_cached copy-clipboard.html %}
- ~~~ shell
- export AWS_REGION='us-east-1'
- export AWS_SECRET_ACCESS_KEY='key'
- export AWS_ACCESS_KEY_ID='id'
- ~~~
+ {% include_cached copy-clipboard.html %}
+ ~~~ shell
+ export AWS_REGION='us-east-1'
+ export AWS_SECRET_ACCESS_KEY='key'
+ export AWS_ACCESS_KEY_ID='id'
+ ~~~
- - Ensure the S3 bucket is created and accessible to CockroachDB.
+ - Ensure the S3 bucket is created and accessible to CockroachDB.
-- If you are using Google Cloud Storage for [cloud storage](#cloud-storage):
+ - If you are using Google Cloud Storage for [cloud storage](#cloud-storage):
- - Ensure that your local environment is authenticated using [Application Default Credentials](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login):
+ - Ensure that your local environment is authenticated using [Application Default Credentials](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login):
- {% include_cached copy-clipboard.html %}
- ~~~ shell
- gcloud init
- gcloud auth application-default login
- ~~~
+ {% include_cached copy-clipboard.html %}
+ ~~~ shell
+ gcloud init
+ gcloud auth application-default login
+ ~~~
+
+ - Ensure the Google Cloud Storage bucket is created and accessible to CockroachDB.
+
+## Best practices
+
+- To prevent connections from terminating prematurely during data export, set the following to high values on the source database:
+
+ - **Maximum allowed number of connections:** MOLT Fetch can export data across multiple connections. The number of connections it will create is the number of shards ([`--export-concurrency`](#global-flags)) multiplied by the number of tables ([`--table-concurrency`](#global-flags)) being exported concurrently.
+ - **Maximum lifetime of a connection:** This is particularly important for MySQL sources, which can only use a single connection to move data. See the following note.
+
+- If a MySQL database is set as a [source](#source-and-target-databases), the [`--table-concurrency`](#global-flags) and [`--export-concurrency`](#global-flags) flags **cannot** be set above `1`. If these values are changed, MOLT Fetch returns an error. This guarantees consistency when moving data from MySQL, due to MySQL limitations. MySQL data is migrated to CockroachDB one table and shard at a time, using [`WITH CONSISTENT SNAPSHOT`](https://dev.mysql.com/doc/refman/8.0/en/commit.html) transactions.
+
+- To prevent memory outages during data export of tables with large rows, estimate the amount of memory used to export a table:
+
+ ~~~
+ --row-batch-size * --export-concurrency * average size of the table rows
+ ~~~
+
+ If you are exporting more than one table at a time (i.e., [`--table-concurrency`](#global-flags) is set higher than `1`), add the estimated memory usage for the tables with the largest row sizes. Ensure that you have sufficient memory to run `molt fetch`, and adjust `--row-batch-size` accordingly.
+
+- If a table in the source database is much larger than the other tables, [filter and export the largest table](#schema-and-table-selection) in its own `molt fetch` task. Repeat this for each of the largest tables. Then export the remaining tables in another task.
+
+- When using [`IMPORT INTO` mode](#fetch-mode) to load tables into CockroachDB, if the fetch process terminates before the import job completes, the hanging import job on the target database will keep the table offline. To make this table accessible again, [manually resume or cancel the job]({% link {{ page.version.version }}/import-into.md %}#view-and-control-import-jobs). Then resume `molt fetch` using [continuation](#fetch-continuation), or restart the process from the beginning.
+
+## Security recommendations
+
+Cockroach Labs **strongly** recommends the following:
+
+### Secure connections
+
+- Use secure connections to the source and [target CockroachDB database]({% link {{ page.version.version }}/connection-parameters.md %}#additional-connection-parameters) whenever possible.
+- By default, insecure connections (i.e., `sslmode=disable` on PostgreSQL; `sslmode` not set on MySQL) are disallowed. When using an insecure connection, `molt fetch` returns an error. To override this check, you can enable the `--allow-tls-mode-disable` flag. Do this **only** for testing, or if a secure SSL/TLS connection to the source or target database is not possible.
+
+### Connection strings
+
+- Avoid plaintext connection strings.
+- Provide your connection strings as environment variables.
+- If possible within your security infrastructure, use an external secrets manager to load the environment variables from stored secrets.
+
+ For example, to export connection strings as environment variables:
- - Ensure the Google Cloud Storage bucket is created and accessible to CockroachDB.
+ ~~~ shell
+ export SOURCE="postgres://postgres:postgres@localhost:5432/molt?sslmode=verify-full"
+ export TARGET="postgres://root@localhost:26257/molt?sslmode=verify-full"
+ ~~~
+
+ Afterward, to pass the environment variables in `molt fetch` commands:
+
+ ~~~ shell
+ molt fetch \
+ --source "$SOURCE" \
+ --target "$TARGET" \
+ --table-filter 'employees' \
+ --bucket-path 's3://molt-test' \
+ --table-handling truncate-if-exists
+ ~~~
+
+### Secure cloud storage
+
+- When using [cloud storage](#cloud-storage) for your intermediate store, ensure that access control is properly configured. Refer to the [GCP](https://cloud.google.com/storage/docs/access-control) or [AWS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/security-iam.html) documentation.
+- Do not use public cloud storage in production.
+
+### Perform a dry run
+
+To verify that your connections and configuration work properly, run MOLT Fetch in a staging environment before moving any data in production. Use a test or development environment that is as similar as possible to production.
+
+## Commands
+
+| Command | Usage |
+|---------|---------------------------------------------------------------------------------------------------|
+| `fetch` | Start the fetch process. This loads data from a source database to a target CockroachDB database. |
+
+### Subcommands
+
+| Command | Usage |
+|--------------|----------------------------------------------------------------------|
+| `tokens list` | List active [continuation tokens](#list-active-continuation-tokens). |
## Flags
-| Flag | Description |
-|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `--source` | (Required) Connection string for the source database. For details, see [Source and target databases](#source-and-target-databases). |
-| `--target` | (Required) Connection string for the target database. For details, see [Source and target databases](#source-and-target-databases). |
-| `--bucket-path` | The path within the [cloud storage](#cloud-storage) bucket where intermediate files are written (e.g., `'s3://bucket/path'` or `'gs://bucket/path'`). |
-| `--cleanup` | Whether to delete intermediate files after moving data using [cloud or local storage](#data-path). **Note:** Cleanup does not occur on [continuation](#fetch-continuation). |
-| `--compression` | Compression method for data when using [`IMPORT INTO` mode](#fetch-mode) (`gzip`/`none`).
-
- To view published messages from your terminal, run the following command:
-
- {% include_cached copy-clipboard.html %}
- ~~~ shell
- gcloud pubsub subscriptions pull movr-users-sub --auto-ack --limit=10
- ~~~
+ {% include_cached copy-clipboard.html %}
+ ~~~ shell
+ gcloud pubsub subscriptions pull movr-users-sub --auto-ack --limit=10
+ ~~~
- This command will **only** pull these messages once per subscription. For example, if you ran this command again you would receive 10 different messages in your output. To receive more than one message at a time, pass the `--limit` flag. For more details, see the [gcloud pubsub subscriptions pull](https://cloud.google.com/sdk/gcloud/reference/pubsub/subscriptions/pull) documentation.
+ This command will **only** pull these messages once per subscription. For example, if you ran this command again you would receive 10 different messages in your output. To receive more than one message at a time, pass the `--limit` flag. For more details, refer to the [gcloud pubsub subscriptions pull](https://cloud.google.com/sdk/gcloud/reference/pubsub/subscriptions/pull) documentation.
- ~~~
- ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬──────────────────┬─────────────────────────────────────────────────────────┬────────────┬──────────────────┐
- │ DATA │ MESSAGE_ID │ ORDERING_KEY │ ATTRIBUTES │ DELIVERY_ATTEMPT │
- ├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────┼─────────────────────────────────────────────────────────┼────────────┼──────────────────┤
- │ {"key":["boston","40ef7cfa-5e16-4bd3-9e14-2f23407a66df"],"value":{"after":{"address":"14980 Gentry Plains Apt. 64","city":"boston","credit_card":"2466765790","id":"40ef7cfa-5e16-4bd3-9e14-2f23407a66df","name":"Vickie Fitzpatrick"}},"topic":"movr-users"} │ 4466153049158588 │ ["boston", "40ef7cfa-5e16-4bd3-9e14-2f23407a66df"] │ │ │
- │ {"key":["los angeles","947ae147-ae14-4800-8000-00000000001d"],"value":{"after":{"address":"35627 Chelsey Tunnel Suite 94","city":"los angeles","credit_card":"2099932769","id":"947ae147-ae14-4800-8000-00000000001d","name":"Kenneth Barnes"}},"topic":"movr-users"} │ 4466144577818136 │ ["los angeles", "947ae147-ae14-4800-8000-00000000001d"] │ │ │
- │ {"key":["amsterdam","c28f5c28-f5c2-4000-8000-000000000026"],"value":{"after":{"address":"14729 Karen Radial","city":"amsterdam","credit_card":"5844236997","id":"c28f5c28-f5c2-4000-8000-000000000026","name":"Maria Weber"}},"topic":"movr-users"} │ 4466151194002912 │ ["amsterdam", "c28f5c28-f5c2-4000-8000-000000000026"] │ │ │
- │ {"key":["new york","6c8ab772-584a-439d-b7b4-fda37767c74c"],"value":{"after":{"address":"34196 Roger Row Suite 6","city":"new york","credit_card":"3117945420","id":"6c8ab772-584a-439d-b7b4-fda37767c74c","name":"James Lang"}},"topic":"movr-users"} │ 4466147099992681 │ ["new york", "6c8ab772-584a-439d-b7b4-fda37767c74c"] │ │ │
- │ {"key":["boston","c56dab0a-63e7-4fbb-a9af-54362c481c41"],"value":{"after":{"address":"83781 Ross Overpass","city":"boston","credit_card":"7044597874","id":"c56dab0a-63e7-4fbb-a9af-54362c481c41","name":"Mark Butler"}},"topic":"movr-users"} │ 4466150752442731 │ ["boston", "c56dab0a-63e7-4fbb-a9af-54362c481c41"] │ │ │
- │ {"key":["amsterdam","f27e09d5-d7cd-4f88-8b65-abb910036f45"],"value":{"after":{"address":"77153 Donald Road Apt. 62","city":"amsterdam","credit_card":"7531160744","id":"f27e09d5-d7cd-4f88-8b65-abb910036f45","name":"Lisa Sandoval"}},"topic":"movr-users"} │ 4466147182359256 │ ["amsterdam", "f27e09d5-d7cd-4f88-8b65-abb910036f45"] │ │ │
- │ {"key":["new york","46d200c0-6924-4cc7-b3c9-3398997acb84"],"value":{"after":{"address":"92843 Carlos Grove","city":"new york","credit_card":"8822366402","id":"46d200c0-6924-4cc7-b3c9-3398997acb84","name":"Mackenzie Malone"}},"topic":"movr-users"} │ 4466142864542016 │ ["new york", "46d200c0-6924-4cc7-b3c9-3398997acb84"] │ │ │
- │ {"key":["boston","52ecbb26-0eab-4e0b-a160-90caa6a7d350"],"value":{"after":{"address":"95044 Eric Corner Suite 33","city":"boston","credit_card":"3982363300","id":"52ecbb26-0eab-4e0b-a160-90caa6a7d350","name":"Brett Porter"}},"topic":"movr-users"} │ 4466152539161631 │ ["boston", "52ecbb26-0eab-4e0b-a160-90caa6a7d350"] │ │ │
- │ {"key":["amsterdam","ae147ae1-47ae-4800-8000-000000000022"],"value":{"after":{"address":"88194 Angela Gardens Suite 94","city":"amsterdam","credit_card":"4443538758","id":"ae147ae1-47ae-4800-8000-000000000022","name":"Tyler Dalton"}},"topic":"movr-users"} │ 4466151398997150 │ ["amsterdam", "ae147ae1-47ae-4800-8000-000000000022"] │ │ │
- │ {"key":["paris","dc28f5c2-8f5c-4800-8000-00000000002b"],"value":{"after":{"address":"2058 Rodriguez Stream","city":"paris","credit_card":"9584502537","id":"dc28f5c2-8f5c-4800-8000-00000000002b","name":"Tony Ortiz"}},"topic":"movr-users"} │ 4466146372222914 │ ["paris", "dc28f5c2-8f5c-4800-8000-00000000002b"] │ │ │
- └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────┴─────────────────────────────────────────────────────────┴────────────┴──────────────────┘
- ~~~
+ ~~~
+ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────────┬──────────────┬────────────┬──────────────────┬────────────┐
+ │ DATA │ MESSAGE_ID │ ORDERING_KEY │ ATTRIBUTES │ DELIVERY_ATTEMPT │ ACK_STATUS │
+ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────┼──────────────┼────────────┼──────────────────┼────────────┤
+ │ {"Key":["amsterdam", "09ee2856-5856-40c4-85d3-7d65bed978f0"],"Value":{"after": {"address": "84579 Peter Divide Apt. 47", "city": "amsterdam", "credit_card": "0100007510", "id": "09ee2856-5856-40c4-85d3-7d65bed978f0", "name": "Timothy Jackson"}},"Topic":"movr-users"} │ 11249015757941393 │ │ │ │ SUCCESS │
+ │ {"Key":["new york", "8803ab9e-5001-4994-a2e6-68d587f95f1d"],"Value":{"after": {"address": "37546 Andrew Roads Apt. 68", "city": "new york", "credit_card": "4731676650", "id": "8803ab9e-5001-4994-a2e6-68d587f95f1d", "name": "Susan Harrington"}},"Topic":"movr-users"} │ 11249015757941394 │ │ │ │ SUCCESS │
+ │ {"Key":["seattle", "32e27201-ca0d-4a0c-ada2-fbf47f6a4711"],"Value":{"after": {"address": "86725 Stephen Gardens", "city": "seattle", "credit_card": "3639690115", "id": "32e27201-ca0d-4a0c-ada2-fbf47f6a4711", "name": "Brad Hill"}},"Topic":"movr-users"} │ 11249015757941395 │ │ │ │ SUCCESS │
+ │ {"Key":["san francisco", "27b03637-ef9f-49a0-9b58-b16d7a9e34f4"],"Value":{"after": {"address": "85467 Tiffany Field", "city": "san francisco", "credit_card": "0016125921", "id": "27b03637-ef9f-49a0-9b58-b16d7a9e34f4", "name": "Mark Garcia"}},"Topic":"movr-users"} │ 11249015757941396 │ │ │ │ SUCCESS │
+ │ {"Key":["rome", "982e1863-88d4-49cb-adee-0a35baae7e0b"],"Value":{"after": {"address": "54918 Sutton Isle Suite 74", "city": "rome", "credit_card": "6015706174", "id": "982e1863-88d4-49cb-adee-0a35baae7e0b", "name": "Kimberly Nichols"}},"Topic":"movr-users"} │ 11249015757941397 │ │ │ │ SUCCESS │
+ │ {"Key":["washington dc", "7b298994-7b12-414c-90ef-353c7105f012"],"Value":{"after": {"address": "45205 Romero Ford Apt. 86", "city": "washington dc", "credit_card": "3519400314", "id": "7b298994-7b12-414c-90ef-353c7105f012", "name": "Taylor Bullock"}},"Topic":"movr-users"} │ 11249015757941398 │ │ │ │ SUCCESS │
+ │ {"Key":["boston", "4f012f57-577b-4853-b5ab-0d79d0df1369"],"Value":{"after": {"address": "15765 Vang Ramp", "city": "boston", "credit_card": "6747715133", "id": "4f012f57-577b-4853-b5ab-0d79d0df1369", "name": "Ryan Garcia"}},"Topic":"movr-users"} │ 11249015757941399 │ │ │ │ SUCCESS │
+ │ {"Key":["seattle", "9ba85917-5545-4674-8ab2-497fa47ac00f"],"Value":{"after": {"address": "24354 Whitney Lodge", "city": "seattle", "credit_card": "8642661685", "id": "9ba85917-5545-4674-8ab2-497fa47ac00f", "name": "Donald Walsh"}},"Topic":"movr-users"} │ 11249015757941400 │ │ │ │ SUCCESS │
+ │ {"Key":["seattle", "98312fb3-230e-412d-9b22-074ec97329ff"],"Value":{"after": {"address": "72777 Carol Shoal", "city": "seattle", "credit_card": "7789799678", "id": "98312fb3-230e-412d-9b22-074ec97329ff", "name": "Christopher Davis"}},"Topic":"movr-users"} │ 11249015757941401 │ │ │ │ SUCCESS │
+ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────────┴──────────────┴────────────┴──────────────────┴────────────┘
+ ~~~
## Create a changefeed connected to a cloud storage sink
diff --git a/src/current/v24.1/changefeed-sinks.md b/src/current/v24.1/changefeed-sinks.md
index 7b771659dbd..0df4882978c 100644
--- a/src/current/v24.1/changefeed-sinks.md
+++ b/src/current/v24.1/changefeed-sinks.md
@@ -239,6 +239,10 @@ The following parameters are also needed, but are **set by default** in Cockroac
Changefeeds can deliver messages to a Google Cloud Pub/Sub sink, which is integrated with Google Cloud Platform.
+{{site.data.alerts.callout_info}}
+Since CockroachDB v23.2, the `changefeed.new_pubsub_sink_enabled` cluster setting is enabled by default, which provides improved throughput. Without this cluster setting enabled, changefeeds emit JSON-encoded events with the top-level message fields all lowercase. With `changefeed.new_pubsub_sink_enabled`, the top-level fields are capitalized. For more details, refer to the [Pub/Sub sink messages](#pub-sub-sink-messages) section.
+{{site.data.alerts.end}}
+
A Pub/Sub sink URI follows this example:
~~~
@@ -313,23 +317,40 @@ pubsub_sink_config = '{ "Flush": {"Messages": 100, "Frequency": "5s"}, "Retry":
### Pub/Sub sink messages
+The `changefeed.new_pubsub_sink_enabled` cluster setting is enabled by default, which provides improved changefeed throughput peformance. With `changefeed.new_pubsub_sink_enabled` enabled, the changefeed JSON-encoded message format have top-level fields that are capitalized:
+
+~~~
+{Key: ..., Value: ..., Topic: ...}
+~~~
+
+{{site.data.alerts.callout_danger}}
+By default in v23.2, the capitalization of top-level fields in the message has changed. Before upgrading to CockroachDB v23.2 and later, you may need to reconfigure downstream systems to parse the new message format.
+{{site.data.alerts.end}}
+
+With `changefeed.new_pubsub_sink_enabled` set to `false`, changefeeds emit JSON messages with the top-level fields all lowercase:
+
+~~~
+{key: ..., value: ..., topic: ...}
+~~~
+
+If `changefeed.new_pubsub_sink_enabled` is set to `false`, changefeeds will not benefit from the improved throughput performance that this setting enables.
+
The following shows the default JSON messages for a changefeed emitting to Pub/Sub. These changefeed messages were emitted as part of the [Create a changefeed connected to a Google Cloud Pub/Sub sink]({% link {{ page.version.version }}/changefeed-examples.md %}#create-a-changefeed-connected-to-a-google-cloud-pub-sub-sink) example:
~~~
-┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬──────────────────┬─────────────────────────────────────────────────────────┬────────────┬──────────────────┐
-│ DATA │ MESSAGE_ID │ ORDERING_KEY │ ATTRIBUTES │ DELIVERY_ATTEMPT │
-├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────┼─────────────────────────────────────────────────────────┼────────────┼──────────────────┤
-│ {"key":["boston","40ef7cfa-5e16-4bd3-9e14-2f23407a66df"],"value":{"after":{"address":"14980 Gentry Plains Apt. 64","city":"boston","credit_card":"2466765790","id":"40ef7cfa-5e16-4bd3-9e14-2f23407a66df","name":"Vickie Fitzpatrick"}},"topic":"movr-users"} │ 4466153049158588 │ ["boston", "40ef7cfa-5e16-4bd3-9e14-2f23407a66df"] │ │ │
-│ {"key":["los angeles","947ae147-ae14-4800-8000-00000000001d"],"value":{"after":{"address":"35627 Chelsey Tunnel Suite 94","city":"los angeles","credit_card":"2099932769","id":"947ae147-ae14-4800-8000-00000000001d","name":"Kenneth Barnes"}},"topic":"movr-users"} │ 4466144577818136 │ ["los angeles", "947ae147-ae14-4800-8000-00000000001d"] │ │ │
-│ {"key":["amsterdam","c28f5c28-f5c2-4000-8000-000000000026"],"value":{"after":{"address":"14729 Karen Radial","city":"amsterdam","credit_card":"5844236997","id":"c28f5c28-f5c2-4000-8000-000000000026","name":"Maria Weber"}},"topic":"movr-users"} │ 4466151194002912 │ ["amsterdam", "c28f5c28-f5c2-4000-8000-000000000026"] │ │ │
-│ {"key":["new york","6c8ab772-584a-439d-b7b4-fda37767c74c"],"value":{"after":{"address":"34196 Roger Row Suite 6","city":"new york","credit_card":"3117945420","id":"6c8ab772-584a-439d-b7b4-fda37767c74c","name":"James Lang"}},"topic":"movr-users"} │ 4466147099992681 │ ["new york", "6c8ab772-584a-439d-b7b4-fda37767c74c"] │ │ │
-│ {"key":["boston","c56dab0a-63e7-4fbb-a9af-54362c481c41"],"value":{"after":{"address":"83781 Ross Overpass","city":"boston","credit_card":"7044597874","id":"c56dab0a-63e7-4fbb-a9af-54362c481c41","name":"Mark Butler"}},"topic":"movr-users"} │ 4466150752442731 │ ["boston", "c56dab0a-63e7-4fbb-a9af-54362c481c41"] │ │ │
-│ {"key":["amsterdam","f27e09d5-d7cd-4f88-8b65-abb910036f45"],"value":{"after":{"address":"77153 Donald Road Apt. 62","city":"amsterdam","credit_card":"7531160744","id":"f27e09d5-d7cd-4f88-8b65-abb910036f45","name":"Lisa Sandoval"}},"topic":"movr-users"} │ 4466147182359256 │ ["amsterdam", "f27e09d5-d7cd-4f88-8b65-abb910036f45"] │ │ │
-│ {"key":["new york","46d200c0-6924-4cc7-b3c9-3398997acb84"],"value":{"after":{"address":"92843 Carlos Grove","city":"new york","credit_card":"8822366402","id":"46d200c0-6924-4cc7-b3c9-3398997acb84","name":"Mackenzie Malone"}},"topic":"movr-users"} │ 4466142864542016 │ ["new york", "46d200c0-6924-4cc7-b3c9-3398997acb84"] │ │ │
-│ {"key":["boston","52ecbb26-0eab-4e0b-a160-90caa6a7d350"],"value":{"after":{"address":"95044 Eric Corner Suite 33","city":"boston","credit_card":"3982363300","id":"52ecbb26-0eab-4e0b-a160-90caa6a7d350","name":"Brett Porter"}},"topic":"movr-users"} │ 4466152539161631 │ ["boston", "52ecbb26-0eab-4e0b-a160-90caa6a7d350"] │ │ │
-│ {"key":["amsterdam","ae147ae1-47ae-4800-8000-000000000022"],"value":{"after":{"address":"88194 Angela Gardens Suite 94","city":"amsterdam","credit_card":"4443538758","id":"ae147ae1-47ae-4800-8000-000000000022","name":"Tyler Dalton"}},"topic":"movr-users"} │ 4466151398997150 │ ["amsterdam", "ae147ae1-47ae-4800-8000-000000000022"] │ │ │
-│ {"key":["paris","dc28f5c2-8f5c-4800-8000-00000000002b"],"value":{"after":{"address":"2058 Rodriguez Stream","city":"paris","credit_card":"9584502537","id":"dc28f5c2-8f5c-4800-8000-00000000002b","name":"Tony Ortiz"}},"topic":"movr-users"} │ 4466146372222914 │ ["paris", "dc28f5c2-8f5c-4800-8000-00000000002b"] │ │ │
-└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────┴─────────────────────────────────────────────────────────┴────────────┴──────────────────┘
+┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────────┬──────────────┬────────────┬──────────────────┬────────────┐
+│ DATA │ MESSAGE_ID │ ORDERING_KEY │ ATTRIBUTES │ DELIVERY_ATTEMPT │ ACK_STATUS │
+├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────┼──────────────┼────────────┼──────────────────┼────────────┤
+│ {"Key":["amsterdam", "09ee2856-5856-40c4-85d3-7d65bed978f0"],"Value":{"after": {"address": "84579 Peter Divide Apt. 47", "city": "amsterdam", "credit_card": "0100007510", "id": "09ee2856-5856-40c4-85d3-7d65bed978f0", "name": "Timothy Jackson"}},"Topic":"users"} │ 11249015757941393 │ │ │ │ SUCCESS │
+│ {"Key":["new york", "8803ab9e-5001-4994-a2e6-68d587f95f1d"],"Value":{"after": {"address": "37546 Andrew Roads Apt. 68", "city": "new york", "credit_card": "4731676650", "id": "8803ab9e-5001-4994-a2e6-68d587f95f1d", "name": "Susan Harrington"}},"Topic":"users"} │ 11249015757941394 │ │ │ │ SUCCESS │
+│ {"Key":["seattle", "32e27201-ca0d-4a0c-ada2-fbf47f6a4711"],"Value":{"after": {"address": "86725 Stephen Gardens", "city": "seattle", "credit_card": "3639690115", "id": "32e27201-ca0d-4a0c-ada2-fbf47f6a4711", "name": "Brad Hill"}},"Topic":"users"} │ 11249015757941395 │ │ │ │ SUCCESS │
+│ {"Key":["san francisco", "27b03637-ef9f-49a0-9b58-b16d7a9e34f4"],"Value":{"after": {"address": "85467 Tiffany Field", "city": "san francisco", "credit_card": "0016125921", "id": "27b03637-ef9f-49a0-9b58-b16d7a9e34f4", "name": "Mark Garcia"}},"Topic":"users"} │ 11249015757941396 │ │ │ │ SUCCESS │
+│ {"Key":["rome", "982e1863-88d4-49cb-adee-0a35baae7e0b"],"Value":{"after": {"address": "54918 Sutton Isle Suite 74", "city": "rome", "credit_card": "6015706174", "id": "982e1863-88d4-49cb-adee-0a35baae7e0b", "name": "Kimberly Nichols"}},"Topic":"users"} │ 11249015757941397 │ │ │ │ SUCCESS │
+│ {"Key":["washington dc", "7b298994-7b12-414c-90ef-353c7105f012"],"Value":{"after": {"address": "45205 Romero Ford Apt. 86", "city": "washington dc", "credit_card": "3519400314", "id": "7b298994-7b12-414c-90ef-353c7105f012", "name": "Taylor Bullock"}},"Topic":"users"} │ 11249015757941398 │ │ │ │ SUCCESS │
+│ {"Key":["boston", "4f012f57-577b-4853-b5ab-0d79d0df1369"],"Value":{"after": {"address": "15765 Vang Ramp", "city": "boston", "credit_card": "6747715133", "id": "4f012f57-577b-4853-b5ab-0d79d0df1369", "name": "Ryan Garcia"}},"Topic":"users"} │ 11249015757941399 │ │ │ │ SUCCESS │
+│ {"Key":["seattle", "9ba85917-5545-4674-8ab2-497fa47ac00f"],"Value":{"after": {"address": "24354 Whitney Lodge", "city": "seattle", "credit_card": "8642661685", "id": "9ba85917-5545-4674-8ab2-497fa47ac00f", "name": "Donald Walsh"}},"Topic":"users"} │ 11249015757941400 │ │ │ │ SUCCESS │
+│ {"Key":["seattle", "98312fb3-230e-412d-9b22-074ec97329ff"],"Value":{"after": {"address": "72777 Carol Shoal", "city": "seattle", "credit_card": "7789799678", "id": "98312fb3-230e-412d-9b22-074ec97329ff", "name": "Christopher Davis"}},"Topic":"users"} │ 11249015757941401 │ │ │ │ SUCCESS │
+└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────────┴──────────────┴────────────┴──────────────────┴────────────┘
~~~
{% include {{ page.version.version }}/cdc/note-changefeed-message-page.md %}
diff --git a/src/current/v24.1/create-changefeed.md b/src/current/v24.1/create-changefeed.md
index 6a084278824..0ec6017731a 100644
--- a/src/current/v24.1/create-changefeed.md
+++ b/src/current/v24.1/create-changefeed.md
@@ -102,6 +102,8 @@ Example of a Google Cloud Pub/Sub sink URI:
'gcpubsub://{project name}?region={region}&topic_name={topic name}&AUTH=specified&CREDENTIALS={base64-encoded key}'
~~~
+In CockroachDB v23.2 and later, the `changefeed.new_pubsub_sink_enabled` cluster setting is enabled by default, which provides improved throughput. For details on the changes to the message format, refer to [Pub/Sub sink messages]({% link {{ page.version.version }}/changefeed-sinks.md %}#pub-sub-sink-messages).
+
[Use Cloud Storage for Bulk Operations]({% link {{ page.version.version }}/cloud-storage-authentication.md %}) explains the requirements for the authentication parameter with `specified` or `implicit`. Refer to [Changefeed Sinks]({% link {{ page.version.version }}/changefeed-sinks.md %}#google-cloud-pub-sub) for further consideration.
#### Kafka
diff --git a/src/current/v24.1/cutover-replication.md b/src/current/v24.1/cutover-replication.md
index 731a020b99f..ec0cb7e14c4 100644
--- a/src/current/v24.1/cutover-replication.md
+++ b/src/current/v24.1/cutover-replication.md
@@ -5,10 +5,6 @@ toc: true
docs_area: manage
---
-{{site.data.alerts.callout_info}}
-{% include feature-phases/preview.md %}
-{{site.data.alerts.end}}
-
[**Physical cluster replication (PCR)**]({% link {{ page.version.version }}/physical-cluster-replication-overview.md %}) allows you to cut over from the active primary cluster to the passive standby cluster that has ingested replicated data. When you complete the replication, it will stop the stream of new data, reset the standby virtual cluster to a point in time where all ingested data is consistent, and then mark the standby virtual cluster as ready to accept traffic.
The cutover is a two-step process on the standby cluster:
@@ -54,6 +50,10 @@ SHOW VIRTUAL CLUSTER main WITH REPLICATION STATUS;
(1 row)
~~~
+{{site.data.alerts.callout_success}}
+You can view the [**Replication Lag** graph]({% link {{ page.version.version }}/ui-physical-cluster-replication-dashboard.md %}#replication-lag) in the standby cluster's DB Console.
+{{site.data.alerts.end}}
+
Run the following from the standby cluster's SQL shell to start the cutover:
{% include_cached copy-clipboard.html %}
@@ -304,13 +304,7 @@ This section illustrates the steps to cut back to the original primary cluster f
SET CLUSTER SETTING server.controller.default_target_cluster='{cluster_a}';
~~~
-<<<<<<< HEAD
-At this point, **Cluster A** is once again the primary and **Cluster B** is once again the standby. The clusters are entirely independent. To direct application traffic to the primary (**Cluster A**), you will need to use your own network load balancers, DNS servers, or other network configuration to direct application traffic to **Cluster A**. To manage replicated jobs on the promoted standby, refer to [Job management](#job-management).
-
-To enable physical cluster replication again, from the primary to the standby (or a completely different cluster), refer to [Set Up Physical Cluster Replication]({% link {{ page.version.version }}/set-up-physical-cluster-replication.md %}).
-=======
At this point, **Cluster A** is once again the primary and **Cluster B** is once again the standby. The clusters are entirely independent. To direct application traffic to the primary (**Cluster A**), you will need to use your own network load balancers, DNS servers, or other network configuration to direct application traffic to **Cluster A**. To enable PCR again, from the primary to the standby (or a completely different cluster), refer to [Set Up Physical Cluster Replication]({% link {{ page.version.version }}/set-up-physical-cluster-replication.md %}).
->>>>>>> c9b9928e7 (Bring physical cluster replication into GA v24.1)
## See also
diff --git a/src/current/v24.1/inverted-indexes.md b/src/current/v24.1/inverted-indexes.md
index f65b54e1c10..6ca9524104b 100644
--- a/src/current/v24.1/inverted-indexes.md
+++ b/src/current/v24.1/inverted-indexes.md
@@ -97,7 +97,7 @@ If a query contains a filter against an indexed `JSONB` or `ARRAY` column that u
In most cases CockroachDB selects the index it calculates will scan the fewest rows (i.e., the fastest). Cases where CockroachDB will use multiple indexes include certain queries that use disjunctions (i.e., predicates with `OR`), as well as [zigzag joins]({% link {{ page.version.version }}/cost-based-optimizer.md %}#zigzag-joins) for some other queries. To learn how to use the [`EXPLAIN`]({% link {{ page.version.version }}/explain.md %}) statement for your query to see which index is being used, see [Index Selection in CockroachDB](https://www.cockroachlabs.com/blog/index-selection-cockroachdb-2/).
-To override CockroachDB's index selection, you can also force a query to use [a specific index]({% link {{ page.version.version }}/table-expressions.md %}#force-index-selection) (also known as "index hinting") or use [an inverted join hint]({% link {{ page.version.version }}/cost-based-optimizer.md %}#supported-join-algorithms).
+To override CockroachDB's index selection, you can use an index hint to force a query to use [a specific index]({% link {{ page.version.version }}/table-expressions.md %}#force-index-selection), including [inverted indexes]({% link {{ page.version.version }}/table-expressions.md %}#force-inverted-index-scan) and [partial GIN indexes]({% link {{ page.version.version }}/table-expressions.md %}#force-partial-gin-index-scan); or use [an inverted join hint]({% link {{ page.version.version }}/cost-based-optimizer.md %}#supported-join-algorithms).
### Storage
diff --git a/src/current/v24.1/known-limitations.md b/src/current/v24.1/known-limitations.md
index 0b83770d0c3..2dae998ab10 100644
--- a/src/current/v24.1/known-limitations.md
+++ b/src/current/v24.1/known-limitations.md
@@ -2,7 +2,7 @@
title: Known Limitations in CockroachDB v24.1
summary: Learn about newly identified limitations in CockroachDB as well as unresolved limitations identified in earlier releases.
toc: true
-keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes
+keywords: limitations, known limitations, unsupported features, PostgreSQL compatibility
docs_area: releases
---
@@ -12,9 +12,24 @@ docs_area: releases
This section describes newly identified limitations in CockroachDB {{ page.version.version }}.
+{% comment %}
{{site.data.alerts.callout_info}}
Limitations will be added as they are discovered.
{{site.data.alerts.end}}
+{% endcomment %}
+
+### PL/pgSQL
+
+- It is not possible to use a variable as a target more than once in the same `INTO` clause. For example, `SELECT 1, 2 INTO x, x;`. [#121605](https://github.com/cockroachdb/cockroach/issues/121605)
+- PLpgSQL variable declarations cannot inherit the type of a table row or column using `%TYPE` or `%ROWTYPE` syntax. [#114676](https://github.com/cockroachdb/cockroach/issues/114676)
+
+### UDFs and stored procedures
+
+- Routines cannot be invoked with named arguments, e.g., `SELECT foo(a => 1, b => 2);` or `SELECT foo(b := 1, a := 2);`. [#122264](https://github.com/cockroachdb/cockroach/issues/122264)
+- Routines cannot be created if they reference temporary tables. [#121375](https://github.com/cockroachdb/cockroach/issues/121375)
+- Routines cannot be created with unnamed `INOUT` parameters. For example, `CREATE PROCEDURE p(INOUT INT) AS $$ BEGIN NULL; END; $$ LANGUAGE PLpgSQL;`. [#121251](https://github.com/cockroachdb/cockroach/issues/121251)
+- Routines cannot be created if they return fewer columns than declared. For example, `CREATE FUNCTION f(OUT sum INT, INOUT a INT, INOUT b INT) LANGUAGE SQL AS $$ SELECT (a + b, b); $$;`. [#121247](https://github.com/cockroachdb/cockroach/issues/121247)
+- A `RECORD`-returning UDF cannot be created without a `RETURN` statement in the root block, which would restrict the wildcard type to a concrete one. [#122945](https://github.com/cockroachdb/cockroach/issues/122945)
## Limitations from {{ previous_version }} and earlier
@@ -98,17 +113,15 @@ By default, CockroachDB orders `NULL`s before all other values. For compatibilit
### Functions and procedures
-#### PL/pgSQL feature support
-
-{% include {{ page.version.version }}/known-limitations/plpgsql-feature-limitations.md %}
-
-#### PL/pgSQL data types
+#### PL/pgSQL support
-{% include {{ page.version.version }}/known-limitations/plpgsql-datatype-limitations.md %}
+{% include {{ page.version.version }}/known-limitations/plpgsql-limitations.md %}
#### UDF and stored procedure support
-{% include {{ page.version.version }}/known-limitations/udf-stored-proc-limitations.md %}
+{% include {{ page.version.version }}/known-limitations/routine-limitations.md %}
+{% include {{ page.version.version }}/known-limitations/stored-proc-limitations.md %}
+{% include {{ page.version.version }}/known-limitations/udf-limitations.md %}
### Transactions
diff --git a/src/current/v24.1/licensing-faqs.md b/src/current/v24.1/licensing-faqs.md
index 1b9bd2a5abd..ae21c3cc79c 100644
--- a/src/current/v24.1/licensing-faqs.md
+++ b/src/current/v24.1/licensing-faqs.md
@@ -36,6 +36,8 @@ For each BSL release all associated alpha, beta, major, and minor (point) releas
CockroachDB version | License | Converts to Apache 2.0
--------------------|---------|----------------------------
+24.1 | Business Source License | May 20, 2027
+23.1 | Business Source License | Feb 5, 2027
23.1 | Business Source License | May 16, 2026
22.2 | Business Source License | Dec 6, 2025
22.1 | Business Source License | May 24, 2025
diff --git a/src/current/v24.1/live-migration-service.md b/src/current/v24.1/live-migration-service.md
index c3b22310421..46befbfc939 100644
--- a/src/current/v24.1/live-migration-service.md
+++ b/src/current/v24.1/live-migration-service.md
@@ -9,7 +9,7 @@ docs_area: migrate
{% include feature-phases/preview.md %}
{{site.data.alerts.end}}
-MOLT LMS (Live Migration Service) is used to perform a [live migration]({% link {{ page.version.version }}/migration-overview.md %}#minimal-downtime) to CockroachDB.
+MOLT LMS (Live Migration Service) is used during a [live migration]({% link {{ page.version.version }}/migration-overview.md %}#minimal-downtime) to CockroachDB.
The LMS is a self-hosted, horizontally scalable proxy that routes traffic between an application, a source database, and a target CockroachDB database. You use the LMS to control which database, as the "source of truth", is serving reads and writes to an application. You can optionally configure the LMS to [shadow production traffic](#shadowing-modes) from the source database and validate the query results on CockroachDB. When you have sufficiently tested your application and are confident with its consistency and performance on CockroachDB, you use the LMS to [perform the cutover](#perform-a-cutover) to CockroachDB.
diff --git a/src/current/v24.1/migrate-from-mysql.md b/src/current/v24.1/migrate-from-mysql.md
index 18150b7df1e..71485a01e5b 100644
--- a/src/current/v24.1/migrate-from-mysql.md
+++ b/src/current/v24.1/migrate-from-mysql.md
@@ -158,7 +158,7 @@ Use the [Schema Conversion Tool](https://www.cockroachlabs.com/docs/cockroachclo
Click **Save**.
- This is a workaround to prevent [data validation](#step-3-validate-the-migrated-data) from failing due to collation mismatches. For more details, see the [MOLT Verify] ({% link {{ page.version.version }}/molt-verify.md %}#limitations) documentation.
+ This is a workaround to prevent [data validation](#step-3-validate-the-migrated-data) from failing due to collation mismatches. For more details, see the [MOLT Verify] ({% link {{ page.version.version }}/molt-verify.md %}#known-limitations) documentation.
1. Click [**Migrate Schema**](https://www.cockroachlabs.com/docs/cockroachcloud/migrations-page?filters=mysql#migrate-the-schema) to create a new {{ site.data.products.serverless }} cluster with the converted schema. Name the database `world`.
diff --git a/src/current/v24.1/migrate-from-oracle.md b/src/current/v24.1/migrate-from-oracle.md
index fe0dc71455d..4d47ffebdac 100644
--- a/src/current/v24.1/migrate-from-oracle.md
+++ b/src/current/v24.1/migrate-from-oracle.md
@@ -6,7 +6,9 @@ docs_area: migrate
---
{{site.data.alerts.callout_danger}}
-The instructions on this page require updates. We currently recommend [using AWS Database Migration Service (DMS) to migrate data]({% link {{ page.version.version }}/aws-dms.md %}) from Oracle to CockroachDB. You can also [migrate from CSV]({% link {{ page.version.version }}/migrate-from-csv.md %}).
+The instructions on this page are outdated. Use the [Schema Conversion Tool]({% link cockroachcloud/migrations-page.md %}?filters=oracle) to convert an Oracle schema into a compatible CockroachDB schema, and a tool such as [AWS Database Migration Service (DMS)]({% link {{ page.version.version }}/aws-dms.md %}) or [Qlik]({% link {{ page.version.version }}/qlik.md %}) to migrate data from Oracle to CockroachDB.
+
+Note that `IMPORT` is deprecated. To move data into CockroachDB, use [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}) or [`COPY FROM`]({% link {{ page.version.version }}/copy-from.md %}).
{{site.data.alerts.end}}
This page has instructions for migrating data from Oracle into CockroachDB by [importing]({% link {{ page.version.version }}/import.md %}) CSV files. Note that `IMPORT` only works for creating new tables. For information on how to add CSV data to existing tables, see [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}).
diff --git a/src/current/v24.1/migration-overview.md b/src/current/v24.1/migration-overview.md
index 3912952a489..3e1372af46a 100644
--- a/src/current/v24.1/migration-overview.md
+++ b/src/current/v24.1/migration-overview.md
@@ -62,17 +62,17 @@ A lift-and-shift approach is the most straightforward. However, it's important t
- *Reduced functionality* takes some, but not all, application functionality offline. For example, you can disable writes but not reads while you migrate the application data, and queue data to be written after completing the migration.
-For an overview of lift-and-shift migrations to CockroachDB, see [Lift and Shift](#lift-and-shift). For considerations and details about the pros and cons of this approach, see [Migration Strategy: Lift and Shift]({% link {{ page.version.version }}/migration-strategy-lift-and-shift.md %}).
+For an overview of lift-and-shift migrations to CockroachDB, see [Lift and Shift](#lift-and-shift).
#### Minimal downtime
-If your application cannot tolerate downtime, then you should aim for a "zero-downtime" approach. "Zero" means that downtime is reduced to either an absolute minimum or zero, such that users do not notice the migration.
+If your application cannot tolerate downtime, then you should aim for a "zero-downtime" approach. This reduces downtime to an absolute minimum, such that users do not notice the migration.
The minimum possible downtime depends on whether you can tolerate inconsistency in the migrated data:
-- *Consistent* migrations reduce downtime to an absolute minimum (i.e., from 30 seconds to sub-seconds) while keeping data synchronized between the source database and CockroachDB. **Consistency requires downtime.** In this approach, downtime occurs right before [cutover](#cutover-strategy), as you drain the remaining transactions from the source database to CockroachDB.
+- Migrations performed using *consistent cutover* reduce downtime to an absolute minimum (i.e., seconds or sub-seconds) while keeping data synchronized between the source database and CockroachDB. **Consistency requires downtime.** In this approach, downtime occurs right before [cutover](#cutover-strategy), as you drain the remaining transactions from the source database to CockroachDB.
-- *Inconsistent* migrations can reduce downtime to zero. These require the most preparation, and typically allow read/write traffic to both databases for at least a small amount of time, thereby sacrificing consistency for availability. {% comment %}You can use the CockroachDB Live Migration Service (MOLT LMS) to run application queries simultaneously on your source database and CockroachDB.{% endcomment %} Without stopping application traffic, you perform an immediate [cutover](#cutover-strategy), while assuming that some writes will not be replicated to CockroachDB. You may want to manually reconcile these data inconsistencies after switching over.
+- Migrations performed using *immediate cutover* can reduce downtime to zero. These require the most preparation, and typically allow read/write traffic to both databases for at least a short period of time, sacrificing consistency for availability. {% comment %}You can use the CockroachDB Live Migration Service (MOLT LMS) to run application queries simultaneously on your source database and CockroachDB.{% endcomment %} Without stopping application traffic, you perform an **immediate** [cutover](#cutover-strategy), while assuming that some writes will not be replicated to CockroachDB. You may want to manually reconcile these data inconsistencies after switching over.
For an overview of zero-downtime migrations to CockroachDB, see [Zero Downtime](#zero-downtime). {% comment %}For details, see [Migration Strategy: Zero Downtime](migration-strategy-zero-downtime).{% endcomment %}
@@ -245,9 +245,11 @@ Then import the converted schema to a CockroachDB cluster:
Before moving data, Cockroach Labs recommends [dropping any indexes]({% link {{ page.version.version }}/drop-index.md %}) on the CockroachDB database. The indexes can be [recreated]({% link {{ page.version.version }}/create-index.md %}) after the data is loaded. Doing so will optimize performance.
{{site.data.alerts.end}}
-After [converting the schema](#convert-the-schema), load your data into CockroachDB so that you can [test your application queries](#validate-queries). Then use one of the following methods to migrate the data (you may need to use additional tooling to extract and/or convert the data to an appropriate file format):
+After [converting the schema](#convert-the-schema), load your data into CockroachDB so that you can [test your application queries](#validate-queries). Then use [MOLT Fetch]({% link {{ page.version.version }}/molt-fetch.md %}) to move the source data to CockroachDB.
-- {% include {{ page.version.version }}/migration/load-data-import-into.md %} Typically, initial data loading during a database migration will not be running concurrently with application traffic, so the fact that `IMPORT INTO` takes the table offline may not have any observable availability impact.
+Alternatively, you can use one of the following methods to migrate the data. Additional tooling may be required to extract or convert the data to a supported file format.
+
+- {% include {{ page.version.version }}/migration/load-data-import-into.md %} Typically during a migration, data is initially loaded before foreground application traffic begins to be served, so the impact of taking the table offline when running `IMPORT INTO` may be minimal.
- {% include {{ page.version.version }}/migration/load-data-third-party.md %} Within the tool, you can select the database tables to migrate to the test cluster.
- {% include {{ page.version.version }}/migration/load-data-copy-from.md %}
@@ -259,9 +261,9 @@ Note that CockroachDB defaults to the [`SERIALIZABLE`]({% link {{ page.version.v
##### Shadowing
-You can "shadow" your production workload by executing your source SQL statements on CockroachDB in parallel. The [CockroachDB Live Migration Service (LMS)]({% link {{ page.version.version }}/live-migration-service.md %}) can perform shadowing. You can then [test the queries](#test-query-results-and-performance) on CockroachDB for consistency, performance, and potential issues with the migration.
+You can "shadow" your production workload by executing your source SQL statements on CockroachDB in parallel. You can then [validate the queries](#test-query-results-and-performance) on CockroachDB for consistency, performance, and potential issues with the migration.
-Shadowing may not be necessary or practical for your workload. For example, because transactions are serialized on CockroachDB, this will limit your ability to validate the performance of high-throughput workloads.
+The [CockroachDB Live Migration Service (MOLT LMS)]({% link {{ page.version.version }}/live-migration-service.md %}) can [perform shadowing]({% link {{ page.version.version }}/live-migration-service.md %}#shadowing-modes). This is intended only for [testing](#test-query-results-and-performance) or [performing a dry run](#perform-a-dry-run). Shadowing should **not** be used in production when performing a [live migration](#zero-downtime).
##### Test query results and performance
@@ -310,35 +312,32 @@ Using this method, consistency is achieved by only performing the cutover once a
The following is a high-level overview of the migration steps. For considerations and details about the pros and cons of this approach, see [Migration Strategy: Lift and Shift]({% link {{ page.version.version }}/migration-strategy-lift-and-shift.md %}).
1. Stop application traffic to your source database. **This begins downtime.**
-1. Move data in one of the following ways:
- - {% include {{ page.version.version }}/migration/load-data-import-into.md %}
- - {% include {{ page.version.version }}/migration/load-data-third-party.md %}
- - {% include {{ page.version.version }}/migration/load-data-copy-from.md %}
-1. After the data is migrated, you can use [MOLT Verify]({% link {{ page.version.version }}/molt-verify.md %}) to validate the consistency of the data between the source database and CockroachDB.
+1. Use [MOLT Fetch]({% link {{ page.version.version }}/molt-fetch.md %}) to move the source data to CockroachDB.
+1. After the data is migrated, use [MOLT Verify]({% link {{ page.version.version }}/molt-verify.md %}) to validate the consistency of the data between the source database and CockroachDB.
1. Perform a [cutover](#cutover-strategy) by resuming application traffic, now to CockroachDB.
{% comment %}1. If you want the ability to [roll back](#all-at-once-rollback) the migration, replicate data back to the source database.{% endcomment %}
### Zero Downtime
-Using this method, downtime is minimized by performing the cutover while writes are still being replicated from the source database to CockroachDB. Inconsistencies are resolved through manual reconciliation.
-
-The following is a high-level overview of the migration steps. {% comment %}For details on this migration strategy, see [Migration Strategy: Zero Downtime]({% link {{ page.version.version }}/migration-strategy-lift-and-shift.md %}).{% endcomment %}
+During a "live migration", downtime is minimized by performing the cutover while writes are still being replicated from the source database to CockroachDB. Inconsistencies are resolved through manual reconciliation.
-{% comment %}You can use the CockroachDB Live Migration Service (MOLT LMS) to run application queries simultaneously on your source database and CockroachDB.{% endcomment %}
+The following is a high-level overview of the migration steps. The two approaches are mutually exclusive, and each has [tradeoffs](#minimal-downtime). {% comment %}For details on this migration strategy, see [Migration Strategy: Zero Downtime]({% link {{ page.version.version }}/migration-strategy-lift-and-shift.md %}).{% endcomment %}
To prioritize consistency and minimize downtime:
-1. {% include {{ page.version.version }}/migration/load-data-third-party.md %} Select the tool's option to **replicate ongoing changes** after performing the initial load of data into CockroachDB.
-1. As the data is migrating, you can use [MOLT Verify]({% link {{ page.version.version }}/molt-verify.md %}) to validate the consistency of the data between the source database and CockroachDB.
-1. Once nearly all data from your source database has been moved to CockroachDB (for example, with a <1 second delay or <1000 rows), stop application traffic to your source database. **This begins downtime.**
-1. Wait for replication to CockroachDB to complete.
-1. Perform a [cutover](#cutover-strategy) by resuming application traffic, now to CockroachDB.
+1. Set up the [CockroachDB Live Migration Service (MOLT LMS)]({% link {{ page.version.version }}/live-migration-service.md %}) to proxy for application traffic between your source database and CockroachDB. Do **not** shadow the application traffic.
+1. Use [MOLT Fetch]({% link {{ page.version.version }}/molt-fetch.md %}) to move the source data to CockroachDB. Use the tool to [**replicate ongoing changes**]({% link {{ page.version.version }}/molt-fetch.md %}#replication) after it performs the initial load of data into CockroachDB.
+1. As the data is migrating, use [MOLT Verify]({% link {{ page.version.version }}/molt-verify.md %}) to validate the consistency of the data between the source database and CockroachDB.
+1. After nearly all data from your source database has been moved to CockroachDB (for example, with a <1-second delay or <1000 rows), use MOLT LMS to begin a [*consistent cutover*]({% link {{ page.version.version }}/live-migration-service.md %}#consistent-cutover) and stop application traffic to your source database. **This begins downtime.**
+1. Wait for MOLT Fetch to finish replicating changes to CockroachDB.
+1. Use MOLT LMS to commit the [consistent cutover]({% link {{ page.version.version }}/live-migration-service.md %}#consistent-cutover). This resumes application traffic, now to CockroachDB.
To achieve zero downtime with inconsistency:
-1. {% include {{ page.version.version }}/migration/load-data-third-party.md %} Select the tool's option to replicate ongoing changes after performing the initial load of data into CockroachDB.
+1. Set up the [CockroachDB Live Migration Service (MOLT LMS)]({% link {{ page.version.version }}/live-migration-service.md %}) to proxy for application traffic between your source database and CockroachDB. Use a [shadowing mode]({% link {{ page.version.version }}/live-migration-service.md %}#shadowing-modes) to run application queries simultaneously on your source database and CockroachDB.
+1. Use [MOLT Fetch]({% link {{ page.version.version }}/molt-fetch.md %}) to move the source data to CockroachDB. Use the tool to **replicate ongoing changes** after performing the initial load of data into CockroachDB.
1. As the data is migrating, you can use [MOLT Verify]({% link {{ page.version.version }}/molt-verify.md %}) to validate the consistency of the data between the source database and CockroachDB.
-1. Once nearly all data from your source database has been moved to CockroachDB (for example, with a <1 second delay or <1000 rows), perform a [cutover](#cutover-strategy) by pointing application traffic to CockroachDB.
+1. After nearly all data from your source database has been moved to CockroachDB (for example, with a <1 second delay or <1000 rows), perform an [*immediate cutover*](#cutover-strategy) by pointing application traffic to CockroachDB.
1. Manually reconcile any inconsistencies caused by writes that were not replicated during the cutover.
1. Close the connection to the source database when you are ready to finish the migration.
diff --git a/src/current/v24.1/molt-fetch.md b/src/current/v24.1/molt-fetch.md
index 811262e216f..7d4098b7d23 100644
--- a/src/current/v24.1/molt-fetch.md
+++ b/src/current/v24.1/molt-fetch.md
@@ -5,13 +5,9 @@ toc: true
docs_area: migrate
---
-{{site.data.alerts.callout_info}}
-{% include feature-phases/preview.md %}
-{{site.data.alerts.end}}
-
MOLT Fetch moves data from a source database into CockroachDB as part of a [database migration]({% link {{ page.version.version }}/migration-overview.md %}).
-MOLT Fetch can use `IMPORT INTO` or `COPY FROM` to move the source data to CockroachDB via cloud storage (Google Cloud Storage or Amazon S3), a local file server, or directly without an intermediate store. For details, see [Usage](#usage).
+MOLT Fetch uses [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}) or [`COPY FROM`]({% link {{ page.version.version }}/copy-from.md %}) to move the source data to cloud storage (Google Cloud Storage or Amazon S3), a local file server, or local memory. Once the data is exported, MOLT Fetch loads the data onto a target CockroachDB database. For details, see [Usage](#usage).
## Supported databases
@@ -31,21 +27,27 @@ To install MOLT Fetch, download the binary that matches your system. To download
| Linux | [Download](https://molt.cockroachdb.com/molt/cli/molt-latest.linux-amd64.tgz) | [Download](https://molt.cockroachdb.com/molt/cli/molt-latest.linux-arm64.tgz) |
| Mac | [Download](https://molt.cockroachdb.com/molt/cli/molt-latest.darwin-amd64.tgz) | [Download](https://molt.cockroachdb.com/molt/cli/molt-latest.darwin-arm64.tgz) |
-For previous binaries, see the [MOLT version manifest](https://molt.cockroachdb.com/molt/cli/versions.html). For releases v0.0.6 and earlier, see the [MOLT repository](https://github.com/cockroachdb/molt/releases).
+For previous binaries, refer to the [MOLT version manifest](https://molt.cockroachdb.com/molt/cli/versions.html).
+
+{{site.data.alerts.callout_info}}
+MOLT Fetch is supported on Red Hat Enterprise Linux (RHEL) 9 and above.
+{{site.data.alerts.end}}
## Setup
Complete the following items before using MOLT Fetch:
-- Ensure that the source and target schemas are identical. Tables with mismatching columns may only be partially migrated.
+- Follow the recommendations in [Best practices](#best-practices) and [Security recommendations](#security-recommendations).
-- Ensure that the SQL user running MOLT Fetch has the required privileges to run [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}#required-privileges) or [`COPY FROM`]({% link {{ page.version.version }}/copy-from.md %}#required-privileges) statements, depending on your intended [mode](#fetch-mode).
+- Ensure that the source and target schemas are identical, unless you enable automatic schema creation with the [`'drop-on-target-and-recreate'`](#target-table-handling) option. If you are creating the target schema manually, review the behaviors in [Mismatch handling](#mismatch-handling).
-- To enable the [CDC cursor](#cdc-cursor) for ongoing replication:
+- Ensure that the SQL user running MOLT Fetch has [`SELECT` privileges]({% link {{ page.version.version }}/grant.md %}#supported-privileges) on the source and target CockroachDB databases, along with the required privileges to run [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}#required-privileges) or [`COPY FROM`]({% link {{ page.version.version }}/copy-from.md %}#required-privileges) (depending on the [fetch mode](#fetch-mode)) on CockroachDB, as described on their respective pages.
- - If you are migrating from PostgreSQL, enable logical replication. Set [wal_level](https://www.postgresql.org/docs/current/runtime-config-wal.html) to `logical` in `postgresql.conf` or in the SQL shell.
+- If you plan to use continuous replication (using either [`--ongoing-replication`](#replication) or the [CDC cursor](#cdc-cursor)):
- - If you are migrating from MySQL, enable [GTID](https://dev.mysql.com/doc/refman/8.0/en/replication-options-gtids.html) consistency. Set `gtid-mode` and `enforce-gtid-consistency` to `ON` in `mysql.cnf`, in the SQL shell, or as flags in the `mysql` start command.
+ - If you are migrating from PostgreSQL, enable logical replication. In `postgresql.conf` or in the SQL shell, set [`wal_level`](https://www.postgresql.org/docs/current/runtime-config-wal.html) to `logical`.
+
+ - If you are migrating from MySQL, enable [GTID](https://dev.mysql.com/doc/refman/8.0/en/replication-options-gtids.html) consistency. In `mysql.cnf`, in the SQL shell, or as flags in the `mysql` start command, set `gtid-mode` and `enforce-gtid-consistency` to `ON` and set `binlog_row_metadata` to `full`.
- Percent-encode the connection strings for the source database and [CockroachDB]({% link {{ page.version.version }}/connect-to-the-database.md %}). This ensures that the MOLT tools can parse special characters in your password.
@@ -91,40 +93,131 @@ Complete the following items before using MOLT Fetch:
~~~
- Ensure the Google Cloud Storage bucket is created and accessible to CockroachDB.
-
+
+## Best practices
+
+- To prevent connections from terminating prematurely during data export, set the following to high values on the source database:
+
+ - **Maximum allowed number of connections:** MOLT Fetch can export data across multiple connections. The number of connections it will create is the number of shards ([`--export-concurrency`](#global-flags)) multiplied by the number of tables ([`--table-concurrency`](#global-flags)) being exported concurrently.
+ - **Maximum lifetime of a connection:** This is particularly important for MySQL sources, which can only use a single connection to move data. See the following note.
+
+- If a MySQL database is set as a [source](#source-and-target-databases), the [`--table-concurrency`](#global-flags) and [`--export-concurrency`](#global-flags) flags **cannot** be set above `1`. If these values are changed, MOLT Fetch returns an error. This guarantees consistency when moving data from MySQL, due to MySQL limitations. MySQL data is migrated to CockroachDB one table and shard at a time, using [`WITH CONSISTENT SNAPSHOT`](https://dev.mysql.com/doc/refman/8.0/en/commit.html) transactions.
+
+- To prevent memory outages during data export of tables with large rows, estimate the amount of memory used to export a table:
+
+ ~~~
+ --row-batch-size * --export-concurrency * average size of the table rows
+ ~~~
+
+ If you are exporting more than one table at a time (i.e., [`--table-concurrency`](#global-flags) is set higher than `1`), add the estimated memory usage for the tables with the largest row sizes. Ensure that you have sufficient memory to run `molt fetch`, and adjust `--row-batch-size` accordingly.
+
+- If a table in the source database is much larger than the other tables, [filter and export the largest table](#schema-and-table-selection) in its own `molt fetch` task. Repeat this for each of the largest tables. Then export the remaining tables in another task.
+
+- When using [`IMPORT INTO` mode](#fetch-mode) to load tables into CockroachDB, if the fetch process terminates before the import job completes, the hanging import job on the target database will keep the table offline. To make this table accessible again, [manually resume or cancel the job]({% link {{ page.version.version }}/import-into.md %}#view-and-control-import-jobs). Then resume `molt fetch` using [continuation](#fetch-continuation), or restart the process from the beginning.
+
+## Security recommendations
+
+Cockroach Labs **strongly** recommends the following:
+
+### Secure connections
+
+- Use secure connections to the source and [target CockroachDB database]({% link {{ page.version.version }}/connection-parameters.md %}#additional-connection-parameters) whenever possible.
+- By default, insecure connections (i.e., `sslmode=disable` on PostgreSQL; `sslmode` not set on MySQL) are disallowed. When using an insecure connection, `molt fetch` returns an error. To override this check, you can enable the `--allow-tls-mode-disable` flag. Do this **only** for testing, or if a secure SSL/TLS connection to the source or target database is not possible.
+
+### Connection strings
+
+- Avoid plaintext connection strings.
+- Provide your connection strings as environment variables.
+- If possible within your security infrastructure, use an external secrets manager to load the environment variables from stored secrets.
+
+ For example, to export connection strings as environment variables:
+
+ ~~~ shell
+ export SOURCE="postgres://postgres:postgres@localhost:5432/molt?sslmode=verify-full"
+ export TARGET="postgres://root@localhost:26257/molt?sslmode=verify-full"
+ ~~~
+
+ Afterward, to pass the environment variables in `molt fetch` commands:
+
+ ~~~ shell
+ molt fetch \
+ --source "$SOURCE" \
+ --target "$TARGET" \
+ --table-filter 'employees' \
+ --bucket-path 's3://molt-test' \
+ --table-handling truncate-if-exists
+ ~~~
+
+### Secure cloud storage
+
+- When using [cloud storage](#cloud-storage) for your intermediate store, ensure that access control is properly configured. Refer to the [GCP](https://cloud.google.com/storage/docs/access-control) or [AWS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/security-iam.html) documentation.
+- Do not use public cloud storage in production.
+
+### Perform a dry run
+
+To verify that your connections and configuration work properly, run MOLT Fetch in a staging environment before moving any data in production. Use a test or development environment that is as similar as possible to production.
+
+## Commands
+
+| Command | Usage |
+|---------|---------------------------------------------------------------------------------------------------|
+| `fetch` | Start the fetch process. This loads data from a source database to a target CockroachDB database. |
+
+### Subcommands
+
+| Command | Usage |
+|--------------|----------------------------------------------------------------------|
+| `tokens list` | List active [continuation tokens](#list-active-continuation-tokens). |
+
## Flags
-| Flag | Description |
-|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `--source` | (Required) Connection string for the source database. For details, see [Source and target databases](#source-and-target-databases). |
-| `--target` | (Required) Connection string for the target database. For details, see [Source and target databases](#source-and-target-databases). |
-| `--bucket-path` | The path within the [cloud storage](#cloud-storage) bucket where intermediate files are written (e.g., `'s3://bucket/path'` or `'gs://bucket/path'`). |
-| `--cleanup` | Whether to delete intermediate files after moving data using [cloud or local storage](#data-path). **Note:** Cleanup does not occur on [continuation](#fetch-continuation). |
-| `--compression` | Compression method for data when using [`IMPORT INTO` mode](#fetch-mode) (`gzip`/`none`).
+
+{% include_cached new-in.html version="v24.1" %} The **Replication Lag** graph displays the [replication lag]({% link {{ page.version.version }}/physical-cluster-replication-technical-overview.md %}) between the primary and standby cluster. This is the time between the most up-to-date replicated time and the actual time.
+
+Hovering over the graph displays:
+
+- The specific date and time of the replication lag.
+- The reported replication lag time.
+
{% include {{ page.version.version }}/ui/ui-summary-events.md %}
## See also
diff --git a/src/current/v24.1/user-defined-functions.md b/src/current/v24.1/user-defined-functions.md
index 6828d1f8895..bbec370c7de 100644
--- a/src/current/v24.1/user-defined-functions.md
+++ b/src/current/v24.1/user-defined-functions.md
@@ -296,7 +296,8 @@ For a deep-dive demo on UDFs, watch the following video:
User-defined functions have the following limitations:
-{% include {{ page.version.version }}/known-limitations/udf-stored-proc-limitations.md %}
+{% include {{ page.version.version }}/known-limitations/udf-limitations.md %}
+{% include {{ page.version.version }}/known-limitations/routine-limitations.md %}
Also refer to the [PL/pgSQL known limitations]({% link {{ page.version.version }}/plpgsql.md %}#known-limitations).