Update the failover docs for PCR #19405
Conversation
✅ Deploy Preview for cockroachdb-api-docs canceled.
|
✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.
|
✅ Netlify Preview
To edit notification comments on pull requests, go to your Netlify site configuration. |
katmayb
force-pushed
the
failover-failback-improvements
branch
6 times, most recently
from
01c9087 to
8f82f5d
Compare
| ## Before you begin | ||
|
|
||
| - Two separate CockroachDB clusters (primary and standby) with a minimum of three nodes each, and each using the same CockroachDB {{page.version.version}} version. | ||
| - Two separate CockroachDB clusters (primary and standby) with a minimum of three nodes each, and each using the same CockroachDB {{page.version.version}} version. The standby cluster should be the same version or one version ahead of the primary cluster. |
There was a problem hiding this comment.
Added this sentence to match the current description in the "Cluster versions and upgrades" section.
| - The [Deploy CockroachDB on Premises]({% link {{ page.version.version }}/deploy-cockroachdb-on-premises.md %}) tutorial creates a self-signed certificate for each {{ site.data.products.core }} cluster. To create certificates signed by an external certificate authority, refer to [Create Security Certificates using OpenSSL]({% link {{ page.version.version }}/create-security-certificates-openssl.md %}). | ||
| - All nodes in each cluster will need access to the Certificate Authority for the other cluster. Refer to [Copy certificates](#step-3-copy-certificates). | ||
| - An [{{ site.data.products.enterprise }} license]({% link {{ page.version.version }}/licensing-faqs.md %}#types-of-licenses) on the primary **and** standby clusters. You must use the system virtual cluster on the primary and standby clusters to enable your {{ site.data.products.enterprise }} license. | ||
| - The primary and standby clusters **must have the same [region topology]({% link {{ page.version.version }}/topology-patterns.md %})**. For example, replicating a multi-region primary cluster to a single-region standby cluster is not supported. Mismatching regions between a multi-region primary and standby cluster is also not supported. |
There was a problem hiding this comment.
Enterprise licensing prereq + steps gone as a result of CRDB license updates.
| - [Create a new virtual cluster]({% link {{ page.version.version }}/set-up-physical-cluster-replication.md %}#step-4-start-replication) (`main`) on cluster **A** from the replication of cluster **B**. Cluster **A** is now virtualized. This will start an initial scan because the PCR stream will ignore the former workload tables in the system virtual cluster that were [originally replicated to **B**]({% link {{ page.version.version }}/set-up-physical-cluster-replication.md %}#set-up-pcr-from-an-existing-cluster). You can [drop the tables]({% link {{ page.version.version }}/drop-table.md %}) that were in the system virtual cluster, because the new virtual cluster will now hold the workload replicating from cluster **B**. | ||
| - [Start an entirely new cluster]({% link {{ page.version.version }}/cockroach-start.md %}) **C** and [create]({% link {{ page.version.version }}/set-up-physical-cluster-replication.md %}#step-4-start-replication) a `main` virtual cluster on it from the replication of cluster **B**. This will start an initial scan because cluster **C** is empty. | ||
| ## Job management |
There was a problem hiding this comment.
This isn't new content, just moved Job management to the bottom of the page after faliback/cutback
alicia-l2
left a comment
alicia-l2
left a comment
There was a problem hiding this comment.
LGTM, left some comments - thank you!
| {{site.data.alerts.end}} | ||
|
|
||
| The failover is a two-step process on the standby cluster: | ||
| _Failover_ in [**physical cluster replication (PCR)**]({% link {{ page.version.version }}/physical-cluster-replication-overview.md %}) allows you to switch from the active primary cluster to the passive standby cluster that has ingested replicated data. When you complete the replication stream to initiate a failover, the job stops the stream of new data, resets the standby [virtual cluster]({% link {{ page.version.version }}/physical-cluster-replication-technical-overview.md %}) to a point in time where all ingested data is consistent, and then marks the standby virtual cluster as ready to accept traffic. |
There was a problem hiding this comment.
stops the stream
Should we just say stops replicating data from the primary?
resets
should we say sets?
marks the standby virtual cluster as ready to accept traffic.
should we say 'makes' instead of marks? Because the standby cluster is ready to accept traffic at that time. or like enables? Something just more explicit that the standby is online
There was a problem hiding this comment.
Oh also - I don't know how long you wanted to make this section, but it could be cool to mention how we let you failover to a time in the past or future. Defer to you though (we could also put that in the overview)!
There was a problem hiding this comment.
Implemented all of these suggestions — thank you!
|
|
||
| 1. [Initiating the failover](#step-1-initiate-the-failover). | ||
| 1. [Completing the failover](#step-2-complete-the-failover). | ||
| _Failback_ in PCR switches operations back to the original primary cluster (or a different cluster) after a failover event. When you initiate a failback, the job ensures the original primary is up to date with writes from the standby that happened after failover. The original primary cluster is then set as ready to accept application traffic once again. |
There was a problem hiding this comment.
Hm, what do you mean by a different cluster? Do you mean like a net new cluster?
There was a problem hiding this comment.
yes, I did mean that. Do you want me to take that out or change to "new cluster"?
There was a problem hiding this comment.
I think maybe new cluster would be more explicit? Only calling it out because i've recently had some folks confused about it when we are talking through PCR
| ~~~ | ||
| The `failover_time` is the timestamp at which the replicated data is consistent. The cluster will revert any data above this timestamp: | ||
| The `failover_time` is the timestamp at which the replicated data is consistent. The cluster will revert any data above this timestamp: |
There was a problem hiding this comment.
Can we add, 'the cluster will revert any replicated data'... 'this timestamp to ensure that the standby is consistent with the primary at that timestamp'
| ## Failback | ||
|
|
||
| During a replication stream, jobs running on the primary cluster will replicate to the standby cluster. Once you have [completed a failover](#step-2-complete-the-failover) (or a [failback](#fail-back-to-the-primary-cluster)), refer to the following sections for details on resuming jobs on the promoted cluster. | ||
| After failing over to the standby cluster, you may need to fail back to the original primary cluster to serve your application. Depending on the state of the primary cluster in the original PCR stream, use one of the following workflows: |
There was a problem hiding this comment.
Should we say, 'you may want to revert to your original primary-standby cluster setup.' Because they technically can serve their application using the standby cluster.
Should we say 'Depending on the configuration of the primary cluster?'
| - [Create a new virtual cluster]({% link {{ page.version.version }}/set-up-physical-cluster-replication.md %}#step-4-start-replication) (`main`) on cluster **A** from the replication of cluster **B**. Cluster **A** is now virtualized. This will start an initial scan because the PCR stream will ignore the former workload tables in the system virtual cluster that were [originally replicated to **B**]({% link {{ page.version.version }}/set-up-physical-cluster-replication.md %}#set-up-pcr-from-an-existing-cluster). You can [drop the tables]({% link {{ page.version.version }}/drop-table.md %}) that were in the system virtual cluster, because the new virtual cluster will now hold the workload replicating from cluster **B**. | ||
| - [Start an entirely new cluster]({% link {{ page.version.version }}/cockroach-start.md %}) **C** and [create]({% link {{ page.version.version }}/set-up-physical-cluster-replication.md %}#step-4-start-replication) a `main` virtual cluster on it from the replication of cluster **B**. This will start an initial scan because cluster **C** is empty. | ||
| ## Job management |
There was a problem hiding this comment.
This section is awesome - thanks so much!
| ### Changefeeds | ||
| [Changefeeds]({% link {{ page.version.version }}/change-data-capture-overview.md %}) will fail on the promoted cluster immediately after failover to avoid two clusters running the same changefeed to one sink. We recommend that you recreate changefeeds on the promoted cluster. | ||
There was a problem hiding this comment.
these two sentences kind of confuse me -- are we saying that changefeed schedules will pause and then currently running changefeeds will fail?
There was a problem hiding this comment.
Yeah, I added "currently running" to the first paragraph. There was a difference in behavior for changefeeds (fail) vs. scheduled changefeeds (pause) when this was written, which I believe is still true.
There was a problem hiding this comment.
ooh, I didn't even know that - thanks!
| Physical cluster replication is supported in CockroachDB {{ site.data.products.core }} clusters. | ||
| {{site.data.alerts.end}} | ||
|
|
||
| {% include_cached new-in.html version="v23.2" %} _Cutover_ in [**physical cluster replication (PCR)**]({% link {{ page.version.version }}/physical-cluster-replication-overview.md %}) allows you to switch from the active primary cluster to the passive standby cluster that has ingested replicated data. When you complete the replication stream to initiate a cutover, the job stops replicating data from the primary, sets the standby [virtual cluster]({% link {{ page.version.version }}/physical-cluster-replication-technical-overview.md %}) to a point in time (in the past or future) where all ingested data is consistent, and then makes the standby virtual cluster ready to accept traffic. |
|
@alicia-l2 Approved via slack! |
| ### Step 1. Initiate the failover | ||
|
|
||
| To initiate a failover to the standby cluster, there are different ways of specifying the point in time for the standby's promotion. That is, the standby cluster's live data at the point of failover. Refer to the following sections for steps: | ||
| To initiate a failover to the standby cluster, you can specify the point in time for the standby's promotion in different ways. That is, the standby cluster's live data at the point of failover. Refer to the following sections for steps: |
There was a problem hiding this comment.
"in the following ways" ???
There was a problem hiding this comment.
Much better! Thanks!
katmayb
force-pushed
the
failover-failback-improvements
branch
from
4e06874 to
0227470
Compare
3 participants
Fixes DOC-12358, DOC-12356
This PR is to clarify the failover/failback (formerly cutover/cutback) docs for PCR.
Changes include:
Note
This is PR 2 in a series of work to improve the PCR docs. Especially UX around non-ua to UA and failover/failback + upgrades and versioning.
Follow-up PR will include improved descriptions on cluster versions and upgrade process.
Preview
https://deploy-preview-19405--cockroachdb-docs.netlify.app/docs/v25.1/failover-replication.html