Cloud migration tutorial #1610

jseldess · 2017-06-21T03:02:12Z

A tutorial showing users how to use a local cluster to simulate cross-cloud deployment and then migration from one cloud to another.

I'm planning to add a simple screencast (with audio or text) demoing this across real cloud platforms, and offering that video as something to watch before you begin. But I'm opening this to get initial feedback on the flow and basic messaging.

cockroach-teamcity · 2017-06-21T03:02:17Z

This change is

spencerkimball

Awesome.

spencerkimball · 2017-06-21T15:16:20Z

demo-cloud-migration.md

+~~~ shell
+# In a new terminal, start node 1 on cloud 1:
+$ cockroach start --insecure \
+--locality=cloud=platform1 \


Why "cloud=platform1" instead of just "cloud=1"?

Good point. Done.

spencerkimball · 2017-06-21T15:18:44Z

demo-cloud-migration.md

+
+## Step 2. Set up HAProxy load balancing
+
+Each CockroachDB node is an equally suitable SQL gateway to your cluster, but to ensure client performance and reliability, it's important to use TCP load balancing. [HAProxy](http://www.haproxy.org/) is one of the most popular open-source TCP load balancers, and CockroachDB includes a built-in command for generating a configuration file that is preset to work with your running cluster, so you'll use that tool here.


How about: "..., but to ensure an even balancing of client requests across available nodes, we can use a TCP load balancer."

spencerkimball · 2017-06-21T15:24:54Z

demo-cloud-migration.md

+
+## Step 7. Migrate all data to "cloud 2"
+
+In a new terminal, edit the default replication zone, adding a [hard constraint](configure-replication-zones.html#replication-constraints) that all replicas must be on nodes with `--locality=cloud=platform2`:


Why don't you provide some colorful additional narration to these steps that are demonstrating the value proposition. For example: "OK, we're now running three nodes one cloud 1. But what if we'd like to start experimenting with resources provided by another cloud vendor? Let's try that by adding three more nodes to a new cloud platform."

"We now have the cluster replicating across our two simulated cloud vendors. But after experimentation, we're happy with cloud vendor 2, and we decide that we'd like to move everything there. Can we do that without interruption to our live client traffic? Yes, and it's as simple as adding a constraint to our replication configuration that all replicas must be chosen from cloud vendor 2."

You get the idea...we need to give people a scenario so it's not just an abstraction that only folks who've been struggling specifically with these problems will understand.

Great feedback. I'll work on this.

cockroach-teamcity · 2017-06-21T15:36:27Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/f8645eb6b95b234363c8ca97c85ab28b6092f424/demo-cloud-migration.html

bdarnell · 2017-06-21T22:44:59Z

LGTM

Review status: 0 of 10 files reviewed at latest revision, 9 unresolved discussions, some commit checks failed.

demo-cloud-migration.md, line 7 at r4 (raw file):

---

CockroachDB's flexible [replication controls](configure-replication-zones.html) make it trivially easy to run a single CockroachDB cluster across cloud platforms or to migrate a cluster from one cloud to another without any service interruption. This page walks you through a local simulation of the process.

Cross-cloud migration is just a special case of cross-DC migration. Since this is just a simulation, it seems weird to say it's a simulation of the special case instead of the general case. But I guess that's what gets the clicks...

demo-cloud-migration.md, line 55 at r4 (raw file):

~~~ shell
$ brew install haproxy

This is mac-specific. I'm OK with not having instructions for other platforms at this point, but we should say something like "In a new terminal, install HAProxy. If you're on a Mac and using Homebrew, use brew install haproxy"

demo-cloud-migration.md, line 64 at r4 (raw file):

~~~

This command generates an `haproxy.cfg` file automatically configured to work with the 3 nodes of your running cluster. In the file, change `bind :26257` to `bind :26270`. This changes the port on which HAProxy accepts requests to a port that is not already in use by a node and that won't be used by the nodes you'll add later.

These port numbers all start to run together. How about a round number like 26000 instead of 26270?

demo-cloud-migration.md, line 66 at r4 (raw file):

This command generates an `haproxy.cfg` file automatically configured to work with the 3 nodes of your running cluster. In the file, change `bind :26257` to `bind :26270`. This changes the port on which HAProxy accepts requests to a port that is not already in use by a node and that won't be used by the nodes you'll add later.

~~~ shell

shell mode doesn't highlight this correctly (it colors bind but not any other keywords). Unless there's a more precise option, better to just remove the shell tag completely.

demo-cloud-migration.md, line 98 at r4 (raw file):

~~~ shell
$ go get github.com/cockroachdb/loadgen/ycsb

There should be an introductory section that tells people they'll need to install Go (and we could put the haproxy installation instructions there too)

demo-cloud-migration.md, line 183 at r4 (raw file):

## Step 9. Stop the cluster

Stop HAProxy and YCSB by switching into their terminals and pressing **CTRL + C**. Do the same for each CockroachDB node.

Stop YCSB first, or it will start spewing errors as soon as you stop haproxy.

Comments from Reviewable

jseldess · 2017-06-22T19:46:52Z

Review status: 0 of 10 files reviewed at latest revision, 9 unresolved discussions, some commit checks failed.

demo-cloud-migration.md, line 64 at r4 (raw file):

Previously, bdarnell (Ben Darnell) wrote…

These port numbers all start to run together. How about a round number like 26000 instead of 26270?

Done.

demo-cloud-migration.md, line 66 at r4 (raw file):

Previously, bdarnell (Ben Darnell) wrote…

shell mode doesn't highlight this correctly (it colors bind but not any other keywords). Unless there's a more precise option, better to just remove the shell tag completely.

Done.

Comments from Reviewable

cockroach-teamcity · 2017-06-22T19:47:27Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/be3014696c690cb648503a8092024eb017585ca4/

nstewart · 2017-06-22T21:03:03Z

Minor feedback (we discussed in person too)

it would be useful to have people start in some sort of ./cloud-migration-test directory so they don't get a bunch of data files in their home
as written, the ycsb command doesn't work after the go-install. need to include the command to build
We shouldn't pre-split the ycsb ranges. I think this breaks (or at least doesn't reinforce) a beginners mental model for how CRDB works. Taking off the split shows new ranges emerging as new load comes in, vs just having it be flat the whole time.
In the final 'aha' graph where the load switches we ask users to hover over the graph to see the replica counts, but in my instance all but 1 were zero, which clearly wasn't correct. I think I would just say look at the graph, it's clear from the shape and color change that the load has moved to the other cloud. For the record, I just realized my crdb local install is beta, but according to Jesse this is still a bug.

jseldess · 2017-06-22T21:15:54Z

Review status: 0 of 10 files reviewed at latest revision, 9 unresolved discussions.

demo-cloud-migration.md, line 55 at r4 (raw file):

Previously, bdarnell (Ben Darnell) wrote…

This is mac-specific. I'm OK with not having instructions for other platforms at this point, but we should say something like "In a new terminal, install HAProxy. If you're on a Mac and using Homebrew, use brew install haproxy"

Done.

demo-cloud-migration.md, line 98 at r4 (raw file):

Previously, bdarnell (Ben Darnell) wrote…

There should be an introductory section that tells people they'll need to install Go (and we could put the haproxy installation instructions there too)

Done.

demo-cloud-migration.md, line 183 at r4 (raw file):

Previously, bdarnell (Ben Darnell) wrote…

Stop YCSB first, or it will start spewing errors as soon as you stop haproxy.

Done.

Comments from Reviewable

cockroach-teamcity · 2017-06-22T21:16:40Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/ff00ea8aaf016751d72212494e6dd71e2b5a99be/

cockroach-teamcity · 2017-06-22T21:20:15Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/2d750409e4ae2c1db9f0532ae914d02774f8b2ea/

cockroach-teamcity · 2017-06-26T20:06:21Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/80c4bbc227863b57de4a1ab51cfb40ae72e44007/

jseldess · 2017-06-26T20:09:33Z

@nstewart:

Added a "before you begin" suggestion to create a new directory for running nodes.
Moved all installations to "before you begin". And since go get does in fact build ycsb in $HOME/go/bin by default, I've added that path to the command to start ycsb.
No longer pre-splitting ranges with ycsb.
Called out that the replica count on hover will be off for a few minutes but that the graph lines are accurate. Also pointed to the node list view for another accurate view of per-node replica counts.

cockroach-teamcity · 2017-06-26T21:20:53Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/02bec3497a9df3647a2347c46e32ae0f850c4359/

cockroach-teamcity · 2017-06-26T21:26:18Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/9a770304ed498e4eda92ada7511890ad79edfb18/

cockroach-teamcity · 2017-06-26T21:39:58Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/9a770304ed498e4eda92ada7511890ad79edfb18/

cockroach-teamcity · 2017-06-27T04:30:05Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/838cb2fb203b365a4b0216d2764931d238704cc4/

jseldess · 2017-06-27T16:40:33Z

Merging. Will open a follow-up PR for the screencast part.

cockroach-teamcity · 2017-06-27T16:55:36Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/5ae031f0b04d0c645a4c1be7d3752d1723ad5d7e/

jseldess requested review from spencerkimball, BramGruneir and bdarnell and removed request for BramGruneir June 21, 2017 03:02

jseldess force-pushed the cloud-migration branch from a457025 to 3357769 Compare June 21, 2017 04:33

cockroachdb deleted a comment from cockroach-teamcity Jun 21, 2017

jseldess requested a review from nstewart June 21, 2017 14:24

jseldess force-pushed the cloud-migration branch from 3357769 to 9de181b Compare June 21, 2017 14:33

cockroachdb deleted a comment from cockroach-teamcity Jun 21, 2017

spencerkimball approved these changes Jun 21, 2017

View reviewed changes

jseldess force-pushed the cloud-migration branch from 9de181b to f8645eb Compare June 21, 2017 15:35

cockroachdb deleted a comment from cockroach-teamcity Jun 21, 2017

jseldess force-pushed the cloud-migration branch from f8645eb to be30146 Compare June 22, 2017 19:46

jseldess force-pushed the cloud-migration branch from be30146 to ff00ea8 Compare June 22, 2017 21:15

jseldess force-pushed the cloud-migration branch from ff00ea8 to 2d75040 Compare June 22, 2017 21:19

jseldess force-pushed the cloud-migration branch from 2d75040 to 80c4bbc Compare June 26, 2017 20:05

jseldess force-pushed the cloud-migration branch from 02bec34 to 9a77030 Compare June 26, 2017 21:25

jseldess changed the title ~~[WIP] Cloud migration tutorial~~ Cloud migration tutorial Jun 27, 2017

jseldess force-pushed the cloud-migration branch from 9a77030 to 838cb2f Compare June 27, 2017 04:29

Jesse Seldess added 3 commits June 27, 2017 12:54

Add tutorial for local simulation of cloud migration

d40b26f

Add links to the new cloud migration tutorial

4f51553

Add copy to clipboard feature

5ae031f

jseldess force-pushed the cloud-migration branch from 838cb2f to 5ae031f Compare June 27, 2017 16:55

jseldess merged commit 0520d55 into master Jun 27, 2017

jseldess deleted the cloud-migration branch June 27, 2017 17:22

jseldess mentioned this pull request Jul 6, 2017

Cloud migration screencast #1668

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cloud migration tutorial #1610

Cloud migration tutorial #1610

jseldess commented Jun 21, 2017 •

edited

cockroach-teamcity commented Jun 21, 2017

spencerkimball left a comment

spencerkimball Jun 21, 2017

jseldess Jun 21, 2017

spencerkimball Jun 21, 2017

jseldess Jun 21, 2017

spencerkimball Jun 21, 2017

jseldess Jun 21, 2017

cockroach-teamcity commented Jun 21, 2017 •

edited by jseldess

bdarnell commented Jun 21, 2017

jseldess commented Jun 22, 2017

cockroach-teamcity commented Jun 22, 2017

nstewart commented Jun 22, 2017

jseldess commented Jun 22, 2017

cockroach-teamcity commented Jun 22, 2017

cockroach-teamcity commented Jun 22, 2017

cockroach-teamcity commented Jun 26, 2017

jseldess commented Jun 26, 2017

cockroach-teamcity commented Jun 26, 2017

cockroach-teamcity commented Jun 26, 2017

cockroach-teamcity commented Jun 26, 2017

cockroach-teamcity commented Jun 27, 2017

jseldess commented Jun 27, 2017

cockroach-teamcity commented Jun 27, 2017


		## Step 2. Set up HAProxy load balancing

		Each CockroachDB node is an equally suitable SQL gateway to your cluster, but to ensure client performance and reliability, it's important to use TCP load balancing. [HAProxy](http://www.haproxy.org/) is one of the most popular open-source TCP load balancers, and CockroachDB includes a built-in command for generating a configuration file that is preset to work with your running cluster, so you'll use that tool here.


		## Step 7. Migrate all data to "cloud 2"

		In a new terminal, edit the default replication zone, adding a [hard constraint](configure-replication-zones.html#replication-constraints) that all replicas must be on nodes with `--locality=cloud=platform2`:

Cloud migration tutorial #1610

Cloud migration tutorial #1610

Conversation

jseldess commented Jun 21, 2017 • edited

cockroach-teamcity commented Jun 21, 2017

spencerkimball left a comment

Choose a reason for hiding this comment

spencerkimball Jun 21, 2017

Choose a reason for hiding this comment

jseldess Jun 21, 2017

Choose a reason for hiding this comment

spencerkimball Jun 21, 2017

Choose a reason for hiding this comment

jseldess Jun 21, 2017

Choose a reason for hiding this comment

spencerkimball Jun 21, 2017

Choose a reason for hiding this comment

jseldess Jun 21, 2017

Choose a reason for hiding this comment

cockroach-teamcity commented Jun 21, 2017 • edited by jseldess

bdarnell commented Jun 21, 2017

jseldess commented Jun 22, 2017

cockroach-teamcity commented Jun 22, 2017

nstewart commented Jun 22, 2017

jseldess commented Jun 22, 2017

cockroach-teamcity commented Jun 22, 2017

cockroach-teamcity commented Jun 22, 2017

cockroach-teamcity commented Jun 26, 2017

jseldess commented Jun 26, 2017

cockroach-teamcity commented Jun 26, 2017

cockroach-teamcity commented Jun 26, 2017

cockroach-teamcity commented Jun 26, 2017

cockroach-teamcity commented Jun 27, 2017

jseldess commented Jun 27, 2017

cockroach-teamcity commented Jun 27, 2017

jseldess commented Jun 21, 2017 •

edited

cockroach-teamcity commented Jun 21, 2017 •

edited by jseldess