Skip to content

Commit

Permalink
Improve narrative
Browse files Browse the repository at this point in the history
  • Loading branch information
Jesse Seldess committed Jun 26, 2017
1 parent 80c4bbc commit 02bec34
Showing 1 changed file with 24 additions and 19 deletions.
43 changes: 24 additions & 19 deletions demo-cloud-migration.md
Expand Up @@ -17,11 +17,11 @@ In this tutorial, you'll use CockroachDB, the HAProxy load balancer, and Cockroa
- Install [Go](https://golang.org/dl/). If you're on a Mac and using Homebrew, use `brew install go`.
- Install the [CockroachDB version of YCSB](https://github.com/cockroachdb/loadgen/tree/master/ycsb): `go get github.com/cockroachdb/loadgen/ycsb`

Also, to make it easy to keep track of the data files and logs for your cluster, you may want to create a new directory (e.g., `mkdir cloud-migration`) and start all your nodes in that directory.
Also, to keep track of the data files and logs for your cluster, you may want to create a new directory (e.g., `mkdir cloud-migration`) and start all your nodes in that directory.

## Step 1. Start a 3-node cluster on "cloud 1"

If you've already [started a local cluster](start-a-local-cluster.html), these commands should be familiar to you. The new flag to note is [`--locality`](configure-replication-zones.html#descriptive-attributes-assigned-to-nodes), which accepts key-value pairs that describe the locality of a node. In this case, you're using the flag to specify that the first 3 nodes are running on "cloud platform 1".
If you've already [started a local cluster](start-a-local-cluster.html), the commands for starting nodes should be familiar to you. The new flag to note is [`--locality`](configure-replication-zones.html#descriptive-attributes-assigned-to-nodes), which accepts key-value pairs that describe the topography of a node. In this case, you're using the flag to specify that the first 3 nodes are running on "cloud 1".

In a new terminal, start node 1 on cloud 1:

Expand Down Expand Up @@ -61,7 +61,7 @@ $ cockroach start --insecure \

## Step 2. Set up HAProxy load balancing

Each CockroachDB node is an equally suitable SQL gateway to your cluster, but to ensure an even balancing of client requests across available nodes, we can use a TCP load balancer. The [HAProxy](http://www.haproxy.org/) load balancer that you installed before getting started is one of the most popular open-source TCP load balancers, and CockroachDB includes a built-in command for generating an HAProxy configuration file that is preset to work with your running cluster.
You're now running 3 nodes in a simulated cloud. Each of these nodes is an equally suitable SQL gateway to your cluster, but to ensure an even balancing of client requests across these nodes, you can use a TCP load balancer. Let's use the open-source [HAProxy](http://www.haproxy.org/) load balancer that you installed earlier.

In a new terminal, run the [`cockroach gen haproxy`](generate-cockroachdb-resources.html) command, specifying the port of any node:

Expand Down Expand Up @@ -98,9 +98,9 @@ $ haproxy -f haproxy.cfg

## Step 3. Start a load generator

The YCSB load generator that you installed before getting started lets you simulate several client connections performing mixed read/write workloads against the cluster.
Now that you have a load balancer running in front of your cluster, let's use the YCSB load generator that you installed earlier to simulate several client connections, each performing mixed read/write workloads.

In a new terminal, start the load generator:
In a new terminal, start `ycsb`, pointing it at HAProxy's port:

~~~ shell
$ $HOME/go/bin/ycsb -duration 20m -tolerate-errors -concurrency 10 -rate-limit 100 'postgresql://root@localhost:26000?sslmode=disable'
Expand All @@ -110,7 +110,7 @@ This command initiates 10 concurrent client workloads for 20 minutes, but limits

## Step 4. Watch data balance across all 3 nodes

Open the Admin UI at `http://localhost:8080` and hover over the **SQL Queries** graph at the top. After a minute or so, you'll see that the load generator is executing approximately 95% reads and 5% writes across all nodes:
Now open the Admin UI at `http://localhost:8080` and hover over the **SQL Queries** graph at the top. After a minute or so, you'll see that the load generator is executing approximately 95% reads and 5% writes across all nodes:

<img src="images/admin_ui_sql_queries.png" alt="CockroachDB Admin UI" style="border:1px solid #eee;max-width:100%" />

Expand All @@ -120,12 +120,11 @@ Scroll down a bit and hover over the **Replicas per Node** graph. Because Cockro

## Step 5. Add 3 nodes on "cloud 2"

Again, the flag to note is [`--locality`](configure-replication-zones.html#descriptive-attributes-assigned-to-nodes), which you're using to specify that these next 3 nodes are running on "cloud platform 2".
At this point, you're running three nodes on cloud 1. But what if you'd like to start experimenting with resources provided by another cloud vendor? Let's try that by adding three more nodes to a new cloud platform. Again, the flag to note is [`--locality`](configure-replication-zones.html#descriptive-attributes-assigned-to-nodes), which you're using to specify that these next 3 nodes are running on "cloud 2".

{{site.data.alerts.callout_info}}If you were running nodes across clouds for real, you'd also configure firewalls to allow inbound and outbound communication between all the nodes. {{site.data.alerts.end}}
In a new terminal, start node 4 on cloud 2:

~~~ shell
# In a new terminal, start node 4 on cloud 2
$ cockroach start --insecure \
--locality=cloud=2 \
--store=cloud2node4 \
Expand All @@ -134,8 +133,11 @@ $ cockroach start --insecure \
--http-port=8083 \
--join=localhost:26257 \
--cache=100MB
~~~

In a new terminal, start node 5 on cloud 2:

# In a new terminal, start node 5 on cloud 2:
~~~ shell
$ cockroach start --insecure \
--locality=cloud=2 \
--store=cloud2node5 \
Expand All @@ -144,8 +146,11 @@ $ cockroach start --insecure \
--http-port=8084 \
--join=localhost:26257 \
--cache=100MB
~~~

# In a new terminal, start node 6 on cloud 2:
In a new terminal, start node 6 on cloud 2:

~~~ shell
$ cockroach start --insecure \
--locality=cloud=2 \
--store=cloud2node6 \
Expand All @@ -158,33 +163,33 @@ $ cockroach start --insecure \

## Step 6. Watch data balance across all 6 nodes

Back in the Admin UI, hover over the **Replicas per Node** graph again. Because you set [`--locality`](configure-replication-zones.html#descriptive-attributes-assigned-to-nodes) settings to specify that nodes are running across 2 clouds, you'll see an approximately even replica count on each node, indicating that CockroachDB has automatically rebalanced replicas across the cloud platforms:
Back in the Admin UI, hover over the **Replicas per Node** graph again. Because you used [`--locality`](configure-replication-zones.html#descriptive-attributes-assigned-to-nodes) to specify that nodes are running on 2 clouds, you'll see an approximately even number of replicas on each node, indicating that CockroachDB has automatically rebalanced replicas across both simulated clouds:

<img src="images/admin_ui_replicas_migration2.png" alt="CockroachDB Admin UI" style="border:1px solid #eee;max-width:100%" />

Note that it takes a few minutes for the Admin UI to show accurate per-node replica counts on hover, which is why the new nodes show 0 replicas in the screenshot above. However, the graph lines are accurate, and you can click **View node list** in the **Summary** area for another accurate view of per-node replica counts.
Note that it takes a few minutes for the Admin UI to show accurate per-node replica counts on hover, which is why the new nodes show 0 replicas in the screenshot above. However, the graph lines are accurate, and you can click **View node list** in the **Summary** area for accurate per-node replica counts as well.

## Step 7. Migrate all data to "cloud 2"

In a new terminal, edit the default replication zone, adding a [hard constraint](configure-replication-zones.html#replication-constraints) that all replicas must be on nodes with `--locality=cloud=2`:
So your cluster is replicating across two simulated clouds. But let's say that after experimentation, you're happy with cloud vendor 2, and you decide that you'd like to move everything there. Can you do that without interruption to your live client traffic? Yes, and it's as simple as running a single command to add a [hard constraint](configure-replication-zones.html#replication-constraints) that all replicas must be on nodes with `--locality=cloud=2`.

In a new terminal, edit the default replication zone:

~~~ shell
$ echo 'constraints: [+cloud=2]' | cockroach zone set .default --insecure --host=localhost -f -
~~~

{{site.data.alerts.callout_info}} As you'll see in the next step, as long as the <code>--locality</code> flag was set properly on nodes, <strong>this single command is all it takes to initiate an automatic migration from one cloud to another</strong>.{{site.data.alerts.end}}

## Step 8. Verify the data migration

Back in the Admin UI, hover over the **Replicas per Node** graph again. Very soon, you should see the replica count double on nodes 4, 5, and 6 and drop to 0 on nodes 1, 2, and 3:
Back in the Admin UI, hover over the **Replicas per Node** graph again. Very soon, you'll see the replica count double on nodes 4, 5, and 6 and drop to 0 on nodes 1, 2, and 3:

<img src="images/admin_ui_replicas_migration3.png" alt="CockroachDB Admin UI" style="border:1px solid #eee;max-width:100%" />

This indicates that all data has been migrated from "cloud 1" to "cloud 2". In a real cloud migration scenario, at this point you would update the load balancer to point to the nodes on "cloud 2" and then stop the nodes on "cloud 1". But for the purpose of this local simulation, there's no need to do that.
This indicates that all data has been migrated from cloud 1 to cloud 2. In a real cloud migration scenario, at this point you would update the load balancer to point to the nodes on cloud 2 and then stop the nodes on cloud 1. But for the purpose of this local simulation, there's no need to do that.

## Step 9. Stop the cluster

Stop YCSB by switching into its terminal and pressing **CTRL + C**. Then do the same for HAProxy and each CockroachDB node.
Once you're done with your cluster, stop YCSB by switching into its terminal and pressing **CTRL + C**. Then do the same for HAProxy and each CockroachDB node.

{{site.data.alerts.callout_success}}For the last node, the shutdown process will take longer (about a minute) and will eventually force kill the node. This is because, with only 1 node still online, a majority of replicas are no longer available (2 of 3), and so the cluster is not operational. To speed up the process, press <strong>CTRL + C</strong> a second time.{{site.data.alerts.end}}

Expand Down

0 comments on commit 02bec34

Please sign in to comment.