cluster-topology-patterns.md

title	summary	toc
Cluster Topology Patterns	Common cluster topology patterns with setup examples and performance considerations.	true

This page covers common cluster topology patterns with setup examples, as well as the benefits and trade-off for each pattern. Before you select a candidate pattern for your cluster, use the following broad patterns as a starting point and consider trade-offs.

Considerations

Before selecting a pattern:

• Review the recommendations and requirements in our Production Checklist.
• Review the CockroachDB architecture. It's especially important to understand how data is stored in ranges, how ranges are replicated, and how one replica in each range serves as the "leaseholder" that coordinates all read and write requests for that range. See our Performance Tuning tutorial for more details about these important concepts and for simple read and write examples.
• Learn about the concept of locality, which makes CockroachDB aware of the location of nodes and able to intelligently balance replicas across localities. Locality is also a prerequisite for the follow-the-workload feature and for enterprise partitioning.

{{site.data.alerts.callout_info}} This page does not factor in hardware differences. {{site.data.alerts.end}}

Single-region clusters

Single datacenter, basic pattern

This first example is of a single-datacenter cluster, with each node on a different machine as per our basic topology recommendations. This pattern is common starting point for smaller organizations who may not have the resources (or need) to worry about a datacenter failure but still want to take advantage of CockroachDB's high availability.

For the diagram above:

Configuration

• App is an application that accesses CockroachDB.
• Load Balancer is a software-based load balancer.
• The 3 nodes are all running in a single datacenter.
• The cluster is using the default replication factor of 3 (represented by r1, r2, r3). Each range has 3 replicas, with each replica on a different node.

Availability expectations

• With the default replication factor of 3, the cluster can tolerate 1 node failure. In such a case, all ranges still have 2 replicas on live nodes and, thus, a majority.
• The cluster cannot tolerate the simultaneous or near-simultaneous failure of 2 or more nodes.

Performance expectations

• The network latency among the nodes is expected to be sub-millisecond.

Single datacenter, more resilient and/or performant

While the basic single-datacenter deployment takes advantage of CockroachDB's high availability, shares the load, and spreads capacity, scaling out the nodes has many benefits:

• Resiliency: There will be more room to increase the replication factor, which increases resiliency against the failure of more than one node. For example, with 5 nodes and a replication factor of 5, each range has 5 replicas, with each replica on a different node. In this case, even with 2 nodes down, each range retains a majority of its replicas (3/5).
• Performance: Adding nodes for more processing power and/or storage typically increases throughput.

There are no constraints on node increments.

Multiple datacenters

Once an organization begins to grow, a datacenter outage isn't acceptable and a cluster needs to be available all of the time. This is where a single-region cluster with multiple datacenters is useful. For example, an organization can do a cloud deployment across multiple datacenters within the same geographical region.

For the diagram above:

Configuration

• App is an application that accesses CockroachDB.
• Load Balancer is a software-based load balancer.
• The 3 nodes are each in a different datacenter, all located in the us-east region. Although not necessary with just 3 nodes, with more than 3 nodes, it's important to start each node with the --locality flag set to identify its datacenter (e.g., --locality=region=us-east,datacenter=us-east-a).
• The cluster is using the default replication factor of 3 (represented by r1, r2, r3).

Availability expectations

• The cluster can withstand a datacenter failure.

Performance expectations

• Since all nodes are within a single region, the network latency between the nodes is still expected to be close sub-millisecond.

Multi-region clusters

Multiple regions, basic pattern

For even more resiliency, use a multi-region cluster. A multi-region cluster is comprised of multiple datacenters in different regions (e.g., East, West), each with multiple nodes. CockroachDB will automatically try to diversify replica placement across localities (i.e., place a replica in each region). Using this setup, many organization will also transition to using different cloud providers (one provider per region).

In this example, the cluster has an asymmetrical setup where Central is closer to the West than the East. This configuration will provide better write latency to the write workloads in the West and Central because there is a lower latency (versus writing in the East). This is assuming you are not using zone configurations.

For this example:

Configuration

• Nodes are spread across 3 regions within a country (West, East, Central).
• A software-based load balancer directs traffic to any of the regions' nodes at random.
• Every region has 3 datacenters.
• Similar to the single-datacenter topology, more regions can be added dynamically
• A homogenous configuration among the regions for simplified operations is recommended
• For sophisticated workloads, each region can have different node count and node specification. This heterogeneous configuration could better handle regional specific concurrency and load characteristics.

When locality is enabled, the load balancer should be setup to load balance on the database nodes within the same locality as the app servers first:

• The West app servers should connect to the West CockroachDB servers
• The Central app servers should connect to the Central CockroachDB servers
• The East app servers should connect to the East CockroachDB servers

Availability expectations

If all of the nodes for a preferred locality are down, then the app will try databases in other localities. The cluster can withstand a datacenter failure. In general, multi-regions can help protect against natural disaster.

Performance expectations

• The latency numbers (e.g., 60ms) in the first diagram represent network round-trip from one datacenter to another.
• Follow-the-workload will keep the performance quick for where the load is so you do not pay cross-country latency on reads.
• Write latencies will not be faster than the slowest quorum between two regions.

Multiple regions, more performant (with partitioning)

While the basic pattern for a multi-region cluster can help protect against regional failures, there will be high latency due to cross-country roundtrips. This is not ideal for organizations who have users spread out across the country. For any multi-region cluster, partitioning should be used to keep data close to the users who access it.

This setup uses a modern multi-tier architecture, which is simplified to global server load balancer (GSLB), App, and Load Balancer layers in the below diagram:

Configuration

• Nodes are spread across 3 regions within a country (West, East, Central)

• A client connects to geographically close app server via GSLB.

• Inside each region, an app server connects to one of the CockroachDB nodes within their geography through a software-based load balancer

• Every region has 3 datacenters

• All CockroachDB nodes communicate with each other

• Tables are partitioned at row-level by locality.

• Rows with the West partition have their leaseholder in the West datacenter.

• Rows with the Central partition have their leaseholder in the Central datacenter.

• Rows with the East partition have their leaseholder in the East datacenter.

• Replicas are evenly distributed among the three datacenters.

• Abbreviated startup flag for each datacenter:

  --loc=Region=East
  --loc=Region=Central
  --loc=Region=West
  

Availability expectations

• Can survive a single datacenter failure, since a majority of the replicas will remain available.

Performance expectations

• Reads respond in a few milliseconds.
• Writes respond in 60ms.
• Symmetrical latency between datacenters.

Application expectations

• West App servers connect to the West CockroachDB nodes.
• Central App servers connect to the Central CockroachDB nodes.
• East App servers connect to the East CockroachDB nodes.

Anti-patterns

Do we want to add a section for bad patterns / things not to do? What should be added here?