Basic topology patterns

[robert-s-lee]

Title

An illustration of common topology patterns

Description

A library of common topology patterns. Can start from a simple local topology and transition to other patterns to meet the requirements.

Outline

Local Topology

3 or more nodes are recommended to provide HA, share the load and spread the capacity. Dynamically scaling out the nodes from 3 to 4, 4 to 5, or any other intervals is supported. There are no constraints on the server increments. Local deployment is a single data center deployment. The network latency among the nodes is expected to be the same and about 1ms range.

The diagram below depicts a node as a letter (A, B, C, D, E ...)

A------C         A-----C           A-----C
 \    /  online  |     |  online   | \ / |
  \  /   ------> |     |  ------>  |  E  |
   \/            |     |           | / \ |
   B             B-----D           B-----D 

Regional Topology

Each region defines an availability zone. 3 or more regions are recommended. Similiar to the Local Topology, more regions can be added dynamically. It is recommended to have homogenous configuration among the regions for simplified operations. For sophisticated workloads, each region could have different node count and node specification. This heterogeneous configuration could better handle regional specific concurrency and load characteristics. The network latency among the regions is expected to be linear to the distance among the nodes. T

The diagram below depicts asymmetrical setup where the Central is closer to the West than the East. This configuration will provide better write latency to the write workloads in the West and Central.

A---C         A---C
 \ /           \ /
  B             B
  West---80m---East
     \         /  
    20ms    60ms  
       \    /
        \  /
         \/
       Central 
        A---C   
         \ / 
          B

Global Topology

The global topology connects the multiple regional topologies together to form a single database that is globally distributed. Transactions are globally consistent.

    West-----East            West-------East
       \      /                \        /
        \Asia/                  \Europe/
         \  /                    \  /
          \/                      \/
        Central                 Central
                Asia------Europe
                   \     /  
                    \   /  
                     \ /  
                   Americas 

               West---------East
                  \          /
                   \Americas/  
                    \      /
                    Central 

High-Performance Regional

Some applications have high-performance requirements. NJ and NY in the below diagram depict two separate data centers that are connected by a high bandwidth low latency network. In this topology, NJ and NY have the performance characteristics of the local topology, but the benefit of Zero RPO and near Zero RTO disaster recovery SLA. CA and NV have been set up with a network capability. The central region serves as the quorum.

   NJ ---1ms--- NY
     \       /  
     20ms  20ms  
        \  /
         \/  
       Central 
         /\  
        /  \
     20ms  20ms  
     /       \  
  CA ---1ms--- NV

n-tier architecture

Modern n-tier architecture is simplified as App and DB layers in the below diagram. A client connects to geographically close app server via GSLB. The app servers connect to one of the CockroachDB nodes within their geography via local balancer. A software-based load balancer, HAProxy, located on the app server, the configuration is provided by CockroachDB. A Network-based load balancer can also be used.

         Clients
           |
          GSLB
           |
  +--------+---------+
West    Central    East
  |        |         |
 App  |   App   |   App
 ---  |   ---   |   ---    
  LB  |    LB   |    LB
      |         |

  West---Central ---East
    \               /
     \ CockroachDB /  
      -------------

Expected Audience

Architects, Developers, DBA, Operations

[jseldess]

See @bdarnell's comment here: #2312 (comment)

Need to clarify that replication factor and # of nodes is distinct.

[jseldess]

Partially addressed in #2850. Moving further work to 2.1.

[jseldess]

For 2.1, this is part of #2935.