Skip to content

Commit

Permalink
revisions for Bram
Browse files Browse the repository at this point in the history
  • Loading branch information
Jesse Seldess committed Jan 26, 2017
1 parent cf8f621 commit f46a42d
Show file tree
Hide file tree
Showing 3 changed files with 13 additions and 13 deletions.
20 changes: 10 additions & 10 deletions configure-replication-zones.md
Expand Up @@ -5,7 +5,7 @@ keywords: ttl, time to live, availability zone
toc: false
---

In CockroachDB, you use **replication zones** to control the number and location of replicas for specific sets of data. Initially, there is a single, default replication zone for the entire cluster. You can adjust this default zone as well as add zones for individual databases and tables as needed. For example, you might use the default zone to replicate most data in a cluster normally within a single datacenter, while creating a specific zone to more highly replicate a certain database or table across multiple datacenters and geographies.
In CockroachDB, you use **replication zones** to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. Initially, there is a single, default replication zone for the entire cluster. You can adjust this default zone as well as add zones for individual databases and tables as needed. For example, you might use the default zone to replicate most data in a cluster normally within a single datacenter, while creating a specific zone to more highly replicate a certain database or table across multiple datacenters and geographies.

This page explains how replication zones work and how to use the `cockroach zone` [command](cockroach-commands.html) to configure them.

Expand Down Expand Up @@ -44,11 +44,11 @@ Field | Description
`range_max_bytes` | The maximum size, in bytes, for a range of data in the zone. When a range reaches this size, CockroachDB will spit it into two ranges.<br><br>**Default:** `67108864` (64MiB)
`ttlseconds` | The number of seconds overwritten values will be retained before garbage collection. Smaller values can save disk space if values are frequently overwritten; larger values increase the range allowed for `AS OF SYSTEM TIME` queries, also know as [Time Travel Queries](select.html#select-historical-data-time-travel).<br><br>It is not recommended to set this below `600` (10 minutes); doing so will cause problems for long-running queries. Also, since all versions of a row are stored in a single range that never splits, it is not recommended to set this so high that all the changes to a row in that time period could add up to more than 64MiB; such oversized ranges could contribute to the server running out of memory or other problems.<br><br>**Default:** `86400` (24 hours)
`num_replicas` | The number of replicas in the zone.<br><br>**Default:** `3`
`constraints` | A comma-separated list of positive, required, and/or prohibited constraints influencing the location of replicas. See [Replica Constraints](#replication-constraints) for more details.<br><br>**Default:** No constraints, with CockroachDB locating each replica on a unique rack, if possible.
`constraints` | A comma-separated list of positive, required, and/or prohibited constraints influencing the location of replicas. See [Replica Constraints](#replication-constraints) for more details.<br><br>**Default:** No constraints, with CockroachDB locating each replica on a unique node, if possible.

### Replication Constraints

The location of replicas is based on the interplay between descriptive attributes assigned to nodes when they are started and constraints set in zone configurations.
The location of replicas, both when they are first added and when they are rebalanced to maintain cluster equilibrium, is based on the interplay between descriptive attributes assigned to nodes and constraints set in zone configurations.

{{site.data.alerts.callout_success}}For demonstrations of how to set node attributes and replication constraints in different scenarios, see <a href="#scenario-based-examples">Scenario-based Examples</a> below.{{site.data.alerts.end}}

Expand All @@ -64,16 +64,16 @@ Attribute Type | Description

#### Constraints in Replication Zones

The node- and store-level descriptive attributes mentioned above can be used as the following types of constraints in replication zones to influence the location of replicas. However, note the following general guidance:
The node-level and store-level descriptive attributes mentioned above can be used as the following types of constraints in replication zones to influence the location of replicas. However, note the following general guidance:

- When locality is the only consideration for replication, it's recommended to set locality on nodes without specifying any constraints in zone configurations. In the absence of constraints, CockroachDB attempts to spread replicas evenly across the cluster based on locality.
- When additional or different constraints are needed, positive constraints are generally sufficient. Required and prohibited constraints are useful in special situations where, for example, data must or must not be stored in a specific country.
- When additional or different constraints are needed, positive constraints are generally sufficient. Required and prohibited constraints are useful in special situations where, for example, data must or must not be stored in a specific country or on a specific type of machine.

Constraint Type | Description | Syntax
----------------|-------------|-------
**Positive** | Replicas will be placed on nodes/stores with as many matching attributes as possible. When there are no matching nodes/stores with capacity, replicas will be added wherever there is capacity.<br><br>For example, `constraints: [ssd, datacenter=us-west-1a]` would cause replicas to be located on different `ssd` drives in the `us-west-1a` datacenter. When there's not sufficient capacity for a new replica on an `ssd` drive in the datacenter, the replica would get added on an available `ssd` drive elsewhere. When there's not sufficient capacity for a new replica on any `ssd` drive, the replica would get added on any other drive with capacity. | `[ssd]`
**Required** | Replicas **must** be placed on nodes/stores with matching attributes. When there are no matching nodes/stores with capacity, new replicas will not be added.<br><br>For example, `constraints: [+datacenter=us-west-1a]` would force replicas to be located on different racks in the `us-west-1a` datacenter. When there's not sufficient capacity for a new replica on a unique rack in the datacenter, the replica would get added on a rack already storing a replica. When there's not sufficient capacity for a new replica on any rack in the datacenter, the replica would not get added. | `[+ssd]`
**Prohibited** | Replicas **must not** be placed on nodes/stores with matching attributes. When there are no alternate nodes/stores with capacity, new replicas will not be added.<br><br>For example, `constraints: [-mem, -us-west-1a]` would force replicas to be located on-disk on different racks outside the `us-west-1a` datacenter. When there's not sufficient on-disk capacity on a unique rack outside the datacenter, the replica would get added on a rack already storing a replica. When there's sufficient capacity for a new replica only in the `us-west-1a` datacenter, the replica would not get added. | `[-ssd]`
**Required** | Replicas **must** be placed on nodes/stores with matching attributes. When there are no matching nodes/stores with capacity, new replicas will not be added.<br><br>For example, `constraints: [+datacenter=us-west-1a]` would force replicas to be located on different nodes in the `us-west-1a` datacenter. When there's not sufficient capacity for a new replica on a unique node in the datacenter, the replica would get added to a node already storing a replica. When there's not sufficient capacity for a new replica on any node in the datacenter, the replica would not get added. | `[+ssd]`
**Prohibited** | Replicas **must not** be placed on nodes/stores with matching attributes. When there are no alternate nodes/stores with capacity, new replicas will not be added.<br><br>For example, `constraints: [-mem, -datacenter=us-west-1a]` would force replicas to be located on-disk on different nodes outside the `us-west-1a` datacenter. When there's not sufficient on-disk capacity on a unique node outside the datacenter, the replica would get added to a node already storing a replica. When there's sufficient capacity for a new replica only in the `us-west-1a` datacenter, the replica would not get added. | `[-ssd]`

### Node/Replica Recommendations

Expand Down Expand Up @@ -170,7 +170,7 @@ constraints: []

### Edit the Default Replication Zone

To edit the default replication zone, create a YAML file with changes, and use the `cockroach zone set .default -f <file.yaml>` command with appropriate flags:
To edit the default replication zone, create a YAML file defining only the values you want to change (other values will not be affected), and use the `cockroach zone set .default -f <file.yaml>` command with appropriate flags:

~~~ shell
$ cat default_update.yaml
Expand Down Expand Up @@ -201,7 +201,7 @@ $ echo 'num_replicas: 5' | cockroach zone set .default -f -

### Create a Replication Zone for a Database

To control replication for a specific database, create a YAML file with changes, and use the `cockroach zone set <database> -f <file.yaml>` command with appropriate flags:
To control replication for a specific database, create a YAML file defining only the values you want to change (other values will not be affected), and use the `cockroach zone set <database> -f <file.yaml>` command with appropriate flags:

~~~ shell
$ cat database_zone.yaml
Expand Down Expand Up @@ -232,7 +232,7 @@ $ echo 'num_replicas: 5' | cockroach zone set db1 -f -

### Create a Replication Zone for a Table

To control replication for a specific table, create a YAML file with changes, and use the `cockroach zone set <database.table> -f <file.yaml>` command with appropriate flags:
To control replication for a specific table, create a YAML file defining only the values you want to change (other values will not be affected), and use the `cockroach zone set <database.table> -f <file.yaml>` command with appropriate flags:

~~~ shell
$ cat table_zone.yaml
Expand Down
4 changes: 2 additions & 2 deletions recommended-production-settings.md
Expand Up @@ -17,9 +17,9 @@ When running a cluster with more than one node, each replica will be on a differ

- Run one node per machine. Since CockroachDB replicates across nodes, running more than one node per machine increases the risk of data unavailability if a machine fails.

- If a machine has multiple disks or SSDs, it's better to run one node with multiple `--store` flags instead of one node per disk, because this informs CockroachDB about the relationship between the stores and ensures that data will be replicated across different machines instead of being assigned to different disks of the same machine. For more details about stores, see [Start a Node](start-a-node.html).
- If a machine has multiple disks or SSDs, it's more efficient to run one node with multiple `--store` flags than one node per disk, because this informs CockroachDB about the relationship between the stores and ensures that data will be replicated across different machines instead of being assigned to different disks of the same machine. For more details about stores, see [Start a Node](start-a-node.html).

- Configurations with odd numbers of replicas are more robust than those with even numbers. Clusters of three and four nodes can each tolerate one node failure and still reach a quorum (2/3 and 3/4 respectively), so the fourth replica doesn't add any extra fault-tolerance. To survive two simultaneous failures, you must have five replicas.
- Configurations with odd numbers of replicas are more robust than those with even numbers. Clusters of three and four nodes can each tolerate one node failure and still reach a majority (2/3 and 3/4 respectively), so the fourth replica doesn't add any extra fault-tolerance. To survive two simultaneous failures, you must have five replicas.

- When replicating across datacenters, it's recommended to use datacenters on a single continent to ensure performance. Inter-continent scenarios will improve in performance soon.

Expand Down
2 changes: 1 addition & 1 deletion start-a-node.md
Expand Up @@ -42,7 +42,7 @@ Flag | Description
`--locality` | Not yet implemented.
`--max-sql-memory` | The total size for storage of temporary data for SQL clients, including prepared queries and intermediate data rows during query execution. This can be in any bytes-based unit, for example:<br><br>`--max-sql-memory=10000000000 ----> 1000000000 bytes`<br>`--max-sql-memory=1GB ----> 1000000000 bytes`<br>`--max-sql-memory=1GiB ----> 1073741824 bytes`<br><br>**Default:** 25% of total system memory (excluding swap), or 512MiB if the memory size cannot be determined
`--port`<br>`-p` | The port to bind to for internal and client communication. <br><br>**Env Variable:** `COCKROACH_PORT`<br>**Default:** `26257`
`--raft-tick-interval` | CockroachDB uses the [Raft consensus algorithm](https://raft.github.io/) to replicate data consistently according to your [replication zone configuration](configure-replication-zones.html). For each replica group, an elected leader heartbeats its followers and keeps their logs replicated. When followers fail to receive heartbeats, a new leader is elected. <br><br>This flag is factored into defining the interval at which replica leaders heartbeat followers. It is not recommended to change the default, but if changed, it must be changed identically on all nodes in the cluster.<br><br>**Default:** 200ms
`--raft-tick-interval` | CockroachDB uses the [Raft consensus algorithm](https://raft.github.io/) to replicate data consistently according to your [replication zone configuration](configure-replication-zones.html). For each replica group, an elected leader heartbeats its followers and keeps their logs replicated. When followers fail to receive heartbeats, a new leader is elected. <br><br>This flag is factored into defining the interval at which replica leaders heartbeat followers. It is not recommended to change the default, but if changed, every node in the cluster must be stopped and restarted with the identical value.<br><br>**Default:** 200ms
`--store`<br>`-s` | The file path to a storage device and, optionally, store attributes and maximum size. When using multiple storage devices for a node, this flag must be specified separately for each device, for example: <br><br>`--store=/mnt/ssd01 --store=/mnt/ssd02` <br><br>For more details, see [`store`](#store) below.

#### `store`
Expand Down

0 comments on commit f46a42d

Please sign in to comment.