Skip to content

Changing default rep factor doesn't affect system ranges #2901

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 28, 2018
Merged

Changing default rep factor doesn't affect system ranges #2901

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion releases/v2.0.0.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ Feature | Description
[Node Readiness Endpoint](../v2.0/monitoring-and-alerting.html#health-ready-1) | The new `/health?ready=1` endpoint returns an `HTTP 503 Service Unavailable` status response code with an error when a node is being decommissioned or is in the process of shutting down and is therefore not able to accept SQL connections and execute queries. This is especially useful for making sure [load balancers](../v2.0/recommended-production-settings.html#load-balancing) don't direct traffic to nodes that are live but not "ready", which is a necessary check during [rolling upgrades](../v2.0/upgrade-cockroach-version.html).
[Node Decommissioning](../v2.0/remove-nodes.html) | Nodes that have been decommissioned and stopped no longer appear in Admin UI and command-line interface metrics.
[Per-Replica Constraints in Replication Zones](../v2.0/configure-replication-zones.html#scope-of-constraints) | When defining a replication zone, unique constraints can be defined for each affected replica, meaning you can effectively pick the exact location of each replica.
[Replication Zone for "Liveness" Range](../v2.0/configure-replication-zones.html#create-a-replication-zone-for-system-ranges) | Clusters now come with a pre-defined replication zone for the "liveness" range, which contains the authoritative information about which nodes are live at any given time.
[Replication Zone for "Liveness" Range](../v2.0/configure-replication-zones.html#create-a-replication-zone-for-a-system-range) | Clusters now come with a pre-defined replication zone for the "liveness" range, which contains the authoritative information about which nodes are live at any given time.
[Timeseries Data Controls](../v2.0/operational-faqs.html#can-i-reduce-or-disable-the-storage-of-timeseries-data-new-in-v2-0) | It is now possible to reduce the amount of timeseries data stored by a CockroachDB cluster or to disable the storage of timeseries data entirely. The latter is recommended only when using a third-party tool such as Prometheus for timeseries monitoring.

## Backward-Incompatible Changes
Expand Down
57 changes: 36 additions & 21 deletions v2.0/configure-replication-zones.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ keywords: ttl, time to live, availability zone
toc: false
---

In CockroachDB, you use **replication zones** to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. Initially, there is are a few special pre-configured replication zones for internal system data along with a default replication zone that applies to the rest of the cluster. You can adjust these pre-configured zones as well as add zones for individual databases, tables, and rows ([enterprise-only](enterprise-licensing.html)) as needed. For example, you might use the default zone to replicate most data in a cluster normally within a single datacenter, while creating a specific zone to more highly replicate a certain database or table across multiple datacenters and geographies.
In CockroachDB, you use **replication zones** to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. Initially, there are some special pre-configured replication zones for internal system data along with a default replication zone that applies to the rest of the cluster. You can adjust these pre-configured zones as well as add zones for individual databases, tables, and rows ([enterprise-only](enterprise-licensing.html)) as needed. For example, you might use the default zone to replicate most data in a cluster normally within a single datacenter, while creating a specific zone to more highly replicate a certain database or table across multiple datacenters and geographies.

This page explains how replication zones work and how to use the `cockroach zone` [command](cockroach-commands.html) to configure them.

Expand All @@ -15,18 +15,32 @@ This page explains how replication zones work and how to use the `cockroach zone

## Overview

### Replication Zone Levels
### Replication Zones Levels

There are four replication zone levels:
There are four replication zone levels for **table data** in a cluster, listed from least to most granular:

- **Cluster:** CockroachDB comes with a single, default replication zone for the entire cluster. See [View the Default Replication Zone](#view-the-default-replication-zone) and [Edit the Default Replication Zone](#edit-the-default-replication-zone) for more details.
- **Database:** You can add replication zones for specific databases. See [Create a Replication Zone for a Database](#create-a-replication-zone-for-a-database) for more details.
- **Table:** You can add replication zones for specific tables. See [Create a Replication Zone for a Table](#create-a-replication-zone-for-a-table) for more details.
- **Row:** ([For enterprise users](enterprise-licensing.html)) You can add replication zones for specific rows in a table by [defining table partitions](partitioning.html). See [Create a Replication Zone for a Table Partition](#create-a-replication-zone-for-a-table-partition-new-in-v2-0) for more details.
Level | Description
------|------------
Cluster | CockroachDB comes with a pre-configured `.default` replication zone that applies to all table data in the cluster not constrained by a database, table, or row-specific replication zone. This zone can be adjusted but not removed. See [View the Default Replication Zone](#view-the-default-replication-zone) and [Edit the Default Replication Zone](#edit-the-default-replication-zone) for more details.
Database | You can add replication zones for specific databases. See [Create a Replication Zone for a Database](#create-a-replication-zone-for-a-database) for more details.
Table | You can add replication zones for specific tables. See [Create a Replication Zone for a Table](#create-a-replication-zone-for-a-table) for more details.<br><br>CockroachDB comes with a pre-configured replication zone for one internal table, `system.jobs`, which stores metadata about long-running jobs such as schema changes and backups. Historical queries are never run against this table and the rows in it are updated frequently, so the pre-configured zone gives this table a lower-than-default `ttlseconds`.
Row ([Enterprise-only](enterprise-licensing.html)) | You can add replication zones for specific rows in a table by [defining table partitions](partitioning.html). See [Create a Replication Zone for a Table Partition](#create-a-replication-zone-for-a-table-partition-new-in-v2-0) for more details.

In addition, CockroachDB stores internal **system data** in what are called system ranges. There are two replication zone levels for this internal system data, listed from least to most granular:

Level | Description
------|------------
Cluster | The `.default` replication zone mentioned above also applies to all system ranges not constrained by a more specific replication zone.
System Range | CockroachDB comes with pre-configured replication zones for the "meta" and "liveness" system ranges. If necessary, you can add replication zones for the "timeseries" range and other "system" ranges as well. See [Create a Replication Zone for a System Range](#create-a-replication-zone-for-a-system-range) for more details.

When replicating a piece of data, CockroachDB uses the most granular zone available: If there's a replication zone for the rows in a table, CockroachDB uses it; otherwise, it uses the replication zone for the table containing the data. If there's no applicable row or table replication zone, CockroachDB uses the database replication zone; otherwise it uses the cluster-wide replication zone.
When replicating data, whether table or system, CockroachDB always uses the most granular replication zone available. For example, for a piece of user data:

In addition to the databases and tables that are visible via SQL, CockroachDB stores additional internal data in what are called system ranges. You can configure replication zones for parts of these internal data ranges if you'd like to override the cluster-wide settings. See [Create a Replication Zone for System Ranges](#create-a-replication-zone-for-system-ranges) for more details.
1. If there's a replication zone for the row, CockroachDB uses it.
2. If there's no applicable row replication zone, CockroachDB uses the table replication zone.
3. If there's no applicable table replication zone, CockroachDB uses the database replication zone.
4. If there's no applicable database replication zone, CockroachDB uses the `.default` cluster-wide replication zone.

{{site.data.alerts.callout_danger}}Changes to the <code>.default</code> replication zone are not automatically applied to other existing replication zones. If you increase the replication factor for <code>.default</code>, for example, you may also want to increase the replication factor for <a href="#create-a-replication-zone-for-a-system-range">important internal data</a> as well.{{site.data.alerts.end}}

### Replication Zone Format

Expand Down Expand Up @@ -172,7 +186,7 @@ These examples focus on the basic approach and syntax for working with zone conf

### List the Pre-Configured Replication Zones

<span class="version-tag">New in v2.0:</span> Newly created CockroachDB clusters start with a few special pre-configured replication zones:
<span class="version-tag">New in v2.0:</span> Newly created CockroachDB clusters start with some special pre-configured replication zones:

{% include copy-clipboard.html %}
~~~ shell
Expand All @@ -186,8 +200,6 @@ $ cockroach zone ls --insecure
system.jobs
~~~

`.default` applies to all data in the cluster not constrained by a more specific replication zone and cannot be deleted. `.liveness` and `.meta` are explained below in the [Create a Replication Zone for System Ranges](#create-a-replication-zone-for-system-ranges) section and are specifically configured to have a lower-than-default `ttlseconds`. `system.jobs` is also configured with a lower-than-default `ttlseconds` and applies to the `system.jobs` SQL table, which stores metadata about long-running jobs such as schema changes and backups.

### View the Default Replication Zone

The cluster-wide replication zone (`.default`) is initially set to replicate data to any three nodes in your cluster, with ranges in each replica splitting once they get larger than 67108864 bytes.
Expand All @@ -211,7 +223,9 @@ constraints: []

### Edit the Default Replication Zone

To edit the default replication zone, create a YAML file defining only the values you want to change (other values will not be affected), and use the `cockroach zone set .default -f <file.yaml>` command with appropriate flags:
{{site.data.alerts.callout_danger}}Changes to the <code>.default</code> replication zone are not automatically applied to other existing replication zones. If you increase the replication factor for <code>.default</code>, for example, you may also want to increase the replication factor for <a href="#create-a-replication-zone-for-a-system-range">important internal data</a> as well.{{site.data.alerts.end}}

To edit the default replication zone, create a YAML file defining only the values you want to change (other values will be copied from the `.default` zone), and use the `cockroach zone set .default -f <file.yaml>` command with appropriate flags:

{% include copy-clipboard.html %}
~~~ shell
Expand Down Expand Up @@ -333,16 +347,16 @@ Apply zone configurations to corresponding partitions:
$ cockroach zone set roachlearn.students_by_list.australia --insecure -f australia.zone.yml
~~~

### Create a Replication Zone for System Ranges
### Create a Replication Zone for a System Range

In addition to the databases and tables that are visible via the SQL interface, CockroachDB stores additional data in what are called system ranges. There are three categories of system ranges for which replication zones can be set:
In addition to the databases and tables that are visible via the SQL interface, CockroachDB stores internal data in what are called system ranges. CockroachDB comes with pre-configured replication zones for some of these ranges:

Zone Name | Description
----------|------------
**.meta** | The "meta" ranges contain the authoritative information about the location of all data in the cluster. If your cluster is running in multiple datacenters, it's a best practice to configure the meta ranges to have a copy in each datacenter.
**.liveness** | <span class="version-tag">New in v2.0:</span> The "liveness" range contains the authoritative information about which nodes are live at any given time. If the liveness range is unavailable, the entire cluster will be unavailable, so giving it a high replication factor is strongly recommended.
**.system** | The ".system" zone config controls the replication of a variety of important internal data, including information needed to allocate new table IDs and track the health of a cluster's nodes.
**.timeseries** | The "timeseries" ranges contain monitoring data about the cluster that powers the graphs in CockroachDB's admin UI.
----------|-----------------|------------
`.meta` | The "meta" ranges contain the authoritative information about the location of all data in the cluster.<br><br>Because historical queries are never run on meta ranges and it is advantageous to keep these ranges smaller for reliable performance, CockroachDB comes with a **pre-configured** `.meta` replication zone giving these ranges a lower-than-default `ttlseconds`.<br><br>If your cluster is running in multiple datacenters, it's a best practice to configure the meta ranges to have a copy in each datacenter.
`.liveness` | <span class="version-tag">New in v2.0:</span> The "liveness" range contains the authoritative information about which nodes are live at any given time.<br><br>Just as for "meta" ranges, historical queries are never run on the liveness range, so CockroachDB comes with a **pre-configured** `.liveness` replication zone giving this range a lower-than-default `ttlseconds`.<br><br>If this range is unavailable, the entire cluster will be unavailable, so giving it a high replication factor is strongly recommended.
`.timeseries` | The "timeseries" ranges contain monitoring data about the cluster that powers the graphs in CockroachDB's admin UI. If necessary, you can add a `.timeseries` replication zone to control the replication of this data.
`.system` | There are system ranges for a variety of other important internal data, including information needed to allocate new table IDs and track the status of a cluster's nodes. If necessary, you can add a `.system` replication zone to control the replication of this data.

To control replication for one of the above sets of system ranges, create a YAML file defining only the values you want to change (other values will not be affected), and use the `cockroach zone set <zone-name> -f <file.yaml>` command with appropriate flags:

Expand Down Expand Up @@ -721,4 +735,5 @@ There's no need to make zone configuration changes; by default, the cluster is c

## See Also

[Other Cockroach Commands](cockroach-commands.html)
- [Other Cockroach Commands](cockroach-commands.html)
- [Table Partitioning](partitioning.html)
Loading