Changing default rep factor doesn't affect system ranges #2901
Conversation
|
This change is |
There was a problem hiding this comment.
We didn't fully explain the general problem. The issue isn't specific to just the system ranges -- zone configs form a hierarchy of sorts, and the problem is in how that hierarchy is handled.
The system ranges' replication settings are controlled by their specific zone configs (.liveness, .system, etc) if they exist. If they don't exist, they're controlled by .default.
A table partition's replication is controlled by the partition's zone config if it exists. If it doesn't, it's controlled by its table's zone config. If that doesn't exist, it's controlled by the database's zone config. If that doesn't exist, it's controlled by .default.
The issue is that if you haven't defined any of the lower-level configs, then everything falls back to .default. Or analogously, if you haven't defined a table-level zone config, the table will fall back to using the database-level config.
This means that in the common case, particularly in the past when we didn't pre-create any zone configs, setting .default was sufficient for controlling the replication of all data in the cluster.
However, setting .default does not affect any data that already has a more specific zone config in place, regardless of whether it's a system range or not. If you've set a zone config for table foo.bar, changing the .default zone config will not affect how foo.bar is replicated. The same goes for setting a zone config on database foo -- it will have no effect on the foo.bar table if you've already set up a zone config for foo.bar.
The getting-started situation is made much worse by the fact that the cluster now starts with a few zone configs in it for certain system ranges, but I think the documentation needs a little more subtlety than to just say that the .default zone don't affect system ranges.
|
Thanks, @a-robinson. I've tried to add more clarity around pre-configured and user-created replication zones. Not sure if these changes fully address your concerns, though. PTAL. |
v2.0/configure-replication-zones.md
Outdated
| @@ -5,7 +5,7 @@ keywords: ttl, time to live, availability zone | |||
| toc: false | |||
| --- | |||
|
|
|||
| In CockroachDB, you use **replication zones** to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. Initially, there is are a few special pre-configured replication zones for internal system data along with a default replication zone that applies to the rest of the cluster. You can adjust these pre-configured zones as well as add zones for individual databases, tables, and rows ([enterprise-only](enterprise-licensing.html)) as needed. For example, you might use the default zone to replicate most data in a cluster normally within a single datacenter, while creating a specific zone to more highly replicate a certain database or table across multiple datacenters and geographies. | |||
| In CockroachDB, you use **replication zones** to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. Initially, there is are some special pre-configured replication zones for internal system data along with a default replication zone that applies to the rest of the cluster. You can adjust these pre-configured zones as well as add zones for individual databases, tables, and rows ([enterprise-only](enterprise-licensing.html)) as needed. For example, you might use the default zone to replicate most data in a cluster normally within a single datacenter, while creating a specific zone to more highly replicate a certain database or table across multiple datacenters and geographies. | |||
There was a problem hiding this comment.
It's preexisting (and probably my fault), but s/there is are/there are/
v2.0/configure-replication-zones.md
Outdated
|
|
||
| There are four replication zone levels: | ||
| CockroachDB comes with pre-configured replication zones for all user data in the cluster and for some internal system data. These replication zones can be adjusted but not deleted. |
There was a problem hiding this comment.
These replication zones can be adjusted but not deleted.
That's not really true, since you can run cockroach zone rm on any of them except for .default, and if you do so then the corresponding ranges will just be governed by the .default zone.
v2.0/configure-replication-zones.md
Outdated
| - **Row:** ([For enterprise users](enterprise-licensing.html)) You can add replication zones for specific rows in a table by [defining table partitions](partitioning.html). See [Create a Replication Zone for a Table Partition](#create-a-replication-zone-for-a-table-partition-new-in-v2-0) for more details. | ||
| Zone Name | Description | ||
| ----------|------------ | ||
| `.default` | This replication zone applies to all user data in the cluster not constrained by more specific, [user-created replication zones](#user-created-replication-zones). |
There was a problem hiding this comment.
It also doesn't apply to more specific system-created replication zones (the other three zones listed below that show up by default in a freshly created v2.0 cluster)
v2.0/configure-replication-zones.md
Outdated
| ----------|------------ | ||
| `.default` | This replication zone applies to all user data in the cluster not constrained by more specific, [user-created replication zones](#user-created-replication-zones). | ||
| `.meta` | The "meta" ranges contain the authoritative information about the location of all data in the cluster. If these ranges are unavailable, the entire cluster will be unavailable, so this replication zone is pre-configured to have a lower-than-default `ttlseconds`. If your cluster is running in multiple datacenters, it's a best practice to configure the meta ranges to have a copy in each datacenter. | ||
| `.liveness` | <span class="version-tag">New in v2.0:</span> The "liveness" range contains the authoritative information about which nodes are live at any given time. If this range is unavailable, the entire cluster will be unavailable, so this replication zone is pre-configured to have a lower-than-default `ttlseconds`. Giving it a high replication factor is also strongly recommended. |
There was a problem hiding this comment.
If this range is unavailable, the entire cluster will be unavailable, so this replication zone is pre-configured to have a lower-than-default
ttlseconds.
That doesn't logically follow. The lower-than-default ttlseconds is because historical queries are never run on the liveness table and its advantageous to keep the liveness range smaller for reliable performance.
The "If this range is unavailable, the entire cluster will be unavailable" bit is a reason to recommend running it with a higher-than-default replication factor.
There was a problem hiding this comment.
Thanks for the clarification. Reworked.
v2.0/configure-replication-zones.md
Outdated
| `.default` | This replication zone applies to all user data in the cluster not constrained by more specific, [user-created replication zones](#user-created-replication-zones). | ||
| `.meta` | The "meta" ranges contain the authoritative information about the location of all data in the cluster. If these ranges are unavailable, the entire cluster will be unavailable, so this replication zone is pre-configured to have a lower-than-default `ttlseconds`. If your cluster is running in multiple datacenters, it's a best practice to configure the meta ranges to have a copy in each datacenter. | ||
| `.liveness` | <span class="version-tag">New in v2.0:</span> The "liveness" range contains the authoritative information about which nodes are live at any given time. If this range is unavailable, the entire cluster will be unavailable, so this replication zone is pre-configured to have a lower-than-default `ttlseconds`. Giving it a high replication factor is also strongly recommended. | ||
| `system.jobs` | This replication zone controls the replication of a variety of important internal data, including information needed to allocate new table IDs and track the health of a cluster's nodes. It is configured with a lower-than-default `ttlseconds` and applies to the `system.jobs` SQL table, which stores metadata about long-running jobs such as schema changes and backups. |
There was a problem hiding this comment.
This replication zone controls the replication of a variety of important internal data, including information needed to allocate new table IDs and track the health of a cluster's nodes.
That's not true. Looks like you copied from the description of the .system zone?
And speaking of the .system zone, it looks like you've removed all mentions of it. Was that intentional?
v2.0/configure-replication-zones.md
Outdated
|
|
||
| To control replication for one of the above sets of system ranges, create a YAML file defining only the values you want to change (other values will not be affected), and use the `cockroach zone set <zone-name> -f <file.yaml>` command with appropriate flags: | ||
| To control replication for a system range, create a YAML file defining only the values you want to change (other values will not be affected), and use the `cockroach zone set <zone-name> -f <file.yaml>` command with appropriate flags: |
There was a problem hiding this comment.
Optional, but we may want to change "other values will not be affected" to "other values will be copied from the .default zone".
v2.0/known-limitations.md
Outdated
| @@ -10,6 +10,10 @@ This page describes newly identified limitations in the CockroachDB v2.0 release | |||
|
|
|||
| ## New Limitations | |||
|
|
|||
| ### Replication factor for system ranges | |||
|
|
|||
| Changes to the <code>.default</code> replication zone are not automatically applied to other <a href="configure-replication-zones.html#edit-the-replication-zone-for-a-system-range">pre-configured replication zones</a> for internal system data or to any <a href="configure-replication-zones.html#user-created-replication-zones">user-created replication zones</a>. If you increase the replication factor for <code>.default</code>, you may also want to increase the replication factor for <code>.meta</code>, <code>.liveness</code>, and <code>system.jobs</code> to ensure that important internal data is as resilient as your user data. | |||
There was a problem hiding this comment.
Maybe we don't care to make the distinction, but this behavior (changes to .default not affecting pre-existing zones) has always been the case -- the new thing in v2.0 is that there are more pre-configured zones to be affected by the existing problem.
There was a problem hiding this comment.
When put like that, I don't think we need to call this out as a "limitation" per se. It's more of a ux-gotcha, so hopefully covering it in the replication zone docs is enough.
|
Reviewed 3 of 3 files at r2. Comments from Reviewable |
|
@a-robinson, please take another look. Hopefully I'm getting closer. Thanks for all the help here. |
|
This looks great, Jesse! Thanks for sticking with it. Reviewed 5 of 5 files at r3. v2.0/configure-replication-zones.md, line 26 at r3 (raw file):
Minor, but I'd phrase the last sentence as: "Historical queries are never run against this table and the rows in it are updated frequently, so the pre-configured zone gives this table a lower-than-default See cockroachdb/cockroach#21575 for the full original reasoning if you're curious v2.0/configure-replication-zones.md, line 356 at r3 (raw file):
The second sentence's grammar is messed up. I'd say: Because historical queries are never run on meta ranges and it is advantageous to keep these ranges smaller for reliable performance, CockroachDB comes with a pre-configured v2.0/configure-replication-zones.md, line 359 at r3 (raw file):
While I'm throwing nits at you, I know I wrote this but in retrospect I'd replace the word "health" here with "status". v2.0/known-limitations.md, line 15 at r2 (raw file): Previously, jseldess (Jesse Seldess) wrote…
👍 Comments from Reviewable |
|
Review status: all files reviewed at latest revision, 5 unresolved discussions, some commit checks failed. v2.0/configure-replication-zones.md, line 26 at r3 (raw file): Previously, a-robinson (Alex Robinson) wrote…
Done. v2.0/configure-replication-zones.md, line 356 at r3 (raw file): Previously, a-robinson (Alex Robinson) wrote…
Done. v2.0/configure-replication-zones.md, line 359 at r3 (raw file): Previously, a-robinson (Alex Robinson) wrote…
Done. Comments from Reviewable |
- Add known limitation - Add warning to "edit default replication zone" example Fixes #2858
Clarify replication zone levels and warning about increasing the default replication factor without increasing the replication factor of system ranges.
Fixes #2858