Permalink
Apr 23, 2015
May 21, 2015
Apr 23, 2015
May 21, 2015
Apr 23, 2015
Apr 23, 2015
Apr 23, 2015
Apr 23, 2015
Apr 23, 2015
May 21, 2015
Apr 23, 2015
Apr 23, 2015
Apr 23, 2015
May 21, 2015
Apr 23, 2015
Apr 23, 2015
May 21, 2015
Apr 23, 2015
Apr 23, 2015
May 21, 2015
Apr 23, 2015
Apr 23, 2015
Apr 23, 2015
Apr 23, 2015
Apr 23, 2015
Apr 23, 2015
Apr 23, 2015
Apr 23, 2015
Apr 23, 2015
Apr 23, 2015
Apr 23, 2015
Apr 23, 2015
Apr 23, 2015
Newer
100644
1263 lines (1048 sloc)
59.6 KB
1
# About
2
This document is an updated version of the original design documents
3
by Spencer Kimball from early 2014.
4
5
# Overview
6
7
Cockroach is a distributed key:value datastore (SQL and structured
8
data layers of cockroach have yet to be defined) which supports **ACID
9
transactional semantics** and **versioned values** as first-class
10
features. The primary design goal is **global consistency and
11
survivability**, hence the name. Cockroach aims to tolerate disk,
12
machine, rack, and even **datacenter failures** with minimal latency
13
disruption and **no manual intervention**. Cockroach nodes are
14
symmetric; a design goal is **homogenous deployment** (one binary) with
15
minimal configuration.
16
17
Cockroach implements a **single, monolithic sorted map** from key to
18
value where both keys and values are byte strings (not unicode).
19
Cockroach **scales linearly** (theoretically up to 4 exabytes (4E) of
20
logical data). The map is composed of one or more ranges and each range
21
is backed by data stored in [RocksDB](http://rocksdb.org/) (a
22
variant of LevelDB), and is replicated to a total of three or more
23
cockroach servers. Ranges are defined by start and end keys. Ranges are
24
merged and split to maintain total byte size within a globally
25
configurable min/max size interval. Range sizes default to target `64M` in
26
order to facilitate quick splits and merges and to distribute load at
27
hotspots within a key range. Range replicas are intended to be located
28
in disparate datacenters for survivability (e.g. `{ US-East, US-West,
29
Japan }`, `{ Ireland, US-East, US-West}`, `{ Ireland, US-East, US-West,
30
Japan, Australia }`).
31
32
Single mutations to ranges are mediated via an instance of a distributed
33
consensus algorithm to ensure consistency. We’ve chosen to use the
34
[Raft consensus
35
algorithm](https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf).
36
All consensus state is stored in RocksDB.
37
38
A single logical mutation may affect multiple key/value pairs. Logical
39
mutations have ACID transactional semantics. If all keys affected by a
40
logical mutation fall within the same range, atomicity and consistency
41
are guaranteed by Raft; this is the **fast commit path**. Otherwise, a
42
**non-locking distributed commit** protocol is employed between affected
43
ranges.
44
45
Cockroach provides [snapshot isolation](http://en.wikipedia.org/wiki/Snapshot_isolation) (SI) and
46
serializable snapshot isolation (SSI) semantics, allowing **externally
47
consistent, lock-free reads and writes**--both from a historical
48
snapshot timestamp and from the current wall clock time. SI provides
49
lock-free reads and writes but still allows write skew. SSI eliminates
50
write skew, but introduces a performance hit in the case of a
51
contentious system. SSI is the default isolation; clients must
52
consciously decide to trade correctness for performance. Cockroach
53
implements [a limited form of linearizability](#linearizability),
54
providing ordering for any observer or chain of observers.
55
56
Similar to
57
[Spanner](http://static.googleusercontent.com/media/research.google.com/en/us/archive/spanner-osdi2012.pdf)
58
directories, Cockroach allows configuration of arbitrary zones of data.
59
This allows replication factor, storage device type, and/or datacenter
60
location to be chosen to optimize performance and/or availability.
61
Unlike Spanner, zones are monolithic and don’t allow movement of fine
62
grained data on the level of entity groups.
63
64
A
65
[Megastore](http://www.cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf)-like
66
message queue mechanism is also provided to 1) efficiently sideline
67
updates which can tolerate asynchronous execution and 2) provide an
68
integrated message queuing system for asynchronous communication between
69
distributed system components.
70
71
# Architecture
72
73
Cockroach implements a layered architecture. The highest level of
74
abstraction is the SQL layer (currently unspecified in this document).
75
It depends directly on the [*structured data
76
API*](#structured-data-api), which provides familiar relational concepts
77
such as schemas, tables, columns, and indexes. The structured data API
78
in turn depends on the [distributed key value store](#key-value-api),
79
which handles the details of range addressing to provide the abstraction
80
of a single, monolithic key value store. The distributed KV store
81
communicates with any number of physical cockroach nodes. Each node
82
contains one or more stores, one per physical device.
83
84
85
86
Each store contains potentially many ranges, the lowest-level unit of
87
key-value data. Ranges are replicated using the Raft consensus protocol.
88
The diagram below is a blown up version of stores from four of the five
89
nodes in the previous diagram. Each range is replicated three ways using
90
raft. The color coding shows associated range replicas.
91
92
93
94
Each physical node exports a RoachNode service. Each RoachNode exports
95
one or more key ranges. RoachNodes are symmetric. Each has the same
96
binary and assumes identical roles.
97
98
Nodes and the ranges they provide access to can be arranged with various
99
physical network topologies to make trade offs between reliability and
100
performance. For example, a triplicated (3-way replica) range could have
101
each replica located on different:
102
103
- disks within a server to tolerate disk failures.
104
- servers within a rack to tolerate server failures.
105
- servers on different racks within a datacenter to tolerate rack power/network failures.
106
- servers in different datacenters to tolerate large scale network or power outages.
107
108
Up to `F` failures can be tolerated, where the total number of replicas `N = 2F + 1` (e.g. with 3x replication, one failure can be tolerated; with 5x replication, two failures, and so on).
109
110
# Cockroach Client
111
112
In order to support diverse client usage, Cockroach clients connect to
113
any node via HTTPS using protocol buffers or JSON. The connected node
114
proxies involved client work including key lookups and write buffering.
115
116
# Keys
117
118
Cockroach keys are arbitrary byte arrays. If textual data is used in
119
keys, utf8 encoding is recommended (this helps for cleaner display of
120
values in debugging tools). User-supplied keys are encoded using an
121
ordered code. System keys are either prefixed with null characters (`\0`
122
or `\0\0`) for system tables, or take the form of
123
`<user-key><system-suffix>` to sort user-key-range specific system
124
keys immediately after the user keys they refer to. Null characters are
125
used in system key prefixes to guarantee that they sort first.
126
127
# Versioned Values
128
129
Cockroach maintains historical versions of values by storing them with
130
associated commit timestamps. Reads and scans can specify a snapshot
131
time to return the most recent writes prior to the snapshot timestamp.
132
Older versions of values are garbage collected by the system during
133
compaction according to a user-specified expiration interval. In order
134
to support long-running scans (e.g. for MapReduce), all versions have a
135
minimum expiration.
136
137
Versioned values are supported via modifications to RocksDB to record
138
commit timestamps and GC expirations per key.
139
140
# Lock-Free Distributed Transactions
141
142
Cockroach provides distributed transactions without locks. Cockroach
143
transactions support two isolation levels:
144
145
- snapshot isolation (SI) and
146
- *serializable* snapshot isolation (SSI).
147
148
*SI* is simple to implement, highly performant, and correct for all but a
149
handful of anomalous conditions (e.g. write skew). *SSI* requires just a touch
150
more complexity, is still highly performant (less so with contention), and has
151
no anomalous conditions. Cockroach’s SSI implementation is based on ideas from
152
the literature and some possibly novel insights.
153
154
SSI is the default level, with SI provided for application developers
155
who are certain enough of their need for performance and the absence of
156
write skew conditions to consciously elect to use it. In a lightly
157
contended system, our implementation of SSI is just as performant as SI,
158
requiring no locking or additional writes. With contention, our
159
implementation of SSI still requires no locking, but will end up
160
aborting more transactions. Cockroach’s SI and SSI implementations
161
prevent starvation scenarios even for arbitrarily long transactions.
162
163
See the [Cahill paper](https://drive.google.com/file/d/0B9GCVTp_FHJIcEVyZVdDWEpYYXVVbFVDWElrYUV0NHFhU2Fv/edit?usp=sharing)
164
for one possible implementation of SSI. This is another [great paper](http://cs.yale.edu/homes/thomson/publications/calvin-sigmod12.pdf).
165
For a discussion of SSI implemented by preventing read-write conflicts
166
(in contrast to detecting them, called write-snapshot isolation), see
167
the [Yabandeh paper](https://drive.google.com/file/d/0B9GCVTp_FHJIMjJ2U2t6aGpHLTFUVHFnMTRUbnBwc2pLa1RN/edit?usp=sharing),
168
which is the source of much inspiration for Cockroach’s SSI.
169
170
Each Cockroach transaction is assigned a random priority and a
171
"candidate timestamp" at start. The candidate timestamp is the
172
provisional timestamp at which the transaction will commit, and is
173
chosen as the current clock time of the node coordinating the
174
transaction. This means that a transaction without conflicts will
175
usually commit with a timestamp that, in absolute time, precedes the
176
actual work done by that transaction.
177
178
In the course of coordinating the transaction between one or more
179
distributed nodes, the candidate timestamp may be increased to accommodate
180
a later read timestamp for any of the values being read, but will
181
never be decreased. The core difference between the two isolation levels
183
and the latter does not.
184
185
Timestamps are a combination of both a physical and a logical component
186
to support monotonic increments without degenerate cases causing
187
timestamps to diverge from wall clock time, following closely the
188
[*Hybrid Logical Clock
189
paper.*](http://www.cse.buffalo.edu/tech-reports/2014-04.pdf)
190
192
193
1. Start the transaction by writing a new entry to the system
194
transaction table (keys prefixed by *\0tx*) with state “PENDING”.
196
transaction.
197
198
2. Write an "intent" value for each datum being written as part of the
199
transaction. These are normal MVCC values, with the addition of a
200
special flag (i.e. “intent”) indicating that the value may be
202
the transaction id (unique and chosen at tx start time by client)
203
is stored with intent values. The tx id is used to refer to the
204
transaction table when there are conflicts and to make
205
tie-breaking decisions on ordering between identical timestamps.
206
Each node returns the timestamp used for the write (which is the
207
candidate timestamp in the absence of conflicts); the client
208
selects the maximum from amongst all writes as the final commit
209
timestamp.
210
211
Each range maintains a small (i.e. latest 10s of read timestamps),
212
*in-memory* cache from key to the latest timestamp at which the
213
key(s) were read. This *latest-read-cache* is consulted on each
214
write. If the write’s candidate timestamp is earlier than the low
215
water mark on the cache itself (i.e. its last evicted timestamp)
216
or if the key being written has a read timestamp later than the
217
write’s candidate timestamp, this later timestamp value is
218
returned with the write. The cache’s entries are evicted oldest
219
timestamp first, updating low water mark as appropriate. If a new
220
range replica leader is elected, it sets the low water mark for
221
the cache to the current wall time + ε (ε = 99^th^ percentile
222
clock skew).
223
224
3. Commit the transaction by updating its entry in the system
225
transaction table (keys prefixed by *\0tx*). The value of the
226
commit entry contains the candidate timestamp (increased as
227
necessary to accommodate any latest read timestamps). Note that
228
the transaction is considered fully committed at this point and
229
control may be returned to the client.
230
231
In the case of an SI transaction, a commit timestamp which was
232
increased to accommodate concurrent readers is perfectly
233
acceptable and the commit may continue. For SSI transactions,
234
however, a gap between candidate and commit timestamps
235
necessitates transaction restart (note: restart is different than
236
abort--see below).
237
239
removing the “intent” flag. The transaction is considered fully
240
committed before this step and does not wait for it to return
241
control to the transaction coordinator.
242
243
In the absence of conflicts, this is the end. Nothing else is necessary
244
to ensure the correctness of the system.
245
246
**Conflict Resolution**
247
248
Things get more interesting when a reader or writer encounters an intent
249
record or newly-committed value in a location that it needs to read or
250
write. This is a conflict, usually causing either of the transactions to
251
abort or restart depending on the type of conflict.
252
253
***Transaction restart:***
254
255
This is the usual (and more efficient) type of behaviour and is used
256
except when the transaction was aborted (for instance by another
257
transaction).
258
In effect, that reduces to two cases, the first being the one outlined
259
above: An SSI transaction that finds (upon attempting to commit) that
260
its commit timestamp has been pushed. In the second case, a transaction
261
actively encounters a conflict, that is, one of its readers or writers
262
runs encounters data that necessitate conflict resolution.
263
264
When a transaction restarts, it changes its priority and/or moves its
265
timestamp forward depending on data tied to the conflict, and the
267
being written change between restarts, a set of keys written during
268
prior attempts at the transaction is maintained by the client as a set of
269
dirty keys. As it replays the transaction from the beginning, it
270
removes keys from the dirty set as it writes them again. The remaining
271
dirty keys--should the transaction run to completion--are crufty
272
write intents which must be deleted *before* the transaction commit
273
record’s status is set to COMMITTED. Many transactions will end up with
274
no dirty keys.
275
276
***Transaction abort:***
277
278
This is the case in which a transaction, upon reading its transaction
279
table entry, finds that it has been aborted. In this case, the
280
transaction can not reuse its intents; it returns control to the client
281
before cleaning them up (other readers and writers would clean up
282
dangling intents as they encounter them) but will make an effort to
283
clean up after itself. The next attempt (if applicable) then runs as a
287
288
There are several scenarios in which transactions interact:
289
290
- **Reader encounters write intent or value with newer timestamp far
291
enough in the future**: This is not a conflict. The reader is free
292
to proceed; after all, it will be reading an older version of the
293
value and so does not conflict. Recall that the write intent may
294
be committed with a later timestamp than its candidate; it will
295
never commit with an earlier one. **Side note**: if a SI transaction
296
reader finds an intent with a newer timestamp which the reader’s own
297
transaction has written, the reader always returns that value.
298
299
- **Reader encounters write intent or value with newer timestamp in the
300
near future:** In this case, we have to be careful. The newer
301
intent may, in absolute terms, have happened in our read's past if
302
the clock of the writer is ahead of the node serving the values.
303
In that case, we would need to take this value into account, but
304
we just don't know. Hence the transaction restarts, using instead
305
a future timestamp (but remembering a maximum timestamp used to
306
limit the uncertainty window to the maximum clock skew). In fact,
307
this is optimized further; see the details under "choosing a time
308
stamp" below.
309
310
- **Reader encounters write intent with older timestamp**: the reader
311
must follow the intent’s transaction id to the transaction table.
312
If the transaction has already been committed, then the reader can
313
just read the value. If the write transaction has not yet been
314
committed, then the reader has two options. If the write conflict
315
is from an SI transaction, the reader can *push that transaction's
316
commit timestamp into the future* (and consequently not have to
317
read it). This is simple to do: the reader just updates the
318
transaction’s commit timestamp to indicate that when/if the
319
transaction does commit, it should use a timestamp *at least* as
320
high. However, if the write conflict is from an SSI transaction,
321
the reader must compare priorities. If it has the higher priority,
322
it pushes the transaction’s commit timestamp, as with SI (that
323
transaction will then notice its timestamp has been pushed, and
324
restart). If it has the lower priority, it retries itself using as
325
a new priority `max(new random priority, conflicting txn’s
327
328
- **Writer encounters uncommitted write intent**:
329
If the write intent has been written by a transaction with a lower
330
priority writer aborts the conflicting transaction. if the intent has
331
a higher or equal priority the transaction retries, using as a new
332
priority *max(new random priority, conflicting txn’s priority - 1)*;
333
the retry occurs after a short, randomized backoff interval.
334
336
The transaction restarts. On restart, the same priority is reused,
337
but the candidate timestamp is moved forward to the encountered
338
value's timestamp.
339
340
**Transaction management**
341
342
Transactions are managed by the client proxy (or gateway in SQL Azure
343
parlance). Unlike in Spanner, writes are not buffered but are sent
344
directly to all implicated ranges. This allows the transaction to abort
345
quickly if it encounters a write conflict. The client proxy keeps track
346
of all written keys in order to cleanup write intents upon transaction
347
completion.
348
349
If a transaction is completed successfully, all intents are upgraded to
350
committed. In the event a transaction is aborted, all written intents
351
are deleted. The client proxy doesn’t guarantee it will cleanup intents;
352
but dangling intents are upgraded or deleted when encountered by future
353
readers and writers and the system does not depend on their timely
354
cleanup for correctness.
355
356
In the event the client proxy restarts before the pending transaction is
357
completed, the dangling transaction would continue to live in the
358
transaction table until aborted by another transaction. Transactions
359
heartbeat the transaction table every five seconds by default.
360
Transactions encountered by readers or writers with dangling intents
361
which haven’t been heartbeat within the required interval are aborted.
362
363
An exploration of retries with contention and abort times with abandoned
364
transaction is
365
[here](https://docs.google.com/document/d/1kBCu4sdGAnvLqpT-_2vaTbomNmX3_saayWEGYu1j7mQ/edit?usp=sharing).
366
367
**Transaction Table**
368
369
Please see [proto/data.proto](https://github.com/cockroachdb/cockroach/blob/master/proto/data.proto) for the up-to-date structures, the best entry point being `message Transaction`.
370
371
**Pros**
372
373
- No requirement for reliable code execution to prevent stalled 2PC
374
protocol.
375
- Readers never block with SI semantics; with SSI semantics, they may
376
abort.
377
- Lower latency than traditional 2PC commit protocol (w/o contention)
378
because second phase requires only a single write to the
379
transaction table instead of a synchronous round to all
380
transaction participants.
381
- Priorities avoid starvation for arbitrarily long transactions and
382
always pick a winner from between contending transactions (no
383
mutual aborts).
384
- Writes not buffered at client; writes fail fast.
385
- No read-locking overhead required for *serializable* SI (in contrast
386
to other SSI implementations).
387
- Well-chosen (i.e. less random) priorities can flexibly give
388
probabilistic guarantees on latency for arbitrary transactions
389
(for example: make OLTP transactions 10x less likely to abort than
390
low priority transactions, such as asynchronously scheduled jobs).
391
392
**Cons**
393
394
- Reads from non-leader replicas still require a ping to the leader to
395
update *latest-read-cache*.
396
- Abandoned transactions may block contending writers for up to the
397
heartbeat interval, though average wait is likely to be
398
considerably shorter (see [graph in link](https://docs.google.com/document/d/1kBCu4sdGAnvLqpT-_2vaTbomNmX3_saayWEGYu1j7mQ/edit?usp=sharing)).
399
This is likely considerably more performant than detecting and
400
restarting 2PC in order to release read and write locks.
401
- Behavior different than other SI implementations: no first writer
402
wins, and shorter transactions do not always finish quickly.
403
Element of surprise for OLTP systems may be a problematic factor.
404
- Aborts can decrease throughput in a contended system compared with
405
two phase locking. Aborts and retries increase read and write
406
traffic, increase latency and decrease throughput.
407
408
**Choosing a Timestamp**
409
410
A key challenge of reading data in a distributed system with clock skew
411
is choosing a timestamp guaranteed to be greater than the latest
412
timestamp of any committed transaction (in absolute time). No system can
413
claim consistency and fail to read already-committed data.
414
415
Accomplishing this for transactions (or just single operations)
416
accessing a single node is easy. The transaction supplies 0 for
417
timestamp, indicating that the node should use its current time (time
418
for a node is kept using a hybrid clock which combines wall time and a
419
logical time). This guarantees data already committed to that node have
420
earlier timestamps.
421
422
For multiple nodes, the timestamp of the node coordinating the
423
transaction `t` is used. In addition, a maximum timestamp `t+ε` is
424
supplied to provide an upper bound on timestamps for already-committed
425
data (`ε` is the maximum clock skew). As the transaction progresses, any
426
data read which have timestamps greater than `t` but less than `t+ε`
427
cause the transaction to abort and retry with the conflicting timestamp
428
t<sub>c</sub>, where t<sub>c</sub> \> t. The maximum timestamp `t+ε` remains the same.
429
Time spent retrying because of reading recently committed data has an
430
upper bound of `ε`. In fact, this is further optimized: upon restarting,
431
the transaction not only takes into account the timestamp of the future
432
value, but the timestamp of the node at the time of the uncertain read.
433
The larger of those two timestamps (typically, the latter) is used to
434
bump up the read timestamp, and additionally the node is marked as
435
“certain”. This means that for future reads to that node within the
436
transaction, we can set `MaxTimestamp = Read Timestamp` (and hence avoid
437
further uncertainty restarts). Correctness follows from the fact that we
438
know that at the time of the read, there exists no version of any key on
439
that node with a higher timestamp; if we ran into one during a future
440
read, that node would have happened (in absolute time) after our
441
transaction started.
442
443
We expect retries will be rare, but this assumption may need to be
444
revisited if retries become problematic. Note that this problem does not
445
apply to historical reads. An alternate approach which does not require
446
retries would be to make a round to all node participants in advance and
447
choose the highest reported node wall time as the timestamp. However,
448
knowing which nodes will be accessed in advance is difficult and
449
potentially limiting. Cockroach could also potentially use a global
450
clock (Google did this with Pinax TODO: link to paper), which would be
451
feasible for smaller, geographically-proximate clusters.
452
453
# Linearizability
454
455
First a word about [***Spanner***](http://research.google.com/archive/spanner.html).
456
By combining judicious use of wait intervals with accurate time signals,
457
Spanner provides a global ordering between any two non-overlapping transactions
458
(in absolute time) with \~14ms latencies. Put another way:
459
Spanner guarantees that if a transaction T<sub>1</sub> commits (in absolute time)
460
before another transaction T<sub>2</sub> starts, then T<sub>1</sub>'s assigned commit
461
timestamp is smaller than T<sub>2</sub>'s. Using atomic clocks and GPS receivers,
462
Spanner reduces their clock skew uncertainty to \< 10ms (`ε`). To make
463
good on the promised guarantee, transactions must take at least double
464
the clock skew uncertainty interval to commit (`2ε`). See [*this
465
article*](http://www.cs.cornell.edu/~ie53/publications/DC-col51-Sep13.pdf)
466
for a helpful overview of Spanner’s concurrency control.
467
468
Cockroach could make the same guarantees without specialized hardware,
469
at the expense of longer wait times. If servers in the cluster were
470
configured to work only with NTP, transaction wait times would likely to
471
be in excess of 150ms. For wide-area zones, this would be somewhat
472
mitigated by overlap from cross datacenter link latencies. If clocks
473
were made more accurate, the minimal limit for commit latencies would
474
improve.
475
476
However, let’s take a step back and evaluate whether Spanner’s external
477
consistency guarantee is worth the automatic commit wait. First, if the
478
commit wait is omitted completely, the system still yields a consistent
479
view of the map at an arbitrary timestamp. However with clock skew, it
480
would become possible for commit timestamps on non-overlapping but
481
causally related transactions to suffer temporal reverse. In other
482
words, the following scenario is possible for a client without global
483
ordering:
484
485
- Start transaction T<sub>1</sub> to modify value `x` with commit time *s<sub>1</sub>*
486
487
- On commit of T<sub>1</sub>, start T<sub>2</sub> to modify value `y` with commit time
488
\> s<sub>2</sub>
489
490
- Read `x` and `y` and discover that s<sub>1</sub> \> s<sub>2</sub> (**!**)
491
492
The external consistency which Spanner guarantees is referred to as
493
**linearizability**. It goes beyond serializability by preserving
494
information about the causality inherent in how external processes
495
interacted with the database. The strength of Spanner’s guarantee can be
496
formulated as follows: any two processes, with clock skew within
497
expected bounds, may independently record their wall times for the
498
completion of transaction T<sub>1</sub> (T<sub>1</sub><sup>end</sup>) and start of transaction
499
T<sub>2</sub> (T<sub>2</sub><sup>start</sup>) respectively, and if later
500
compared such that T<sub>1</sub><sup>end</sup> \< T<sub>2</sub><sup>start</sup>,
501
then commit timestamps s<sub>1</sub> \< s<sub>2</sub>.
502
This guarantee is broad enough to completely cover all cases of explicit
503
causality, in addition to covering any and all imaginable scenarios of implicit
504
causality.
505
506
Our contention is that causality is chiefly important from the
507
perspective of a single client or a chain of successive clients (*if a
508
tree falls in the forest and nobody hears…*). As such, Cockroach
509
provides two mechanisms to provide linearizability for the vast majority
510
of use cases without a mandatory transaction commit wait or an elaborate
511
system to minimize clock skew.
512
513
1. Clients provide the highest transaction commit timestamp with
514
> successive transactions. This allows node clocks from previous
515
> transactions to effectively participate in the formulation of the
516
> commit timestamp for the current transaction. This guarantees
517
> linearizability for transactions committed by this client.
518
>
519
> Newly launched clients wait at least 2 \* ε from process start
520
> time before beginning their first transaction. This preserves the
521
> same property even on client restart, and the wait will be
522
> mitigated by process initialization.
523
>
524
> All causally-related events within Cockroach maintain
525
> linearizability. Message queues, for example, guarantee that the
526
> receipt timestamp is greater than send timestamp, and that
527
> delivered messages may not be reaped until after the commit wait.
528
529
2. Committed transactions respond with a commit wait parameter which
530
> represents the remaining time in the nominal commit wait. This
531
> will typically be less than the full commit wait as the consensus
532
> write at the coordinator accounts for a portion of it.
533
>
534
> Clients taking any action outside of another Cockroach transaction
535
> (e.g. writing to another distributed system component) can either
536
> choose to wait the remaining interval before proceeding, or
537
> alternatively, pass the wait and/or commit timestamp to the
538
> execution of the outside action for its consideration. This pushes
539
> the burden of linearizability to clients, but is a useful tool in
540
> mitigating commit latencies if the clock skew is potentially
541
> large. This functionality can be used for ordering in the face of
542
> backchannel dependencies as mentioned in the
543
> [AugmentedTime](http://www.cse.buffalo.edu/~demirbas/publications/augmentedTime.pdf)
544
> paper.
545
546
Using these mechanisms in place of commit wait, Cockroach’s guarantee can be
547
formulated as follows: any process which signals the start of transaction
548
T<sub>2</sub> (T<sub>2</sub><sup>start</sup>) after the completion of
549
transaction T<sub>1</sub> (T<sub>1</sub><sup>end</sup>), will have commit
550
timestamps such thats<sub>1</sub> \< s<sub>2</sub>.
551
552
# Logical Map Content
553
554
Logically, the map contains a series of reserved system key / value
555
pairs covering accounting, range metadata, node accounting and
556
permissions before the actual key / value pairs for non-system data
557
(e.g. the actual meat of the map).
558
559
- `\0\0meta1` Range metadata for location of `\0\0meta2`.
560
- `\0\0meta1<key1>` Range metadata for location of `\0\0meta2<key1>`.
561
- ...
562
- `\0\0meta1<keyN>`: Range metadata for location of `\0\0meta2<keyN>`.
563
- `\0\0meta2`: Range metadata for location of first non-range metadata key.
564
- `\0\0meta2<key1>`: Range metadata for location of `<key1>`.
565
- ...
566
- `\0\0meta2<keyN>`: Range metadata for location of `<keyN>`.
567
- `\0acct<key0>`: Accounting for key prefix key0.
568
- ...
569
- `\0acct<keyN>`: Accounting for key prefix keyN.
570
- `\0node<node-address0>`: Accounting data for node 0.
571
- ...
572
- `\0node<node-addressN>`: Accounting data for node N.
573
- `\0perm<key0><user0>`: Permissions for user0 for key prefix key0.
574
- ...
575
- `\0perm<keyN><userN>`: Permissions for userN for key prefix keyN.
576
- `\0tree_root`: Range key for root of range-spanning tree.
577
- `\0tx<tx-id0>`: Transaction record for transaction 0.
578
- ...
579
- `\0tx<tx-idN>`: Transaction record for transaction N.
580
- `\0zone<key0>`: Zone information for key prefix key0.
581
- ...
582
- `\0zone<keyN>`: Zone information for key prefix keyN.
583
- `<>acctd<metric0>`: Accounting data for Metric 0 for empty key prefix.
584
- ...
585
- `<>acctd<metricN>`: Accounting data for Metric N for empty key prefix.
586
- `<key0>`: `<value0>` The first user data key.**
587
- ...
588
- `<keyN>`: `<valueN>` The last user data key.**
589
590
There are some additional system entries sprinkled amongst the
591
non-system keys. See the Key-Prefix Accounting section in this document
592
for further details.
593
594
# Node Storage
595
596
Nodes maintain a separate instance of RocksDB for each disk. Each
597
RocksDB instance hosts any number of ranges. RPCs arriving at a
598
RoachNode are multiplexed based on the disk name to the appropriate
599
RocksDB instance. A single instance per disk is used to avoid
600
contention. If every range maintained its own RocksDB, global management
601
of available cache memory would be impossible and writers for each range
602
would compete for non-contiguous writes to multiple RocksDB logs.
603
604
In addition to the key/value pairs of the range itself, various range
605
metadata is maintained.
606
607
- range-spanning tree node links
608
609
- participating replicas
610
611
- consensus metadata
612
613
- split/merge activity
614
615
A really good reference on tuning Linux installations with RocksDB is
616
[here](http://docs.basho.com/riak/latest/ops/advanced/backends/leveldb/).
617
618
# Range Metadata
619
620
The default approximate size of a range is 64M (2\^26 B). In order to
621
support 1P (2\^50 B) of logical data, metadata is needed for roughly
622
2\^(50 - 26) = 2\^24 ranges. A reasonable upper bound on range metadata
623
size is roughly 256 bytes (*3 \* 12 bytes for the triplicated node
624
locations and 220 bytes for the range key itself*). 2\^24 ranges \* 2\^8
625
B would require roughly 4G (2\^32 B) to store--too much to duplicate
626
between machines. Our conclusion is that range metadata must be
627
distributed for large installations.
628
629
To distribute the range metadata and keep key lookups relatively fast,
630
we use two levels of indirection. All of the range metadata sorts first
631
in our key-value map. We accomplish this by prefixing range metadata
632
with two null characters (*\0\0*). The *meta1* or *meta2* suffixes are
633
additionally appended to distinguish between the first level and second
634
level of **`r`**a***ng***e metadata. In order to do a lookup for *key1*,
635
we first locate the range information for the lower bound of
636
`\0\0meta1<key1>`, and then use that range to locate the lower bound
637
of `\0\0meta2<key1>`. The range specified there will indicate the
638
range location of `<key1>` (refer to examples below). Using two levels
639
of indirection, **our map can address approximately 2\^62 B of data, or
640
roughly 4E** (*each metadata range addresses 2\^(26-8) = 2\^18 ranges;
641
with two levels of indirection, we can address 2\^(18 + 18) = 2\^36
642
ranges; each range addresses 2\^26 B; total is 2\^(36+26) B = 2\^62 B =
643
4E*).
644
645
Note: we append the end key of each range to meta[12] records because
646
the RocksDB iterator only supports a Seek() interface which acts as a
647
Ceil(). Using the start key of the range would cause Seek() to find the
648
key *after* the meta indexing record we’re looking for, which would
649
result in having to back the iterator up, an option which is both less
650
efficient and not available in all cases.
651
652
The following example shows the directory structure for a map with
653
three ranges worth of data. The key/values in red show range
654
metadata. The key/values in black show actual data. Ellipses
655
indicate additional key/value pairs to fill out entire range of
656
data. Except for the fact that splitting ranges requires updates
657
to the range metadata with knowledge of the metadata layout, the
658
range metadata itself requires no special treatment or
659
bootstrapping.
660
661
**Range 0** (located on servers `dcrama1:8000`, `dcrama2:8000`,
662
`dcrama3:8000`)
663
664
- `\0\0meta1\xff`: `dcrama1:8000`, `dcrama2:8000`, `dcrama3:8000`
665
- `\0\0meta2<lastkey0>`: `dcrama1:8000`, `dcrama2:8000`, `dcrama3:8000`
666
- `\0\0meta2<lastkey1>`: `dcrama4:8000`, `dcrama5:8000`, `dcrama6:8000`
667
- `\0\0meta2\xff`: `dcrama7:8000`, `dcrama8:8000`, `dcrama9:8000`
668
- ...
669
- `<lastkey0>`: `<lastvalue0>`
670
671
**Range 1** (located on servers `dcrama4:8000`, `dcrama5:8000`,
672
`dcrama6:8000`)
673
674
- ...
675
- `<lastkey1>`: `<lastvalue1>`
676
677
**Range 2** (located on servers `dcrama7:8000`, `dcrama8:8000`,
678
`dcrama9:8000`)
679
680
- ...
681
- `<lastkey2>`: `<lastvalue2>`
682
683
Consider a simpler example of a map containing less than a single
684
range of data. In this case, all range metadata and all data are
685
located in the same range:
686
687
**Range 0** (located on servers `dcrama1:8000`, `dcrama2:8000`,
688
`dcrama3:8000`)*
689
690
- `\0\0meta1\xff`: `dcrama1:8000`, `dcrama2:8000`, `dcrama3:8000`
691
- `\0\0meta2\xff`: `dcrama1:8000`, `dcrama2:8000`, `dcrama3:8000`
692
- `<key0>`: `<value0>`
693
- `...`
694
695
Finally, a map large enough to need both levels of indirection would
696
look like (note that instead of showing range replicas, this
697
example is simplified to just show range indexes):
698
699
**Range 0**
700
701
- `\0\0meta1<lastkeyN-1>`: Range 0
702
- `\0\0meta1\xff`: Range 1
703
- `\0\0meta2<lastkey1>`: Range 1
704
- `\0\0meta2<lastkey2>`: Range 2
705
- `\0\0meta2<lastkey3>`: Range 3
706
- ...
707
- `\0\0meta2<lastkeyN-1>`: Range 262143
708
709
**Range 1**
710
711
- `\0\0meta2<lastkeyN>`: Range 262144
712
- `\0\0meta2<lastkeyN+1>`: Range 262145
713
- ...
714
- `\0\0meta2\xff`: Range 500,000
715
- ...
716
- `<lastkey1>`: `<lastvalue1>`
717
718
**Range 2**
719
720
- ...
721
- `<lastkey2>`: `<lastvalue2>`
722
723
**Range 3**
724
725
- ...
726
- `<lastkey3>`: `<lastvalue3>`
727
728
**Range 262144**
729
730
- ...
731
- `<lastkeyN>`: `<lastvalueN>`
732
733
**Range 262145**
734
735
- ...
736
- `<lastkeyN+1>`: `<lastvalueN+1>`
737
738
Note that the choice of range `262144` is just an approximation. The
739
actual number of ranges addressable via a single metadata range is
740
dependent on the size of the keys. If efforts are made to keep key sizes
741
small, the total number of addressable ranges would increase and vice
742
versa.
743
744
From the examples above it’s clear that key location lookups require at
745
most three reads to get the value for `<key>`:
746
747
1. lower bound of `\0\0meta1<key>`
748
2. lower bound of `\0\0meta2<key>`,
749
3. `<key>`.
750
751
For small maps, the entire lookup is satisfied in a single RPC to Range 0. Maps
752
containing less than 16T of data would require two lookups. Clients cache both
753
levels of range metadata, and we expect that data locality for individual
754
clients will be high. Clients may end up with stale cache entries. If on a
755
lookup, the range consulted does not match the client’s expectations, the
756
client evicts the stale entries and possibly does a new lookup.
757
758
# Range-Spanning Binary Tree
759
760
A crucial enhancement to the organization of range metadata is to
761
augment the bi-level range metadata lookup with a minimum spanning tree,
762
implemented as a left-leaning red-black tree over all ranges in the map.
763
This tree structure allows the system to start at any key prefix and
764
efficiently traverse an arbitrary key range with minimal RPC traffic,
765
minimal fan-in and fan-out, and with bounded time complexity equal to
766
`2*log N` steps, where `N` is the total number of ranges in the system.
767
768
Unlike the range metadata rows prefixed with `\0\0meta[1|2]`, the
769
metadata for the range-spanning tree (e.g. parent range and left / right
770
child ranges) is stored directly at the ranges as non-map metadata. The
771
metadata for each node of the tree (e.g. links to parent range, left
772
child range, and right child range) is stored with the range metadata.
773
In effect, the tree metadata is stored implicitly. In order to traverse
774
the tree, for example, you’d need to query each range in turn for its
775
metadata.
776
777
Any time a range is split or merged, both the bi-level range lookup
778
metadata and the per-range binary tree metadata are updated as part of
779
the same distributed transaction. The total number of nodes involved in
780
the update is bounded by 2 + log N (i.e. 2 updates for meta1 and
781
meta2, and up to log N updates to balance the range-spanning tree).
782
The range corresponding to the root node of the tree is stored in
784
785
As an example, consider the following set of nine ranges and their
786
associated range-spanning tree:
787
788
R0: `aa - cc`, R1: `*cc - lll`, R2: `*lll - llr`, R3: `*llr - nn`, R4: `*nn - rr`, R5: `*rr - ssss`, R6: `*ssss - sst`, R7: `*sst - vvv`, R8: `*vvv - zzzz`.
789
790
791
792
The range-spanning tree has many beneficial uses in Cockroach. It makes
793
the problem of efficiently aggregating accounting information of
794
potentially vast ranges of data tractable. Imagine a subrange of data
795
over which accounting is being kept. For example, the *photos* table in
796
a public photo sharing site. To efficiently keep track of data about the
797
table (e.g. total size, number of rows, etc.), messages can be passed
798
first up the tree and then down to the left until updates arrive at the
799
key prefix under which accounting is aggregated. This makes worst case
800
number of hops for an update to propagate into the accounting totals
801
2 \* log N. A 64T database will require 1M ranges, meaning 40 hops
802
worst case. In our experience, accounting tasks over vast ranges of data
803
are most often map/reduce jobs scheduled with coarse-grained
804
periodicity. By contrast, we expect Cockroach to maintain statistics
805
with sub 10s accuracy and with minimal cycles and minimal IOPs.
806
807
Another use for the range-spanning tree is to push accounting, zones and
808
permissions configurations to all ranges. In the case of zones and
809
permissions, this is an efficient way to pass updated configuration
810
information with exponential fan-out. When adding accounting
811
configurations (i.e. specifying a new key prefix to track), the
812
implicated ranges are transactionally scanned and zero-state accounting
813
information is computed as well. Deleting accounting configurations is
814
similar, except accounting records are deleted.
815
816
Last but *not* least, the range-spanning tree provides a convenient
817
mechanism for planning and executing parallel queries. These provide the
818
basis for
819
[Dremel](http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/36632.pdf)-like
820
query execution trees and it’s easy to imagine supporting a subset of
821
SQL or even javascript-based user functions for complex data analysis
822
tasks.
823
824
# Raft - Consistency of Range Replicas
825
826
Each range is configured to consist of three or more replicas. The
827
replicas in a range maintain their own instance of a distributed
828
consensus algorithm. We use the [*Raft consensus
829
algorithm*](https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf)
830
as it is simpler to reason about and includes a reference implementation
831
covering important details. Every write to replicas is logged twice.
832
Once to RocksDB’s internal log and once to levedb itself as part of the
833
Raft consensus log.
834
[ePaxos](https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf) has
835
promising performance characteristics for WAN-distributed replicas, but
836
it does not guarantee a consistent ordering between replicas.
837
838
Raft elects a relatively long-lived leader which must be involved to
839
propose writes. It heartbeats followers periodically to keep their logs
840
replicated. In the absence of heartbeats, followers become candidates
841
after randomized election timeouts and proceed to hold new leader
842
elections. Cockroach weights random timeouts such that the replicas with
843
shorter round trip times to peers are more likely to hold elections
844
first. Although only the leader can propose a new write, and as such
845
must be involved in any write to the consensus log, any replica can
846
service reads if the read is for a timestamp which the replica knows is
847
safe based on the last committed consensus write and the state of any
848
pending transactions.
849
850
Only the leader can propose a new write, but Cockroach accepts writes at
851
any replica. The replica merely forwards the write to the leader.
852
Instead of resending the write, the leader has only to acknowledge the
853
write to the forwarding replica using a log sequence number, as though
854
it were proposing it in the first place. The other replicas receive the
855
full write as though the leader were the originator.
856
857
Having a stable leader provides the choice of replica to handle
858
range-specific maintenance and processing tasks, such as delivering
859
pending message queues, handling splits and merges, rebalancing, etc.
860
861
# Splitting / Merging Ranges
862
863
RoachNodes split or merge ranges based on whether they exceed maximum or
864
minimum thresholds for capacity or load. Ranges exceeding maximums for
865
either capacity or load are split; ranges below minimums for *both*
866
capacity and load are merged.
867
868
Ranges maintain the same accounting statistics as accounting key
869
prefixes. These boil down to a time series of data points with minute
870
granularity. Everything from number of bytes to read/write queue sizes.
871
Arbitrary distillations of the accounting stats can be determined as the
872
basis for splitting / merging. Two sensical metrics for use with
873
split/merge are range size in bytes and IOps. A good metric for
874
rebalancing a replica from one node to another would be total read/write
875
queue wait times. These metrics are gossipped, with each range / node
876
passing along relevant metrics if they’re in the bottom or top of the
877
range it’s aware of.
878
879
A range finding itself exceeding either capacity or load threshold
880
splits. To this end, the range leader computes an appropriate split key
881
candidate and issues the split through Raft. In contrast to splitting,
882
merging requires a range to be below the minimum threshold for both
883
capacity *and* load. A range being merged chooses the smaller of the
884
ranges immediately preceding and succeeding it.
885
886
Splitting, merging, rebalancing and recovering all follow the same basic
887
algorithm for moving data between roach nodes. New target replicas are
888
created and added to the replica set of source range. Then each new
889
replica is brought up to date by either replaying the log in full or
890
copying a snapshot of the source replica data and then replaying the log
891
from the timestamp of the snapshot to catch up fully. Once the new
892
replicas are fully up to date, the range metadata is updated and old,
893
source replica(s) deleted if applicable.
894
895
**Coordinator** (leader replica)
896
900
only after being completed locally, are moved to new target replicas.
901
else if merging
902
Choose new replicas on same servers as target range replicas;
903
add to replica set.
904
else if rebalancing || recovering
905
Choose new replica(s) on least loaded servers; add to replica set.
906
```
907
908
**New Replica**
909
910
*Bring replica up to date:*
911
912
```
913
if all info can be read from replicated log
914
copy replicated log
915
else
916
snapshot source replica
917
send successive ReadRange requests to source replica
918
referencing snapshot
919
920
if merging
921
combine ranges on all replicas
922
else if rebalancing || recovering
923
remove old range replica(s)
924
```
925
926
RoachNodes split ranges when the total data in a range exceeds a
927
configurable maximum threshold. Similarly, ranges are merged when the
928
total data falls below a configurable minimum threshold.
929
930
**TBD: flesh this out**.
931
932
Ranges are rebalanced if a node determines its load or capacity is one
933
of the worst in the cluster based on gossipped load stats. A node with
934
spare capacity is chosen in the same datacenter and a special-case split
935
is done which simply duplicates the data 1:1 and resets the range
936
configuration metadata.
937
938
# Message Queues
939
940
Each range maintains an array of incoming message queues, referred to
941
here as **inboxes**. Additionally, each range maintains and *processes*
942
an array of outgoing message queues, referred to here as **outboxes**.
943
Both inboxes and outboxes are assigned to keys; messages can be sent or
944
received on behalf of any key. Inboxes and outboxes can contain any
945
number of pending messages.
946
947
Messages can be *deliverable*, or *executable.*
948
949
Deliverable messages are defined by Value objects - simple byte arrays -
950
that are delivered to a key’s inbox, awaiting collection by a client
951
invoking the ReapQueue operation. These are typically used by client
952
applications wishing to be notified of changes to an entry for further
953
processing, such as expensive offline operations like sending emails,
954
SMSs, etc.
955
956
Executable messages are *outgoing-only*, and are instances of
957
PutRequest,IncrementRequest, DeleteRequest, DeleteRangeRequest
958
orAccountingRequest. Rather than being delivered to a key’s inbox, are
959
executed when encountered. These are primarily useful when updates that
960
are nominally part of a transaction can tolerate asynchronous execution
961
(e.g. eventual consistency), and are otherwise too busy or numerous to
962
make including them in the original [distributed] transaction efficient.
963
Examples may include updates to the accounting for successive key
964
prefixes (potentially busy) or updates to a full-text index (potentially
965
numerous).
966
967
These two types of messages are enqueued in different outboxes too - see
968
key formats below.
969
970
At commit time, the range processing the transaction places messages
971
into a shared outbox located at the start of the range metadata. This is
972
effectively free as it’s part of the same consensus write for the range
973
as the COMMIT record. Outgoing messages are processed asynchronously by
974
the range. To make processing easy, all outboxes are co-located at the
975
start of the range. To make lookup easy, all inboxes are located
976
immediately after the recipient key. The leader replica of a range is
977
responsible for processing message queues.
978
979
A dispatcher polls a given range’s *deliverable message outbox*
980
periodically (configurable), and delivers those messages to the target
981
key’s inbox. The dispatcher is also woken up whenever a new message is
982
added to the outbox. A separate executor also polls the range’s
983
*executable message outbox* periodically as well (again, configurable),
984
and executes those commands. The exeecutor, too, is woken up whenever a
985
new message is added to the outbox.
986
987
Formats follow in the table below. Notice that inbox messages for a
988
given key sort by the `<outbox-timestamp>`. This doesn’t provide a
989
precise ordering, but it does allow clients to scan messages in an
990
approximate ordering of when they were originally lodged with senders.
991
NTP offers walltime deltas to within 100s of milliseconds. The
992
`<sender-range-key>` suffix provides uniqueness.
993
994
**Outbox**
995
`<sender-range-key>deliverable-outbox:<recipient-key><outbox-timestamp>`
996
`<sender-range-key>executable-outbox:<recipient-key><outbox-timestamp>`
997
998
**Inbox**
999
`<recipient-key>inbox:<outbox-timestamp><sender-range-key>`
1000
1001
Messages are processed and then deleted as part of a single distributed
1002
transaction. The message will be executed or delivered exactly once,
1003
regardless of failures at either sender or receiver.
1004
1005
Delivered messages may be read by clients via the ReapQueue operation.
1006
This operation may only be used as part of a transaction. Clients should
1007
commit only after having processed the message. If the transaction is
1008
committed, scanned messages are automatically deleted. The operation
1009
name was chosen to reflect its mutating side effect. Deletion of read
1010
messages is mandatory because senders deliver messages asynchronously
1011
and a delay could cause insertion of messages at arbitrary points in the
1012
inbox queue. If clients require persistence, they should re-save read
1013
messages manually; the ReapQueue operation can be incorporated into
1014
normal transactional updates.
1015
1016
# Node Allocation (via Gossip)
1017
1018
New nodes must be allocated when a range is split. Instead of requiring
1019
every RoachNode to know about the status of all or even a large number
1020
of peer nodes --or-- alternatively requiring a specialized curator or
1021
master with sufficiently global knowledge, we use a gossip protocol to
1022
efficiently communicate only interesting information between all of the
1023
nodes in the cluster. What’s interesting information? One example would
1024
be whether a particular node has a lot of spare capacity. Each node,
1025
when gossiping, compares each topic of gossip to its own state. If its
1026
own state is somehow “more interesting” than the least interesting item
1027
in the topic it’s seen recently, it includes its own state as part of
1028
the next gossip session with a peer node. In this way, a node with
1029
capacity sufficiently in excess of the mean quickly becomes discovered
1030
by the entire cluster. To avoid piling onto outliers, nodes from the
1031
high capacity set are selected at random for allocation.
1032
1033
The gossip protocol itself contains two primary components:
1034
1035
- **Peer Selection**: each node maintains up to N peers with which it
1036
regularly communicates. It selects peers with an eye towards
1037
maximizing fanout. A peer node which itself communicates with an
1038
array of otherwise unknown nodes will be selected over one which
1039
communicates with a set containing significant overlap. Each time
1040
gossip is initiated, each nodes’ set of peers is exchanged. Each
1041
node is then free to incorporate the other’s peers as it sees fit.
1042
To avoid any node suffering from excess incoming requests, a node
1043
may refuse to answer a gossip exchange. Each node is biased
1044
towards answering requests from nodes without significant overlap
1045
and refusing requests otherwise.
1046
1047
Peers are efficiently selected using a heuristic as described in
1048
[Agarwal & Trachtenberg (2006)](https://drive.google.com/file/d/0B9GCVTp_FHJISmFRTThkOEZSM1U/edit?usp=sharing).
1049
1050
**TBD**: how to avoid partitions? Need to work out a simulation of
1051
the protocol to tune the behavior and see empirically how well it
1052
works.
1053
1054
- **Gossip Selection**: what to communicate. Gossip is divided into
1055
topics. Load characteristics (capacity per disk, cpu load, and
1056
state [e.g. draining, ok, failure]) are used to drive node
1057
allocation. Range statistics (range read/write load, missing
1058
replicas, unavailable ranges) and network topology (inter-rack
1059
bandwidth/latency, inter-datacenter bandwidth/latency, subnet
1060
outages) are used for determining when to split ranges, when to
1061
recover replicas vs. wait for network connectivity, and for
1062
debugging / sysops. In all cases, a set of minimums and a set of
1063
maximums is propagated; each node applies its own view of the
1064
world to augment the values. Each minimum and maximum value is
1065
tagged with the reporting node and other accompanying contextual
1066
information. Each topic of gossip has its own protobuf to hold the
1067
structured data. The number of items of gossip in each topic is
1068
limited by a configurable bound.
1069
1070
For efficiency, nodes assign each new item of gossip a sequence
1071
number and keep track of the highest sequence number each peer
1072
node has seen. Each round of gossip communicates only the delta
1073
containing new items.
1074
1075
# Node Accounting
1076
1077
The gossip protocol discussed in the previous section is useful to
1078
quickly communicate fragments of important information in a
1079
decentralized manner. However, complete accounting for each node is also
1080
stored to a central location, available to any dashboard process. This
1081
is done using the map itself. Each node periodically writes its state to
1082
the map with keys prefixed by `\0node`, similar to the first level of
1083
range metadata, but with an ‘`node`’ suffix. Each value is a protobuf
1084
containing the full complement of node statistics--everything
1085
communicated normally via the gossip protocol plus other useful, but
1086
non-critical data.
1087
1088
The range containing the first key in the node accounting table is
1089
responsible for gossiping the total count of nodes. This total count is
1090
used by the gossip network to most efficiently organize itself. In
1091
particular, the maximum number of hops for gossipped information to take
1092
before reaching a node is given by `ceil(log(node count) / log(max
1093
fanout)) + 1`.
1094
1095
# Key-prefix Accounting, Zones & Permissions
1096
1097
Arbitrarily fine-grained accounting and permissions are specified via
1098
key prefixes. Key prefixes can overlap, as is necessary for capturing
1099
hierarchical relationships. For illustrative purposes, let’s say keys
1100
specifying rows in a set of databases have the following format:
1101
1102
`<db>:<table>:<primary-key>[:<secondary-key>]`
1103
1104
In this case, we might collect accounting or specify permissions with
1105
key prefixes:
1106
1107
`db1`, `db1:user`, `db1:order`,
1108
1109
Accounting is kept for the entire map by default.
1110
1111
## Accounting
1112
to keep accounting for a range defined by a key prefix, an entry is created in
1113
the accounting system table. The format of accounting table keys is:
1114
1115
`\0acct<key-prefix>`
1116
1117
In practice, we assume each RoachNode capable of caching the
1118
entire accounting table as it is likely to be relatively small.
1119
1120
Accounting is kept for key prefix ranges with eventual consistency
1121
for efficiency. Updates to accounting values propagate through the
1122
system using the message queue facility if the accounting keys do
1123
not reside on the same range as ongoing activity (true for all but
1124
the smallest systems). There are two types of values which
1125
comprise accounting: counts and occurrences, for lack of better
1126
terms. Counts describe system state, such as the total number of
1127
bytes, rows, etc. Occurrences include transient performance and
1128
load metrics. Both types of accounting are captured as time series
1129
with minute granularity. The length of time accounting metrics are
1130
kept is configurable. Below are examples of each type of
1131
accounting value.
1132
1133
**System State Counters/Performance**
1134
1135
- Count of items (e.g. rows)
1136
- Total bytes
1137
- Total key bytes
1138
- Total value length
1139
- Queued message count
1140
- Queued message total bytes
1141
- Count of values \< 16B
1142
- Count of values \< 64B
1143
- Count of values \< 256B
1144
- Count of values \< 1K
1145
- Count of values \< 4K
1146
- Count of values \< 16K
1147
- Count of values \< 64K
1148
- Count of values \< 256K
1149
- Count of values \< 1M
1150
- Count of values \> 1M
1151
- Total bytes of accounting
1152
1153
1154
**Load Occurences**
1155
1156
Get op count
1157
Get total MB
1158
Put op count
1159
Put total MB
1160
Delete op count
1161
Delete total MB
1162
Delete range op count
1163
Delete range total MB
1164
Scan op count
1165
Scan op MB
1166
Split count
1167
Merge count
1168
1169
Because accounting information is kept as time series and over many
1170
possible metrics of interest, the data can become numerous. Accounting
1171
data are stored in the map near the key prefix described, in order to
1172
distribute load (for both aggregation and storage).
1173
1174
Accounting keys for system state have the form:
1175
`<key-prefix>|acctd<metric-name>*`. Notice the leading ‘pipe’
1176
character. It’s meant to sort the root level account AFTER any other
1177
system tables. They must increment the same underlying values as they
1178
are permanent counts, and not transient activity. Logic at the
1179
RoachNode takes care of snapshotting the value into an appropriately
1180
suffixed (e.g. with timestamp hour) multi-value time series entry.
1181
1182
Keys for perf/load metrics:
1183
`<key-prefix>acctd<metric-name><hourly-timestamp>`.
1184
1185
`<hourly-timestamp>`-suffixed accounting entries are multi-valued,
1186
containing a varint64 entry for each minute with activity during the
1187
specified hour.
1188
1189
To efficiently keep accounting over large key ranges, the task of
1190
aggregation must be distributed. If activity occurs within the same
1191
range as the key prefix for accounting, the updates are made as part
1192
of the consensus write. If the ranges differ, then a message is sent
1193
to the parent range to increment the accounting. If upon receiving the
1194
message, the parent range also does not include the key prefix, it in
1195
turn forwards it to its parent or left child in the balanced binary
1196
tree which is maintained to describe the range hierarchy. This limits
1197
the number of messages before an update is visible at the root to `2*log N`,
1198
where `N` is the number of ranges in the key prefix.
1199
1200
## Zones
1201
zones are stored in the map with keys prefixed by
1202
`\0zone` followed by the key prefix to which the zone
1203
configuration applies. Zone values specify a protobuf containing
1204
the datacenters from which replicas for ranges which fall under
1205
the zone must be chosen.
1206
1207
Please see [proto/config.proto](https://github.com/cockroachdb/cockroach/blob/master/proto/config.proto) for up-to-date data structures used, the best entry point being `message ZoneConfig`.
1208
1209
If zones are modified in situ, each RoachNode verifies the
1210
existing zones for its ranges against the zone configuration. If
1211
it discovers differences, it reconfigures ranges in the same way
1212
that it rebalances away from busy nodes, via special-case 1:1
1213
split to a duplicate range comprising the new configuration.
1214
1215
### Permissions
1216
permissions are stored in the map with keys prefixed by *\0perm* followed by
1217
the key prefix and user to which the specified permissions apply. The format of
1218
permissions keys is:
1219
1220
`\0perm<key-prefix><user>`
1221
1222
Permission values are a protobuf containing the permission details;
1223
please see [proto/config.proto](https://github.com/cockroachdb/cockroach/blob/master/proto/config.proto) for up-to-date data structures used, the best entry point being `message PermConfig`.
1224
1225
A default system root permission is assumed for the entire map
1226
with an empty key prefix and read/write as true.
1227
1228
When determining whether or not to allow a read or a write a key
1229
value (e.g. `db1:user:1` for user `spencer`), a RoachNode would
1230
read the following permissions values:
1231
1232
```
1233
\0perm<db1:user:1>spencer
1234
\0perm<db1:user>spencer
1235
\0perm<db1>spencer
1236
\0perm<>spencer
1237
```
1238
1239
If any prefix in the hierarchy provides the required permission,
1240
the request is satisfied; otherwise, the request returns an
1241
error.
1242
1243
The priority for a user permission is used to order requests at
1244
Raft consensus ranges and for choosing an initial priority for
1245
distributed transactions. When scheduling operations at the Raft
1246
consensus range, all outstanding requests are ordered by key
1247
prefix and each assigned priorities according to key, user and
1248
arrival time. The next request is chosen probabilistically using
1249
priorities to weight the choice. Each key can have multiple
1250
priorities as they’re hierarchical (e.g. for /user/key, one
1251
priority for root ‘/’, and one for ‘/user/key’). The most general
1252
priority is used first. If two keys share the most general, then
1253
they’re compared with the next most general if applicable, and so on.
1254
1255
# Key-Value API
1256
1257
see the protobufs in [proto/](https://github.com/cockroachdb/cockroach/blob/master/proto),
1258
in particular [proto/api.proto](https://github.com/cockroachdb/cockroach/blob/master/proto/api.proto) and the comments within.
1259
1260
# Structured Data API
1261
1262
A preliminary design can be found in the [Go source documentation](http://godoc.org/github.com/cockroachdb/cockroach/structured).