-
Notifications
You must be signed in to change notification settings - Fork 94
/
sizing-feeding.html
631 lines (590 loc) · 24.3 KB
/
sizing-feeding.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
---
# Copyright Vespa.ai. All rights reserved.
title: "Vespa Feed Sizing Guide"
redirect_from:
- /documentation/performance/sizing-feeding.html
---
<p>
Vespa is optimized to sustain a high feed load while serving -
also during planned and unplanned changes to the instance.
This guide provides an overview of how to optimize feed performance and also understand bottlenecks.
</p>
<p>
The <a href="../reads-and-writes.html">reads and writes</a> guide has an overview of the Vespa architecture and relevant
APIs for feeding and searching. One key takeaway is that Vespa is split into two main service types:
<ul>
<li>Stateless container cluster(s)</li>
<li>Stateful content cluster(s)</li>
</ul>
</p>
<p>
The stateless container cluster is responsible for processing all document operations to Vespa.
The stateful content cluster is responsible for writing and syncing all document operations (persisting state and managing data structures).
</p><p>
Generally, Vespa cannot sustain a higher write rate than the underlying storage can handle (MB/s and IOPS). To understand
resource utilization it is critical that resource usage like CPU, memory, disk and network are monitored. Only this way can
a system be sized correctly and bottlenecks identified.
</p>
<h2 id="stateless-container-cluster">Stateless container cluster</h2>
The processing of all document operations to Vespa are routed through the stateless <em>container</em> cluster.
Processing includes both Vespa processing like <a href="../linguistics.html">linguistic processing</a>
and custom <a href="../document-processing.html">document processing</a>. The
stateless container cluster is also responsible for <a href="../embedding.html">embedding</a> inference. Embedding
inference can be compute-resource intensive, depending on number of <em>embed</em> calls and the size of the embedding model.
See <a href="../embedding.html#embedder-performance">embedding performance</a>.
</p>
<p>
The stateless cluster is compute (CPU/GPU util) bound and processing rates should scale linearly with the number of nodes and the number of V-CPU's in the cluster as
long as the client can deliver enough operations over the network.
</p><p>
See <a href="/en/operations-selfhosted/routing.html#multiple-container-clusters">multiple container clusters</a>
for how to separate search and write to different container clusters. Isolated container clusters
are useful for high load scenarios where the container cluster is compute bound and where there is concurrent search and write load
and where we want to avoid write operations to impact search queries due to compute-related resource contention.
</p><p>
The stateless container cluster is implemented in Java (JVM), ensure enough memory allocated for heap to avoid excessive JVM garbage collection.
See <a href="container-tuning.html">stateless container tuning</a> for tuning options. The default max heap size is 1.5GB in self-hosted deployments
unless overridden.
</p>
<h2 id="stateful-content-cluster">Stateful content cluster</h2>
<p>
<strong>All feed operations</strong> to Vespa are <strong>written and synced</strong> to the <a href="../proton.html#transaction-log">transaction log</a> on
the content node(s). This include both writing new documents and updating existing documents.
The Vespa transaction log is a write-ahead log (WAL) that ensures durability of the data. A reply is only sent back to the client
when the operation is written successfully to the transaction log and applied (visible in search/get).
</p>
<p>
The write pattern is append and sequential (not random) IO. Note that Vespa cannot sustain a higher write rate than
the underlaying storage can handle. Feeding might be impacted severely if the content nodes are using network attached storage
where the sync operation (for durability) has a much higher cost than on local attached storage (e.g. SSD).
See <a href="../reference/services-content.html#sync-transactionlog">sync-transactionlog</a>.
</p>
<h3 id="document-store">Document store</h3>
<p>
Documents are written to the <a href="../proton.html#document-store">document store</a> in all
<a href="../reference/services-content.html#document">indexing modes</a> -
this is where the copy of the document is persisted.
</p>
<p>
Adding new documents to the document store is append-only with a sequential IO pattern.
Writing a new version of a document (PUT a document that already exists) is the same as for a
document id that does not exist. The in-memory mapping from
document id to summary data file position is updated to point to the latest version in both cases. The
summary files are <a href="../proton.html#defragmentation">defragmented</a> to remove old versions of documents.
</p>
<h3 id="attribute-store">Attribute store</h3>
<p>
Fields that are defined with the <a href="../attributes.html">attribute</a> property are in-memory fields that
supports in-place updates with higher <a href="../partial-updates.html">partial update</a> throughput than fields that are indexed
with `index` property (avoiding read-apply-write pattern).
The attribute store is a memory-only data structure that is regularly persisted to disk in the <a href="../proton.html#attributes">attribute store</a>.
</p>
<pre>
schema ticker {
document ticker {
field volume type int {
indexing: summary | attribute
}
}
}
</pre>
<p>
See <a href="../partial-updates.html">partial updates</a> for details.
</p>
<h4 id="redundancy-settings">Redundancy settings</h4>
<p>
To achieve memory-only updates (plus transaction log writing),
make sure all attributes to update are <a href="../proton.html#sub-databases">ready</a>,
meaning the content node has loaded the attribute field into memory:
</p>
<ul>
<li>
One way to ensure this is to set
<a href="../reference/services-content.html#searchable-copies">searchable copies</a> equal to
<a href="../reference/services-content.html#redundancy">redundancy</a> -
i.e. all nodes that has a replica of the document has loaded it as searchable
</li>
<li>
Another way is by setting
<a href="../reference/schema-reference.html#attribute">fast-access</a> on each attribute to update
</li>
</ul>
<h3 id="index">Index</h3>
<p>
Changes to index fields are written to the <a href="#document-store">document store</a>
and the <a href="../proton.html#index">index</a>.
Note that an UPDATE operation requires a read-modify-write to the document store and limits throughput.
Refer to <a href="../partial-updates.html">partial updates</a> for more details.
</p>
<pre>
schema music {
document music {
field artist type string {
indexing: summary | index
}
}
}
</pre>
<h3 id="content-node-thread-pools">Content Node Thread pools</h3>
<p>
Several thread pools are involved when handling write operations on a content node.
These are summarized in the following table. Not all mutating operations can be handled in parallel and tracking
these metrics can help identify bottlenecks. For example, if you notice that feed throughput is not increasing beyond
a certain CPU utilization, it might be that one of the thread pools is saturated.
Metrics are available for each thread pool, see
<a href="../reference/vespa-set-metrics-reference.html#searchnode-metrics">searchnode metrics</a>
for details.
</p>
<p>
To analyse performance and bottlenecks, the most relevant metrics are <em>.utilization</em> and <em>.queuesize</em>.
In addition, <em>.saturation</em> is relevant for the <a href="#field-writer-executor">field writer</a> thread pool.
See <a href="#bottlenecks">bottlenecks</a> for details.
</p>
<table class="table">
<thead>
<tr>
<th>Thread pool</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<th rowspan="4">master</th>
<td colspan="2">
<p id="master-thread">
Updates the
<a href="../attributes.html#document-meta-store">document metastore</a>,
prepares tasks to the <a href="#index-thread">index</a> and <a href="#summary-thread">summary</a> threads,
and splits a write operation into a set of tasks to update individual
<a href="../proton.html#attributes">attributes</a>,
executed by the threads in the <a href="#field-writer-executor">field writer</a>.
</p>
</td>
<td></td>
</tr>
<tr>
<th>Threads</th>
<td>1</td>
</tr>
<tr>
<th>Instances</th>
<td>One instance per document database.</td>
</tr>
<tr>
<th style="white-space:nowrap;">Metric prefix</th>
<td><em>content.proton.documentdb.threading_service.master.</em></td>
</tr>
<tr>
<th rowspan="4">index</th>
<td colspan="2">
<p id="index-thread">
Manages writing of index fields in the <a href="../proton.html#index">memory index</a>.
It splits a write operation into a set of tasks to update individual index fields,
executed by the threads in the <a href="#field-writer-executor">field writer</a>.
</p>
</td>
</tr>
<tr>
<th>Threads</th>
<td>1</td>
</tr>
<tr>
<th>Instances</th>
<td>One instance per document database.</td>
</tr>
<tr>
<th style="white-space:nowrap;">Metric prefix</th>
<td><em>content.proton.documentdb.threading_service.index.</em></td>
</tr>
<tr>
<th rowspan="4">summary</th>
<td colspan="2">
<p id="summary-thread">
Writes documents to the
<a href="../proton.html#document-store">document store</a>.
</p>
</td>
</tr>
<tr>
<th>Threads</th>
<td>1</td>
</tr>
<tr>
<th>Instances</th>
<td>One instance per document database.</td>
</tr>
<tr>
<th style="white-space:nowrap;">Metric prefix</th>
<td><em>content.proton.documentdb.threading_service.summary.</em></td>
</tr>
<tr>
<th rowspan="4">field writer</th>
<td colspan="2">
<p id="field-writer-executor">
The threads in this thread pool are used to invert index fields,
write changes to the memory index, and write changes to attributes.
Index fields and attribute fields across all document databases are randomly assigned
to one of the threads in this thread pool.
A field that is costly to write or update might become the bottleneck during feeding.
</p>
</td>
</tr>
<tr>
<th>Threads</th>
<td>Many, controlled by <a href="../reference/services-content.html#feeding">feeding concurrency</a>.</td>
</tr>
<tr>
<th>Instances</th>
<td>One instance shared between all document databases.</td>
</tr>
<tr>
<th style="white-space:nowrap;">Metric prefix</th>
<td><em>content.proton.executor.field_writer.</em></td>
</tr>
<tr>
<th rowspan="4">shared</th>
<td colspan="2">
<p id="shared-executor">
The threads in this thread pool are among other used to compress and
de-compress documents in the
<a href="../proton.html#document-store">document store</a>,
merge files as part of
<a href="../proton.html#disk-index-fusion">disk index fusion</a>,
and prepare for inserting a vector into a
<a href="../reference/schema-reference.html#index-hnsw">HNSW index</a>.
</p>
</td>
</tr>
<tr>
<th>Threads</th>
<td>Many, controlled by <a href="../reference/services-content.html#feeding">feeding concurrency</a>.</td>
</tr>
<tr>
<th>Instances</th>
<td>One instance shared between all document databases.</td>
</tr>
<tr>
<th style="white-space:nowrap;">Metric prefix</th>
<td><em>content.proton.executor.shared.</em></td>
</tr>
</table>
<h2 id="multivalue-attribute">Multivalue attribute</h2>
<p>
<a href="../reference/schema-reference.html#field">Multivalued attributes</a> are
<em>weightedset</em>, <em>array of struct/map</em>, <em>map of struct/map</em> and <em>tensor</em>.
The attributes have different characteristics, which affects write performance.
Generally, updates to multivalue fields are more expensive as the field size grows:
</p>
<table class="table">
<thead>
<tr>
<th>Attribute</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<th>weightedset</th>
<td>
Memory-only operation when updating: read full set, update, write back.
Make the update as inexpensive as possible using numeric types instead of strings, where possible
Example: a weighted set of string with many (1000+) elements.
Adding an element to the set means an enum store lookup/add and add/sort of the attribute multivalue map -
details in <a href="../attributes.html">attributes</a>.
Use a numeric type instead to speed this up - this has no string comparisons.
</td>
</tr><tr>
<th style="white-space: nowrap">array/map of struct/map</th>
<td>
Update to array of struct/map and map of struct/map requires a read from the
<a href="../proton.html#document-store">document store</a> and will reduce update rate -
see <a href="https://github.com/vespa-engine/vespa/issues/10892">#10892</a>.
</td>
</tr><tr>
<th>tensor</th>
<td>
Updating tensor cell values is a memory-only operation: copy tensor, update, write back.
For large tensors, this implicates reading and writing a large chunk of memory for single cell updates.
</td>
</tr>
</tbody>
</table>
<h2 id="parent-child">Parent/child</h2>
<p>
<a href="../parent-child.html">Parent documents</a> are global, i.e. has a replica on all nodes.
Writing to fields in parent documents often simplify logic,
compared to the de-normalized case where <span style="text-decoration: underline">all</span>
(child) documents are updated.
Write performance depends on the average number of child documents vs number of nodes in the cluster - examples:
</p>
<ul>
<li>10-node cluster, avg number of children=100, redundancy=2:
A parent write means 10 writes, compared to 200 writes, or 20x better</li>
<li>50-node cluster, avg number of children=10, redundancy=2:
A parent write means 50 writes, compared to 20 writes, or 2.5x worse</li>
</ul>
<p>
Hence, the more children, the better performance effect for parent writes.
</p>
<h2 id="conditional-updates">Conditional updates</h2>
<p>
A conditional update looks like:
</p>
<pre>
{
"update" : "id:namespace:myDoc::1",
"condition" : "myDoc.myField == \"abc\"",
"fields" : { "myTimestamp" : { "assign" : 1570187817 } }
}
</pre>
<p>
If the <a href="../proton.html#document-store">document store</a>
is accessed when evaluating the condition, performance drops significantly because you get random access instead of
just appending to the persisted data structures.
Conditions should be evaluated using attribute values for high performance -
in the example above, <em>myField</em> should be an attribute.
</p>
<p>
Note: If the condition uses struct or map, values are read from the document store:
</p>
<pre>
"condition" : "myDoc.myMap{1} == 3"
</pre>
<p>
This is true even though all struct fields are defined as attribute.
Improvements to this is tracked in
<a href="https://github.com/vespa-engine/vespa/issues/10892">#10892</a>.
</p>
<!-- ToDo: can probably use the "SummaryAdapter::get" log message above, once made -->
<h2 id="client-roundtrips">Client roundtrips</h2>
<p>
Consider the difference when sending two fields assignments to the same document:
</p>
<pre>
{
"update" : "id:namespace:doctype::1",
"fields" : {
"myMap{1}" : { "assign" : { "timestamp" : 1570187817 } }
"myMap{2}" : { "assign" : { "timestamp" : 1570187818 } }
}
}
</pre>
<p>vs.</p>
<pre>
{
"update" : "id:namespace:doctype::1",
"fields" : {
"myMap{1}" : { "assign" : { "timestamp" : 1570187817 } }
}
}
{
"update" : "id:namespace:doctype::1",
"fields" : {
"myMap{2}" : { "assign" : { "timestamp" : 1570187818 } }
}
}
</pre>
<p>
In the first case, <em>one</em> update operation is sent from
<a href="../vespa-cli.html">vespa feed</a> -
in the latter, the client will send the second update operation <em>after</em> receiving an ack for the first.
When updating multiple fields, put the updates in as few operations as possible.
See <a href="../content/content-nodes.html#ordering">ordering details</a>.
</p>
<h2 id="feed-vs-search">Feed vs. search</h2>
<p>
A content node normally has a fixed set of resources (CPU, memory, disk).
Configure the CPU allocation for feeding vs. searching in
<a href="../reference/services-content.html#feeding">concurrency</a> -
value from 0 to 1.0 - a higher value means more CPU resources for feeding.
</p>
<p>
In addition you can also control priority of feed versus search, or rather how nice feeding shall be.
Since a process needs root privileges for increasing feed, we have opted to reduce priority (be nice)
of feeding. This is controlled by a <a href="../reference/services-content.html#feeding-niceness">niceness</a>
number from 0 to 1.0 - higher value will favor search over feed. 0 is default.
</p>
<h2 id="feed-testing">Feed testing</h2>
<p>
When testing for feeding capacity:
</p>
<ol>
<li>Use <a href="../vespa-cli.html">vespa feed</a>.</li>
<li>Test using one content node to find its capacity and where the bottlenecks are (resource utilization metrics) and Vespa metrics</li>
<li>Test feeding performance by adding feeder instances.
Make sure network and CPU (content and container node) usage increases, until saturation.</li>
<li>See troubleshooting at end to make sure there are no errors.</li>
</ol>
<p>
Other scenarios: Feed testing for capacity for sustained load in a system in steady state,
during state changes, during query load.
</p>
<h2 id="troubleshooting">Troubleshooting</h2>
{% include note.html content="Use the
<a href='/en/operations-selfhosted/monitoring.html#monitoring-with-grafana'>monitoring sample app</a>
to set up a sample system, with a document/query feed and dashboards, to familiarize with metrics." %}
<table class="table">
<tr>
<th>Metrics</th>
<td>
<p id="metrics">
Use <a href="../reference/vespa-set-metrics-reference.html#storage-metrics">metrics</a>
from content nodes and look at queues -
queue wait time and queue size (all metrics in milliseconds):
</p>
<pre>
vds.filestor.averagequeuewait.sum
vds.filestor.queuesize
</pre>
<p>Check content node metrics across all nodes to see if there are any outliers.
Also check latency metrics per operation type:</p>
<pre>
vds.filestor.allthreads.put.latency
vds.filestor.allthreads.update.latency
vds.filestor.allthreads.remove.latency
</pre>
</td>
</tr><tr>
<th>Bottlenecks</th>
<td>
<p id="bottlenecks">
One of the <a href="#content-node-thread-pools">threads</a> used to handle write operations
might become the bottleneck during feeding. Look at the <em>.utilization</em>
metrics for all thread pools:
</p>
<pre>
content.proton.documentdb.threading_service.master.utilization
content.proton.documentdb.threading_service.index.utilization
content.proton.documentdb.threading_service.summary.utilization
content.proton.executor.field_writer.utilization
content.proton.executor.shared.utilization
</pre>
<p>
If utilization is high for
<a href="#field-writer-executor">field writer</a> or
<a href="#shared-executor">shared</a>, adjust
<a href="../reference/services-content.html#feeding">feeding concurrency</a>
to allow more CPU cores to be used for feeding.
</p>
<p>
For the field writer also look at the <em>.saturation</em> metric:
</p>
<pre>
content.proton.executor.field_writer.saturation
</pre>
<p>
If this is close to 1.0 and higher than <em>.utilization</em> it indicates that
one of its worker threads is a bottleneck. The reason can be that this particular thread
is handling a large index or attribute field that is naturally expensive to write and update.
Use the <a href="../proton.html#custom-component-state-api">custom component state API</a>
to find which index and attribute fields are assigned to which thread (identified by <em>executor_id</em>),
and look at the detailed statistics of the field writer to find which thread is the actual bottleneck:
</p>
<pre>
state/v1/custom/component/documentdb/mydoctype/subdb/ready/index
state/v1/custom/component/documentdb/mydoctype/subdb/ready/attributewriter
state/v1/custom/component/threadpools/field_writer
</pre>
</td>
</tr><tr>
<th>Failure rates</th>
<td>
<p id="failure-rates">Inspect these metrics for failures during load testing:</p>
<pre>
vds.distributor.updates.latency
vds.distributor.updates.ok
vds.distributor.updates.failures.total
vds.distributor.puts.latency
vds.distributor.puts.ok
vds.distributor.puts.failures.total
vds.distributor.removes.latency
vds.distributor.removes.ok
vds.distributor.removes.failures.total
</pre>
</td>
</tr><tr>
<th>Blocked feeding</th>
<td>
<p id="blocked-feeding">
This metric should be 0 - refer to <a href="../operations/feed-block.html">feed block</a>:
</p>
<pre>
content.proton.resource_usage.feeding_blocked
</pre>
</td>
</tr><tr>
<th style="white-space: nowrap">Concurrent mutations</th>
<td>
<p id="concurrent-mutations">Multiple clients updating the same document concurrently will stall writes:</p>
<pre>
vds.distributor.updates.failures.concurrent_mutations
</pre>
Mutating client operations towards a given document ID are sequenced on the
<a href="../content/content-nodes.html#distributor">distributors</a>.
If an operation is already active towards a document,
a subsequently arriving one will be bounced back to the client with a transient failure code.
Usually this happens when users send feed from multiple clients concurrently without synchronisation.
Note that feed operations sent by a single client are sequenced client-side,
so this should not be observed with a single client only.
Bounced operations are never sent on to the backends and should not cause elevated latencies there,
although the client will observe higher latencies due to automatic retries with back-off.
</td>
</tr><tr>
<th>Wrong distribution</th>
<td>
<p id="wrong-distribution"></p>
<pre>
vds.distributor.updates.failures.wrongdistributor
</pre>
Indicates that clients keep sending to the wrong distributor.
Normally this happens infrequently
(but is <em>does</em> happen on client startup or distributor state transitions),
as clients update and cache all state required to route directly to the correct distributor
(Vespa uses a deterministic CRUSH-based algorithmic distribution).
Some potential reasons for this:
<ol>
<li>Clients are being constantly re-created with no cached state.</li>
<li>The system is in some kind of flux where the underlying state keeps changing constantly.</li>
<li>The client distribution policy has received so many errors
that it throws away its cached state to start with a clean slate to
e.g. avoid the case where it only has cached information for the bad side of a network partition.</li>
<li>The system has somehow failed to converge to a shared cluster state,
causing parts of the cluster to have a different idea of the correct state than others.</li>
</ol>
</td>
</tr><tr>
<th>Cluster out of sync</th>
<td>
<p id="cluster-out-of-sync">
<em>update_puts/gets</em> indicate "two-phase" updates: <!-- ToDo: explain two-phase updates / link -->
</p>
<pre>
vds.distributor.update_puts.latency
vds.distributor.update_puts.ok
vds.distributor.update_gets.latency
vds.distributor.update_gets.ok
vds.distributor.update_gets.failures.total
vds.distributor.update_gets.failures.notfound
</pre>
<p>
If replicas are out of sync,
updates cannot be applied directly on the replica nodes
as they risk ending up with diverging state.
In this case, Vespa performs an explicit read-consolidate-write (write repair) operation on the distributors.
This is usually a lot slower than the regular update path because it doesn't happen in parallel.
It also happens in the write-path of other operations,
so risks blocking these if the updates are expensive in terms of CPU.
</p><p>
Replicas being out of sync is by definition not the expected steady state of the system.
For example, replica divergence can happen if one or more replica nodes are unable to process or persist operations.
Track (pending) merges:
</p>
<pre>
vds.idealstate.buckets
vds.idealstate.merge_bucket.pending
vds.idealstate.merge_bucket.done_ok
vds.idealstate.merge_bucket.done_failed
</pre>
</td>
</tr>
</table>