/
sharding-balancer-administration.txt
287 lines (209 loc) · 10.7 KB
/
sharding-balancer-administration.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
.. index:: balancing; migration
.. _sharding-balancing:
========================
Sharded Cluster Balancer
========================
.. default-domain:: mongodb
.. contents:: On this page
:local:
:backlinks: none
:depth: 1
:class: singlecol
The MongoDB balancer is a background process that monitors the number of
:term:`chunks <chunk>` on each :term:`shard`. When the number of chunks on a
given shard reaches specific :ref:`migration thresholds
<sharding-migration-thresholds>`, the balancer attempts to automatically
migrate chunks between shards and reach an equal number of chunks per shard.
The balancing procedure for :term:`sharded clusters <sharded cluster>` is
entirely transparent to the user and application layer, though there may be
some performance impact while the procedure takes place.
.. include:: /images/sharding-migrating.rst
.. include:: /includes/toc/sharding-balancing.rst
.. index:: balancing; internals
.. _sharding-balancing-internals:
.. _sharding-internals-balancing:
Cluster Balancer
----------------
The :term:`balancer` process is responsible for redistributing the
chunks of a sharded collection evenly among the shards for every
sharded collection. By default, the balancer process is always enabled.
Any :binary:`~bin.mongos` instance in the cluster can start a balancing
round. When a balancer process is active, the responsible
:binary:`~bin.mongos` acquires a "lock" by modifying a document in the
``lock`` collection in the :ref:`config-database`.
.. versionchanged:: 3.2
With replica set config servers, clock skew does not affect
distributed lock management. If you are using *mirrored* config
servers, large differences in timekeeping can lead to failed
distributed locks. With *mirrored* config servers, minimize clock skew
by running the network time protocol (NTP) ``ntpd`` on your servers.
To address uneven chunk distribution for a sharded collection, the
balancer :doc:`migrates chunks </core/sharding-balancer-administration>` from
shards with more chunks to shards with a fewer number of chunks. The
balancer migrates the chunks, one at a time, until there is an even
distribution of chunks for the collection across the shards. For details
about chunk migration, see :ref:`chunk-migration-procedure`.
.. include:: /includes/fact-archiveMovedChunks.rst
Chunk migrations carry some overhead in terms of bandwidth and
workload, both of which can impact database performance. The
:term:`balancer` attempts to minimize the impact by:
- Moving only one chunk at a time. See also
:ref:`chunk-migration-queuing`.
- Starting a balancing round **only** when the difference in the
number of chunks between the shard with the greatest number of chunks
for a sharded collection and the shard with the lowest number of
chunks for that collection reaches the :ref:`migration threshold
<sharding-migration-thresholds>`.
You may disable the balancer temporarily for maintenance. See
:ref:`sharding-balancing-disable-temporally` for details.
You can also limit the window during which the balancer runs to prevent
it from impacting production traffic. See :ref:`Schedule the Balancing
Window <sharding-schedule-balancing-window>` for details.
.. note::
The specification of the balancing window is relative to the local
time zone of all individual :binary:`~bin.mongos` instances in the
cluster.
.. seealso:: :doc:`/tutorial/manage-sharded-cluster-balancer`.
Adding and Removing Shards from the Cluster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Adding a shard to a cluster creates an imbalance, since the new
shard has no chunks. While MongoDB begins migrating data to the new
shard immediately, it can take some time before the cluster balances. See the
:doc:`/tutorial/add-shards-to-shard-cluster` tutorial for instructions on
adding a shard to a cluster.
Removing a shard from a cluster creates a similar imbalance, since chunks
residing on that shard must be redistributed throughout the cluster. While
MongoDB begins draining a removed shard immediately, it can take some time
before the cluster balances. *Do not* shutdown the servers associated
to the removed shard during this process.
.. include:: /includes/fact-remove-shard-balance-order.rst
See the :doc:`/tutorial/remove-shards-from-cluster` tutorial for
instructions on safely removing a shard from a cluster.
.. _chunk-migration-procedure:
Chunk Migration Procedure
-------------------------
All chunk migrations use the following procedure:
#. The balancer process sends the :dbcommand:`moveChunk` command to
the source shard.
#. The source starts the move with an internal :dbcommand:`moveChunk`
command. During the migration process, operations to the chunk
route to the source shard. The source shard is responsible for
incoming write operations for the chunk.
#. The destination shard builds any indexes required by the source
that do not exist on the destination.
#. The destination shard begins requesting documents in the chunk and
starts receiving copies of the data.
#. After receiving the final document in the chunk, the destination
shard starts a synchronization process to ensure that it has the
changes to the migrated documents that occurred during the migration.
#. When fully synchronized, the source shard connects to the
:term:`config database` and updates the cluster metadata with the new
location for the chunk.
#. After the source shard completes the update of the metadata,
and once there are no open cursors on the chunk, the source shard
deletes its copy of the documents.
.. note::
If the balancer needs to perform additional chunk migrations from
the source shard, the balancer can start the next chunk migration
without waiting for the current migration process to finish this
deletion step. See :ref:`chunk-migration-queuing`.
.. versionchanged:: 2.6
The source shard automatically archives the migrated documents by
default. For more information, see :ref:`moveChunk-directory`.
The migration process ensures consistency and maximizes the availability of
chunks during balancing.
.. _sharding-migration-thresholds:
Migration Thresholds
~~~~~~~~~~~~~~~~~~~~
To minimize the impact of balancing on the cluster, the
:term:`balancer` only begins balancing after the distribution of
chunks for a sharded collection has reached certain thresholds. The
thresholds apply to the difference in number of :term:`chunks <chunk>`
between the shard with the most chunks for the collection and the shard
with the fewest chunks for that collection. The balancer has the
following thresholds:
.. list-table::
:header-rows: 1
* - Number of Chunks
- Migration Threshold
* - Fewer than 20
- 2
* - 20-79
- 4
* - 80 and greater
- 8
The balancer stops running on the target collection when the difference
between the number of chunks on any two shards for that collection is *less
than two*, or a chunk migration fails.
.. _chunk-migration-queuing:
.. _asynchronous-chunk-migration-cleanup:
Asynchronous Chunk Migration Cleanup
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To migrate multiple chunks from a shard, the balancer migrates the
chunks one at a time. However, the balancer does not wait for the
current migration's delete phase to complete before starting the next
chunk migration. See :ref:`sharding-chunk-migration` for the chunk
migration process and the delete phase.
This queuing behavior allows shards to unload chunks more quickly in
cases of heavily imbalanced cluster, such as when performing initial
data loads without pre-splitting and when adding new shards.
This behavior also affects the :dbcommand:`moveChunk` command, and
migration scripts that use the :dbcommand:`moveChunk` command may
proceed more quickly.
In some cases, the delete phases may persist longer. If multiple delete
phases are queued but not yet complete, a crash of the replica set's
primary can orphan data from multiple migrations.
The ``_waitForDelete``, available as a setting for the balancer as well
as the :dbcommand:`moveChunk` command, can alter the behavior so that
the delete phase of the current migration blocks the start of the next
chunk migration. The ``_waitForDelete`` is generally for internal
testing purposes. For more information, see
:ref:`wait-for-delete-setting`.
.. _chunk-migration-replication:
Chunk Migration and Replication
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. versionchanged:: 3.0
The default value ``secondaryThrottle`` became ``true`` for all
chunk migrations.
The new ``writeConcern`` field in the balancer configuration
document allows you to specify a :doc:`write concern
</reference/write-concern>` semantics with the
``_secondaryThrottle`` option.
By default, each document operation during chunk migration propagates to at
least one secondary before the balancer proceeds with the next
document, which is equivalent to a write concern of ``{ w: 2 }``. You
can set the ``writeConcern`` option on the balancer configuration to
set different write concern semantics.
To override this behavior and allow the balancer to continue without
waiting for replication to a secondary, set the ``_secondaryThrottle``
parameter to ``false``. See :ref:`sharded-cluster-config-secondary-throttle`
to update the ``_secondaryThrottle`` parameter for the balancer.
For the :dbcommand:`moveChunk` command, the ``secondaryThrottle``
parameter is independent of the ``_secondaryThrottle`` parameter for
the balancer.
Independent of the ``secondaryThrottle`` setting, certain phases of
the chunk migration have the following replication policy:
- MongoDB briefly pauses all application reads and writes to the
collection being migrated, on the source shard, before updating the
config servers with the new location for the chunk, and resumes the
application reads and writes after the update. The chunk move requires
all writes to be acknowledged by majority of the members of the
replica set both before and after committing the chunk move to config
servers.
- When an outgoing chunk migration finishes and cleanup occurs, all
writes must be replicated to a majority of servers before further
cleanup (from other outgoing migrations) or new incoming migrations
can proceed.
.. _migration-chunk-size-limit:
Maximum Number of Documents Per Chunk to Migrate
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/limits-sharding-maximum-documents-chunk.rst
.. _sharding-shard-size:
Shard Size
----------
By default, MongoDB attempts to fill all available disk space with
data on every shard as the data set grows. To ensure that the cluster
always has the capacity to handle data growth, monitor disk
usage as well as other performance metrics.
See the :ref:`sharded-cluster-config-max-shard-size` tutorial for
instructions on setting the maximum size for a shard.