rgw: Add dynamic resharding documentation #15941

oritwas · 2017-06-27T09:27:43Z

Fixes: http://tracker.ceph.com/issues/21553
Signed-off-by: Orit Wasserman owasserm@redhat.com

mattbenjamin · 2017-06-27T13:17:19Z

doc/radosgw/dynamicresharding.rst

+
+.. versionadded:: Luminous
+
+The dynamic resharding feature automatically increase the number of bucket index shards.


s/increase/increases/

mattbenjamin · 2017-06-27T13:17:48Z

doc/radosgw/dynamicresharding.rst

+.. versionadded:: Luminous
+
+The dynamic resharding feature automatically increase the number of bucket index shards.
+It is a background process that periodically checks all the buckets and detect buckets require resharding.


s/require/that require/

mattbenjamin · 2017-06-27T13:19:54Z

doc/radosgw/dynamicresharding.rst

+ ::
+   # radosgw-admin bucket  reshard process
+
+Cancel bucket resharding


scheduled? can this cancel a resharding already in progress?

theanalyst · 2017-08-08T16:31:00Z

@mattbenjamin are you ok with the changes

theanalyst · 2017-08-08T16:31:54Z

doc/radosgw/dynamicresharding.rst

+
+The dynamic resharding feature automatically increases the number of bucket index shards.
+It is a background process that periodically checks all the buckets and detect buckets that require resharding.
+Those bucket are inserted into the resharding queue and will be resharded in the background.


theanalyst · 2017-08-08T16:33:02Z

doc/radosgw/dynamicresharding.rst

+
+.. versionadded:: Luminous
+
+The dynamic resharding feature automatically increases the number of bucket index shards.


Maybe to be more clear we mention dynamic bucket index resharding, though I'm ok either way

theanalyst · 2017-08-08T16:33:45Z

doc/radosgw/dynamicresharding.rst

+List resharding queue
+------------------------------
+::
+   # radosgw-admon bucket  reshard list


list resharding queue by 'radosgw-admin reshard list'

mattbenjamin · 2017-08-18T13:47:17Z

@oritwas @theanalyst sparse, but lgtm

alfredodeza · 2017-08-19T00:08:26Z

jenkins test docs

mattbenjamin

I marked a few minor grammar points, but content-wise, this looks ready to merge.

mattbenjamin · 2017-09-27T11:09:00Z

doc/radosgw/dynamicresharding.rst

+ online bucket resharding.
+
+Each bucket index shard  can handle efficiently up to a threshold of entries, when exceeding this threshold the system
+will encounters performance issues.


if this threshold is exceeded, the system will encounter

mattbenjamin · 2017-09-27T11:09:49Z

doc/radosgw/dynamicresharding.rst

+
+.. versionadded:: Luminous
+
+ A large bucket index can lead to performance problems, in order to address this problem we introduced bucket index sharding.


problems. In order

mattbenjamin · 2017-09-27T11:10:19Z

doc/radosgw/dynamicresharding.rst

+
+Each bucket index shard  can handle efficiently up to a threshold of entries, when exceeding this threshold the system
+will encounters performance issues.
+The dynamic resharding feature detects this situation and increase automatically the number of shards used by bucket index,


the bucket index

mattbenjamin · 2017-09-27T11:10:31Z

doc/radosgw/dynamicresharding.rst

+Each bucket index shard  can handle efficiently up to a threshold of entries, when exceeding this threshold the system
+will encounters performance issues.
+The dynamic resharding feature detects this situation and increase automatically the number of shards used by bucket index,
+resulting in a reduction in  number of entries in each bucket index shard. This process is transparent to the user.


mattbenjamin · 2017-09-27T11:10:42Z

doc/radosgw/dynamicresharding.rst

+The dynamic resharding feature detects this situation and increase automatically the number of shards used by bucket index,
+resulting in a reduction in  number of entries in each bucket index shard. This process is transparent to the user.
+
+The detection process consist of two parts:


btw, great section

oritwas · 2017-09-27T17:18:54Z

@mattbenjamin , fixed

yehudasa

@oritwas see my comments

yehudasa · 2017-09-27T17:54:30Z

doc/radosgw/dynamicresharding.rst

+.. versionadded:: Luminous
+
+ A large bucket index can lead to performance problems. In order to address this problem we introduced bucket index sharding.
+ Until Luminous changing the number of bucket shards (resharding) required to offline the bucket, from Luminous we support


s/required to offline the bucket/needed to be done offline

yehudasa · 2017-09-27T17:57:10Z

doc/radosgw/dynamicresharding.rst

+ Until Luminous changing the number of bucket shards (resharding) required to offline the bucket, from Luminous we support
+ online bucket resharding.
+
+Each bucket index shard can handle efficiently up to a threshold of entries, if this threshold is exceeded the system


Each bucket index shard can handle its entries efficiently up until reaching a certain threshold number of entries. If this threashold is exceeded, the system could encounter performance issues.

yehudasa · 2017-09-27T17:57:37Z

doc/radosgw/dynamicresharding.rst

+
+Each bucket index shard can handle efficiently up to a threshold of entries, if this threshold is exceeded the system
+will encounters performance issues.
+The dynamic resharding feature detects this situation and increase automatically the number of shards used by the bucket index,


and increases

yehudasa · 2017-09-27T17:59:02Z

doc/radosgw/dynamicresharding.rst

+resulting in the reduction of the number of entries in each bucket index shard. This process is transparent to the user.
+
+The detection process consists of two parts:
+The first is executed during addition of new objects to the bucket and the second is  a background process that periodically scans all the


The first part happens when new objects are added to the bucket, and the second part is via a background process that periodically scans all the buckets.

yehudasa · 2017-09-27T18:00:47Z

doc/radosgw/dynamicresharding.rst

+
+The detection process consists of two parts:
+The first is executed during addition of new objects to the bucket and the second is  a background process that periodically scans all the
+buckets to handler older buckets in the system that are not being updated.


s/to handler .../This is needed in order to deal with existing buckets that are not being updated.

yehudasa · 2017-09-27T18:01:08Z

doc/radosgw/dynamicresharding.rst

+The detection process consists of two parts:
+The first is executed during addition of new objects to the bucket and the second is  a background process that periodically scans all the
+buckets to handler older buckets in the system that are not being updated.
+A bucket that requires resharding is added to  the reshard_log queue and will be scheduled to be resharding later.


s/to be resharding/to be resharded

yehudasa · 2017-09-27T18:12:01Z

doc/radosgw/dynamicresharding.rst

+The first is executed during addition of new objects to the bucket and the second is  a background process that periodically scans all the
+buckets to handler older buckets in the system that are not being updated.
+A bucket that requires resharding is added to  the reshard_log queue and will be scheduled to be resharding later.
+The reshard threads run in the background and executes the scheduled resharding, one at a time.


s/executes/execute

yehudasa · 2017-09-27T18:16:05Z

doc/radosgw/dynamicresharding.rst

+
+-``rgw_reshard_bucket_lock_duration``: duration of lock on bucket obj during resharding, default:  120 seconds.
+
+-``rgw_max_objs_per_shard``: Maximal number of objects per bucket index shard, default: 100000 objects.


s/Maximal/maximum

yehudasa · 2017-09-27T18:16:16Z

doc/radosgw/dynamicresharding.rst

+
+Parameters to control the resahrding process in Ceph configuration fie:
+
+-``rgw_reshard_num_logs``: Number of shards for the resharding log, default: 16


s/Number/number

alfredodeza · 2017-10-02T14:36:39Z

doc/radosgw/dynamicresharding.rst

+
+Each bucket index shard can handle its entries efficiently up until
+reaching a certain threshold of entries. If this threshold is exceeded the system
+could encounters performance issues.


s/encounters/encounter/

alfredodeza · 2017-10-02T14:37:10Z

doc/radosgw/dynamicresharding.rst

+
+A large bucket index can lead to performance problems. In order
+to address this problem we introduced bucket index sharding.
+Until Luminous changing the number of bucket shards (resharding)


s/Until Luminous/Until Luminous,/

alfredodeza · 2017-10-02T14:40:04Z

doc/radosgw/dynamicresharding.rst

+resulting in the reduction of the number of entries in each bucket index shard.
+This process is transparent to the user.
+
+The detection process consists of two parts:


This could be automatically enumerated, which would allow you to remove 'first' and 'second':

The detection process consists of: #. Objects are added to the bucket #. A background process that periodically scans all the buckets

alfredodeza · 2017-10-02T14:40:25Z

doc/radosgw/dynamicresharding.rst

+The detection process consists of two parts:
+The first part happens when new objects are added to the bucket,
+and the second is via a background process that periodically scans all the buckets.
+This is needed in order to deal with exisitng buckets in the system that are not being updated.


s/exisiting/existing/

alfredodeza · 2017-10-02T14:41:06Z

doc/radosgw/dynamicresharding.rst

+The first part happens when new objects are added to the bucket,
+and the second is via a background process that periodically scans all the buckets.
+This is needed in order to deal with exisitng buckets in the system that are not being updated.
+A bucket that requires resharding is added to the reshard_log queue and will be


Seems like 'reshard_log' could use backticks to be rendered in monospaced fonts, like ``reshard_log``

alfredodeza · 2017-10-02T14:43:11Z

doc/radosgw/dynamicresharding.rst

+
+
+Admin commands
+===============


this underline needs to match the length of the title

alfredodeza · 2017-10-02T14:43:18Z

doc/radosgw/dynamicresharding.rst

+  # radosgw-admin reshard add  --bucket <bucket_name> --num-shards <new number of shards>
+
+List resharding queue
+------------------------------


this underline needs to match the length of the sub-title

alfredodeza · 2017-10-02T14:43:37Z

doc/radosgw/dynamicresharding.rst

+   # radosgw-admon reshard list
+
+Process/Schedule a bucket resharding
+-----------------------------------------------------


same length issue

alfredodeza · 2017-10-02T14:43:41Z

doc/radosgw/dynamicresharding.rst

+   # radosgw-admin reshard process
+
+Cancel pending bucket resharding
+----------------------------------------------


same length issue

alfredodeza · 2017-10-02T14:43:47Z

doc/radosgw/dynamicresharding.rst

+
+
+Manual  bucket resharding
+------------------------------------


same length issue

oritwas · 2017-10-10T09:56:17Z

@theanalyst @alfredodeza updated

theanalyst · 2017-10-10T11:07:11Z

doc/radosgw/dynamicresharding.rst

+Until Luminous, changing the number of bucket shards (resharding)
+needed to be done offline, from Luminous we support
+online bucket resharding.
+


can you add a note here mentioning not to use this in multisite scenarios

right, will add it

changes addressed

Fixes: http://tracker.ceph.com/issues/21553 Signed-off-by: Orit Wasserman <owasserm@redhat.com>

oritwas added documentation rgw labels Jun 27, 2017

oritwas assigned yehudasa Jun 27, 2017

oritwas requested a review from yehudasa June 27, 2017 09:27

mattbenjamin reviewed Jun 27, 2017

View reviewed changes

oritwas force-pushed the wip-rgw-dynamic-resharding-doc branch 2 times, most recently from 50f1872 to e97d742 Compare June 27, 2017 14:31

theanalyst previously requested changes Aug 8, 2017

View reviewed changes

oritwas force-pushed the wip-rgw-dynamic-resharding-doc branch 8 times, most recently from e9612b0 to 052bbdf Compare September 27, 2017 07:57

mattbenjamin reviewed Sep 27, 2017

View reviewed changes

oritwas force-pushed the wip-rgw-dynamic-resharding-doc branch from 052bbdf to 5bc9e18 Compare September 27, 2017 14:35

yehudasa reviewed Sep 27, 2017

View reviewed changes

oritwas force-pushed the wip-rgw-dynamic-resharding-doc branch 6 times, most recently from 7d4115e to 2f9348d Compare September 28, 2017 06:20

alfredodeza suggested changes Oct 2, 2017

View reviewed changes

oritwas force-pushed the wip-rgw-dynamic-resharding-doc branch 8 times, most recently from 2d88ae0 to 98d1b29 Compare October 8, 2017 13:33

theanalyst reviewed Oct 10, 2017

View reviewed changes

alfredodeza approved these changes Oct 10, 2017

View reviewed changes

oritwas force-pushed the wip-rgw-dynamic-resharding-doc branch 2 times, most recently from 71c5abe to 4f3ab7c Compare October 15, 2017 12:01

rgw: Add dynamic resharding documentation

4f3ab7c

Fixes: http://tracker.ceph.com/issues/21553 Signed-off-by: Orit Wasserman <owasserm@redhat.com>

theanalyst merged commit 2a9cbc3 into ceph:master Oct 16, 2017


		.. versionadded:: Luminous

		The dynamic resharding feature automatically increase the number of bucket index shards.


		.. versionadded:: Luminous

		A large bucket index can lead to performance problems, in order to address this problem we introduced bucket index sharding.


		-``rgw_reshard_bucket_lock_duration``: duration of lock on bucket obj during resharding, default: 120 seconds.

		-``rgw_max_objs_per_shard``: Maximal number of objects per bucket index shard, default: 100000 objects.


		Parameters to control the resahrding process in Ceph configuration fie:

		-``rgw_reshard_num_logs``: Number of shards for the resharding log, default: 16



		Manual bucket resharding
		------------------------------------



		Admin commands
		===============

rgw: Add dynamic resharding documentation #15941

rgw: Add dynamic resharding documentation #15941

Conversation

oritwas commented Jun 27, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

theanalyst commented Aug 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viduship Aug 18, 2017 • edited Loading

Choose a reason for hiding this comment

mattbenjamin commented Aug 18, 2017

alfredodeza commented Aug 19, 2017

mattbenjamin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oritwas commented Sep 27, 2017

yehudasa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oritwas commented Oct 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oritwas commented Jun 27, 2017 •

edited

Loading

viduship Aug 18, 2017 •

edited

Loading