Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rgw: Add dynamic resharding documentation #15941

Merged
merged 1 commit into from Oct 16, 2017

Conversation

@oritwas
Copy link
Contributor

commented Jun 27, 2017

Fixes: http://tracker.ceph.com/issues/21553
Signed-off-by: Orit Wasserman owasserm@redhat.com


.. versionadded:: Luminous

The dynamic resharding feature automatically increase the number of bucket index shards.

This comment has been minimized.

Copy link
@mattbenjamin

mattbenjamin Jun 27, 2017

Contributor

s/increase/increases/

.. versionadded:: Luminous

The dynamic resharding feature automatically increase the number of bucket index shards.
It is a background process that periodically checks all the buckets and detect buckets require resharding.

This comment has been minimized.

Copy link
@mattbenjamin

mattbenjamin Jun 27, 2017

Contributor

s/require/that require/

::
# radosgw-admin bucket reshard process

Cancel bucket resharding

This comment has been minimized.

Copy link
@mattbenjamin

mattbenjamin Jun 27, 2017

Contributor

scheduled? can this cancel a resharding already in progress?

@oritwas oritwas force-pushed the oritwas:wip-rgw-dynamic-resharding-doc branch 2 times, most recently from 50f1872 to e97d742 Jun 27, 2017

@theanalyst

This comment has been minimized.

Copy link
Member

commented Aug 8, 2017

@mattbenjamin are you ok with the changes


The dynamic resharding feature automatically increases the number of bucket index shards.
It is a background process that periodically checks all the buckets and detect buckets that require resharding.
Those bucket are inserted into the resharding queue and will be resharded in the background.

This comment has been minimized.

Copy link
@theanalyst

theanalyst Aug 8, 2017

Member

buckets


.. versionadded:: Luminous

The dynamic resharding feature automatically increases the number of bucket index shards.

This comment has been minimized.

Copy link
@theanalyst

theanalyst Aug 8, 2017

Member

Maybe to be more clear we mention dynamic bucket index resharding, though I'm ok either way

List resharding queue
------------------------------
::
# radosgw-admon bucket reshard list

This comment has been minimized.

Copy link
@theanalyst

This comment has been minimized.

Copy link
@viduship

viduship Aug 18, 2017

list resharding queue by 'radosgw-admin reshard list'

@mattbenjamin

This comment has been minimized.

Copy link
Contributor

commented Aug 18, 2017

@oritwas @theanalyst sparse, but lgtm

@alfredodeza

This comment has been minimized.

Copy link
Contributor

commented Aug 19, 2017

jenkins test docs

@oritwas oritwas force-pushed the oritwas:wip-rgw-dynamic-resharding-doc branch 8 times, most recently from e9612b0 to 052bbdf Sep 26, 2017

@mattbenjamin
Copy link
Contributor

left a comment

I marked a few minor grammar points, but content-wise, this looks ready to merge.

online bucket resharding.

Each bucket index shard can handle efficiently up to a threshold of entries, when exceeding this threshold the system
will encounters performance issues.

This comment has been minimized.

Copy link
@mattbenjamin

mattbenjamin Sep 27, 2017

Contributor

if this threshold is exceeded, the system will encounter


.. versionadded:: Luminous

A large bucket index can lead to performance problems, in order to address this problem we introduced bucket index sharding.

This comment has been minimized.

Copy link
@mattbenjamin

mattbenjamin Sep 27, 2017

Contributor

problems. In order


Each bucket index shard can handle efficiently up to a threshold of entries, when exceeding this threshold the system
will encounters performance issues.
The dynamic resharding feature detects this situation and increase automatically the number of shards used by bucket index,

This comment has been minimized.

Copy link
@mattbenjamin

mattbenjamin Sep 27, 2017

Contributor

the bucket index

Each bucket index shard can handle efficiently up to a threshold of entries, when exceeding this threshold the system
will encounters performance issues.
The dynamic resharding feature detects this situation and increase automatically the number of shards used by bucket index,
resulting in a reduction in number of entries in each bucket index shard. This process is transparent to the user.

This comment has been minimized.

Copy link
@mattbenjamin

mattbenjamin Sep 27, 2017

Contributor

in the

The dynamic resharding feature detects this situation and increase automatically the number of shards used by bucket index,
resulting in a reduction in number of entries in each bucket index shard. This process is transparent to the user.

The detection process consist of two parts:

This comment has been minimized.

Copy link
@mattbenjamin

mattbenjamin Sep 27, 2017

Contributor

consists

This comment has been minimized.

Copy link
@mattbenjamin

mattbenjamin Sep 27, 2017

Contributor

btw, great section

@oritwas oritwas force-pushed the oritwas:wip-rgw-dynamic-resharding-doc branch from 052bbdf to 5bc9e18 Sep 27, 2017

@oritwas

This comment has been minimized.

Copy link
Contributor Author

commented Sep 27, 2017

@mattbenjamin , fixed

@yehudasa
Copy link
Member

left a comment

@oritwas see my comments

.. versionadded:: Luminous

A large bucket index can lead to performance problems. In order to address this problem we introduced bucket index sharding.
Until Luminous changing the number of bucket shards (resharding) required to offline the bucket, from Luminous we support

This comment has been minimized.

Copy link
@yehudasa

yehudasa Sep 27, 2017

Member

s/required to offline the bucket/needed to be done offline

Until Luminous changing the number of bucket shards (resharding) required to offline the bucket, from Luminous we support
online bucket resharding.

Each bucket index shard can handle efficiently up to a threshold of entries, if this threshold is exceeded the system

This comment has been minimized.

Copy link
@yehudasa

yehudasa Sep 27, 2017

Member

Each bucket index shard can handle its entries efficiently up until reaching a certain threshold number of entries. If this threashold is exceeded, the system could encounter performance issues.


Each bucket index shard can handle efficiently up to a threshold of entries, if this threshold is exceeded the system
will encounters performance issues.
The dynamic resharding feature detects this situation and increase automatically the number of shards used by the bucket index,

This comment has been minimized.

Copy link
@yehudasa

yehudasa Sep 27, 2017

Member

and increases

resulting in the reduction of the number of entries in each bucket index shard. This process is transparent to the user.

The detection process consists of two parts:
The first is executed during addition of new objects to the bucket and the second is a background process that periodically scans all the

This comment has been minimized.

Copy link
@yehudasa

yehudasa Sep 27, 2017

Member

The first part happens when new objects are added to the bucket, and the second part is via a background process that periodically scans all the buckets.


The detection process consists of two parts:
The first is executed during addition of new objects to the bucket and the second is a background process that periodically scans all the
buckets to handler older buckets in the system that are not being updated.

This comment has been minimized.

Copy link
@yehudasa

yehudasa Sep 27, 2017

Member

s/to handler .../This is needed in order to deal with existing buckets that are not being updated.

The detection process consists of two parts:
The first is executed during addition of new objects to the bucket and the second is a background process that periodically scans all the
buckets to handler older buckets in the system that are not being updated.
A bucket that requires resharding is added to the reshard_log queue and will be scheduled to be resharding later.

This comment has been minimized.

Copy link
@yehudasa

yehudasa Sep 27, 2017

Member

s/to be resharding/to be resharded

The first is executed during addition of new objects to the bucket and the second is a background process that periodically scans all the
buckets to handler older buckets in the system that are not being updated.
A bucket that requires resharding is added to the reshard_log queue and will be scheduled to be resharding later.
The reshard threads run in the background and executes the scheduled resharding, one at a time.

This comment has been minimized.

Copy link
@yehudasa

yehudasa Sep 27, 2017

Member

s/executes/execute


-``rgw_reshard_bucket_lock_duration``: duration of lock on bucket obj during resharding, default: 120 seconds.

-``rgw_max_objs_per_shard``: Maximal number of objects per bucket index shard, default: 100000 objects.

This comment has been minimized.

Copy link
@yehudasa

yehudasa Sep 27, 2017

Member

s/Maximal/maximum


Parameters to control the resahrding process in Ceph configuration fie:

-``rgw_reshard_num_logs``: Number of shards for the resharding log, default: 16

This comment has been minimized.

Copy link
@yehudasa

yehudasa Sep 27, 2017

Member

s/Number/number

@oritwas oritwas force-pushed the oritwas:wip-rgw-dynamic-resharding-doc branch 6 times, most recently from 7d4115e to 2f9348d Sep 28, 2017


Each bucket index shard can handle its entries efficiently up until
reaching a certain threshold of entries. If this threshold is exceeded the system
could encounters performance issues.

This comment has been minimized.

Copy link
@alfredodeza

alfredodeza Oct 2, 2017

Contributor

s/encounters/encounter/


A large bucket index can lead to performance problems. In order
to address this problem we introduced bucket index sharding.
Until Luminous changing the number of bucket shards (resharding)

This comment has been minimized.

Copy link
@alfredodeza

alfredodeza Oct 2, 2017

Contributor

s/Until Luminous/Until Luminous,/

resulting in the reduction of the number of entries in each bucket index shard.
This process is transparent to the user.

The detection process consists of two parts:

This comment has been minimized.

Copy link
@alfredodeza

alfredodeza Oct 2, 2017

Contributor

This could be automatically enumerated, which would allow you to remove 'first' and 'second':

The detection process consists of:

#. Objects are added to the bucket
#. A background process that periodically scans all the buckets
The detection process consists of two parts:
The first part happens when new objects are added to the bucket,
and the second is via a background process that periodically scans all the buckets.
This is needed in order to deal with exisitng buckets in the system that are not being updated.

This comment has been minimized.

Copy link
@alfredodeza

alfredodeza Oct 2, 2017

Contributor

s/exisiting/existing/

The first part happens when new objects are added to the bucket,
and the second is via a background process that periodically scans all the buckets.
This is needed in order to deal with exisitng buckets in the system that are not being updated.
A bucket that requires resharding is added to the reshard_log queue and will be

This comment has been minimized.

Copy link
@alfredodeza

alfredodeza Oct 2, 2017

Contributor

Seems like 'reshard_log' could use backticks to be rendered in monospaced fonts, like ``reshard_log``



Admin commands
===============

This comment has been minimized.

Copy link
@alfredodeza

alfredodeza Oct 2, 2017

Contributor

this underline needs to match the length of the title

# radosgw-admin reshard add --bucket <bucket_name> --num-shards <new number of shards>

List resharding queue
------------------------------

This comment has been minimized.

Copy link
@alfredodeza

alfredodeza Oct 2, 2017

Contributor

this underline needs to match the length of the sub-title

# radosgw-admon reshard list

Process/Schedule a bucket resharding
-----------------------------------------------------

This comment has been minimized.

Copy link
@alfredodeza

alfredodeza Oct 2, 2017

Contributor

same length issue

# radosgw-admin reshard process

Cancel pending bucket resharding
----------------------------------------------

This comment has been minimized.

Copy link
@alfredodeza

alfredodeza Oct 2, 2017

Contributor

same length issue



Manual bucket resharding
------------------------------------

This comment has been minimized.

Copy link
@alfredodeza

alfredodeza Oct 2, 2017

Contributor

same length issue

@oritwas oritwas force-pushed the oritwas:wip-rgw-dynamic-resharding-doc branch 8 times, most recently from 2d88ae0 to 98d1b29 Oct 8, 2017

@oritwas

This comment has been minimized.

Copy link
Contributor Author

commented Oct 10, 2017

Until Luminous, changing the number of bucket shards (resharding)
needed to be done offline, from Luminous we support
online bucket resharding.

This comment has been minimized.

Copy link
@theanalyst

theanalyst Oct 10, 2017

Member

can you add a note here mentioning not to use this in multisite scenarios

This comment has been minimized.

Copy link
@oritwas

oritwas Oct 10, 2017

Author Contributor

right, will add it

changes addressed

@oritwas oritwas force-pushed the oritwas:wip-rgw-dynamic-resharding-doc branch 2 times, most recently from 71c5abe to 4f3ab7c Oct 15, 2017

rgw: Add dynamic resharding documentation
Fixes: http://tracker.ceph.com/issues/21553
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

@theanalyst theanalyst merged commit 2a9cbc3 into ceph:master Oct 16, 2017

5 checks passed

Docs: build check OK - docs built
Details
Signed-off-by all commits in this PR are signed
Details
Unmodified Submodules submodules for project are unmodified
Details
make check make check succeeded
Details
make check (arm64) make check succeeded
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.