rgw: Increase the default number of RGW bucket shards #32660

cbodley · 2020-01-15T20:54:10Z

each zone in the zonegroup has a "bucket_index_max_shards" field, but the only way to change it is by manually editing the json format using radosgw-admin zonegroup get/set

this change extends the zone modify and zonegroup modify commands to accept the --max-bucket-index-shards option to set this field - either for an individual zone, or for all zones in a given zonegroup

it also pulls in the change from #30875 to raise its default value to 11

cbodley · 2020-01-29T15:50:13Z

jenkins test make check

cbodley · 2020-02-03T17:12:38Z

@ivancich could you please help review this for octopus?

ivancich

I have some suggestions that are more on the nit level. Please consider, but I won't hold up the approval.

ivancich · 2020-02-03T17:44:58Z

src/rgw/rgw_admin.cc

@@ -350,6 +350,7 @@ void usage()
  cout << "                             set list of zones to sync from\n";
  cout << "   --sync-from-rm=[zone-name][,...]\n";
  cout << "                             remove zones from list of zones to sync from\n";
+  cout << "   --bucket-index-max-shards override a zone/zonegroup's default bucket index shard count\n";


does this need to indicate that there is a number included in the command-line-argument?

i'm torn on this. some options do specify that they require a certain argument, and others don't. in this case, it would feel overly verbose and need a line break. and if there is confusion, the error message is pretty clear:

$ radosgw-admin zone modify --bucket-index-max-shards Option --bucket-index-max-shards requires an argument.

src/rgw/rgw_zone.cc

ivancich · 2020-02-03T17:48:22Z

src/rgw/rgw_zone.h

@@ -568,7 +568,7 @@ struct RGWZone {
  bool sync_from_all;
  set<std::string> sync_from; /* list of zones to sync from */

-  RGWZone() : log_meta(false), log_data(false), read_only(false), bucket_index_max_shards(0),
+  RGWZone() : log_meta(false), log_data(false), read_only(false), bucket_index_max_shards(11),


what about making 11 a constexpr?

good point! turning this into a constexpr variable also allows us to document it

ivancich · 2020-02-03T17:48:59Z

src/test/cli/radosgw-admin/help.t

@@ -245,6 +245,7 @@
                               set list of zones to sync from
     --sync-from-rm=[zone-name][,...]
                               remove zones from list of zones to sync from
+     --bucket-index-max-shards override a zone/zonegroup's default bucket index shard count


again, indicate a value is expected

src/rgw/rgw_zone.h

doc/man/8/radosgw-admin.rst

Signed-off-by: Mark Nelson <mnelson@redhat.com> Modified-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cbodley · 2020-02-27T21:23:06Z

http://pulpito.ceph.com/cbodley-2020-02-27_18:44:45-rgw-wip-cbodley-testing-distro-basic-smithi/

i'm seeing a number of s3test failures around listing sharded buckets:

s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... FAIL
s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... FAIL
s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... FAIL
s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... FAIL
s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... FAIL

i think these are real bugs that we just haven't hit because we always test with num_shards=1

ivancich · 2020-02-27T21:28:43Z

http://pulpito.ceph.com/cbodley-2020-02-27_18:44:45-rgw-wip-cbodley-testing-distro-basic-smithi/

i'm seeing a number of s3test failures around listing sharded buckets:
s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... FAIL
s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... FAIL
s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... FAIL
s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... FAIL
s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... FAIL
i think these are real bugs that we just haven't hit because we always test with num_shards=1

I suspect you're right. This area of code has been touched by a number of recent PRs.

Would you like me to take a look?

cbodley · 2020-02-27T21:34:24Z

@ivancich i'll take a stab first and keep you posted

cbodley · 2020-02-28T18:12:32Z

i pushed a fix to #33628, and i'll test these two prs together

Signed-off-by: Casey Bodley <cbodley@redhat.com>

ivancich · 2020-02-28T19:47:45Z

qa/suites/rgw/verify/sharding$

@@ -0,0 +1 @@
+.qa/rgw_bucket_sharding


cbodley · 2020-03-02T14:08:59Z

jenkins test make check

cbodley · 2020-03-02T14:11:51Z

http://pulpito.ceph.com/cbodley-2020-02-28_20:21:50-rgw-wip-cbodley-testing-distro-basic-smithi/

some of the many-sharded tests died with osd "out of order op" crashes, but those appear to have been resolved on master with https://tracker.ceph.com/issues/42328

the bucket listing tests are all succeeding now with #33628

cbodley added the rgw label Jan 15, 2020

cbodley mentioned this pull request Jan 15, 2020

common/options: Increase the default number of RGW bucket shards #30875

Closed

3 tasks

cbodley requested a review from ivancich January 15, 2020 21:35

cbodley force-pushed the wip-rgw-admin-zone-shards branch 2 times, most recently from 759eac7 to 343fa48 Compare January 28, 2020 21:50

cbodley requested a review from markhpc January 28, 2020 21:53

cbodley force-pushed the wip-rgw-admin-zone-shards branch from 343fa48 to 8fb1302 Compare January 30, 2020 15:50

markhpc added the performance label Jan 30, 2020

cbodley changed the title ~~radosgw-admin: allow zone[group] modify to configure index shards~~ rgw: Increase the default number of RGW bucket shards Jan 30, 2020

ivancich approved these changes Feb 3, 2020

View reviewed changes

cbodley force-pushed the wip-rgw-admin-zone-shards branch 2 times, most recently from 6e80a25 to af65bf0 Compare February 3, 2020 18:49

cbodley added the wip-cbodley-testing label Feb 27, 2020

Mark Nelson and others added 5 commits February 27, 2020 15:12

rgw: Increase the default number of RGW bucket shards

8c6f57d

Signed-off-by: Mark Nelson <mnelson@redhat.com> Modified-by: Casey Bodley <cbodley@redhat.com>

rgw: add a config description for rgw_override_bucket_index_max_shards

fba41d6

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: allow zone[group] modify to configure index shards

f57bf02

Signed-off-by: Casey Bodley <cbodley@redhat.com>

PendingReleaseNotes: default bucket index shard count raised to 11

1ec77ec

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'zonegroup modify' does not require realm

f62e89b

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cbodley force-pushed the wip-rgw-admin-zone-shards branch from af65bf0 to f62e89b Compare February 27, 2020 21:14

cbodley added 2 commits February 28, 2020 14:36

qa/rgw: add different bucket sharding overrides

101f41e

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: verify suite selects a random bucket sharding configuration

08e0a42

Signed-off-by: Casey Bodley <cbodley@redhat.com>

ivancich reviewed Feb 28, 2020

View reviewed changes

qa/suites/rgw/verify/sharding$

@@ -0,0 +1 @@

.qa/rgw_bucket_sharding

Copy link

Member

ivancich Feb 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool!

cbodley merged commit affc79b into ceph:master Mar 2, 2020

cbodley deleted the wip-rgw-admin-zone-shards branch March 2, 2020 14:38

theanalyst mentioned this pull request Mar 19, 2020

doc: update octopus release notes for rgw #34028

Merged

3 tasks

rgw: Increase the default number of RGW bucket shards #32660

rgw: Increase the default number of RGW bucket shards #32660

cbodley commented Jan 15, 2020 •

edited

cbodley commented Jan 29, 2020

cbodley commented Feb 3, 2020

ivancich left a comment

ivancich Feb 3, 2020

cbodley Feb 3, 2020

ivancich Feb 3, 2020

cbodley Feb 3, 2020

ivancich Feb 3, 2020

cbodley commented Feb 27, 2020

ivancich commented Feb 27, 2020

cbodley commented Feb 27, 2020

cbodley commented Feb 28, 2020

ivancich Feb 28, 2020

cbodley commented Mar 2, 2020

cbodley commented Mar 2, 2020

rgw: Increase the default number of RGW bucket shards #32660

rgw: Increase the default number of RGW bucket shards #32660

Conversation

cbodley commented Jan 15, 2020 • edited

cbodley commented Jan 29, 2020

cbodley commented Feb 3, 2020

ivancich left a comment

Choose a reason for hiding this comment

ivancich Feb 3, 2020

Choose a reason for hiding this comment

cbodley Feb 3, 2020

Choose a reason for hiding this comment

ivancich Feb 3, 2020

Choose a reason for hiding this comment

cbodley Feb 3, 2020

Choose a reason for hiding this comment

ivancich Feb 3, 2020

Choose a reason for hiding this comment

cbodley commented Feb 27, 2020

ivancich commented Feb 27, 2020

cbodley commented Feb 27, 2020

cbodley commented Feb 28, 2020

ivancich Feb 28, 2020

Choose a reason for hiding this comment

cbodley commented Mar 2, 2020

cbodley commented Mar 2, 2020

cbodley commented Jan 15, 2020 •

edited