Skip to content

Bug 1851311: Set the minimum rgw PGs for the metadata pools to the smaller default - #73

Merged
travisn merged 4 commits into
red-hat-storage:release-4.4from
travisn:backport-rgw-meta-pgs
Jun 26, 2020
Merged

Bug 1851311: Set the minimum rgw PGs for the metadata pools to the smaller default#73
travisn merged 4 commits into
red-hat-storage:release-4.4from
travisn:backport-rgw-meta-pgs

Conversation

@travisn

@travisn travisn commented Jun 26, 2020

Copy link
Copy Markdown

Description of your changes:
The object store was not getting created because of the PG limit. The rook operator log shows:

2020-06-26 10:30:39.906986 I | exec: Running command: ceph osd pool create ocs-storagecluster-cephobjectstore.rgw.buckets.index 0 replicated ocs-storagecluster-cephobjectstore.rgw.buckets.index --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json --out-file /tmp/875991478
2020-06-26 10:30:40.952434 I | exec: Error ERANGE: pg_num 32 size 3 would mean 768 total pgs, which exceeds max 750 (mon_max_pg_per_osd 250 * num_in_osds 3)
2020-06-26 10:30:40.952568 E | op-object: failed to create or update object store ocs-storagecluster-cephobjectstore. failed to create pools: failed to create object pools: failed to create metadata pools: failed to create pool ocs-storagecluster-cephobjectstore.rgw.buckets.index for object store ocs-storagecluster-cephobjectstore

This is due to a change in Ceph 14.2.8 when the default PG count was set for all pools to 32. The expectation is that the rgw metadata pools should only set 8 PGs by default, which will keep us well below the max PG count. However, these changes had not been backported to 4.4. We are just now hitting it since the base image is 14.2.8 or newer.

This backport includes three PRs:
rook#5096
rook#5177
rook#5489
Which issue is resolved by this Pull Request:
Resolves #https://bugzilla.redhat.com/show_bug.cgi?id=1851311

Checklist:

  • Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
  • Skip Tests for Docs: Add the flag for skipping the build if this is only a documentation change. See here for the flag.
  • Skip Unrelated Tests: Add a flag to run tests for a specific storage provider. See test options.
  • Reviewed the developer guide on Submitting a Pull Request
  • Documentation has been updated, if necessary.
  • Unit tests have been added, if necessary.
  • Integration tests have been added, if necessary.
  • Pending release notes updated with breaking and/or notable changes, if necessary.
  • Upgrade from previous release is tested and upgrade user guide is updated, if necessary.
  • Code generation (make codegen) has been run to update object specifications, if necessary.

travisn and others added 3 commits June 26, 2020 11:58
@travisn
ceph: set pg_num for rgw metadata pools to lower default
dba0522 
The PG count on metadata pools should default to rgw_rados_pool_pg_num_min
instead of the more general default pg count. This means rgw pools
will default to 8 PGs instead of 32 PGs, which means a lot more pools
can be created before hitting the default PG limit.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit ebb075f)
n.fraison @travisn
ceph: set pg_num_min for rgw metadata pools
b972351 
Since e396379 rook create rgw metadata pools with 8 PGs instead of
32. But the pg_autoscaler increase it back to 32 as the pg_num_min parameter
is not set on those pools.

Signed-off-by: n.fraison <n.fraison@criteo.com>
(cherry picked from commit 2757973)
@travisn
ceph: set pg_num_min on new rgw metadata pools
c72beaf 
pg_num_min needs to be set on all new and existing rgw
metadata pools so the autoscaler won't immediately
increase the pg count. The pg_num_min was only being set
on existing pools and not on new pools.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 6d13086)
(cherry picked from commit 62d3039)
@openshift-ci-robot openshift-ci-robot added the bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. label Jun 26, 2020
@openshift-ci-robot

openshift-ci-robot commented Jun 26, 2020

Copy link
Copy Markdown

@travisn: This pull request references Bugzilla bug 1851311, which is invalid:

  • expected the bug to target the "4.4.z" release, but it targets "---" instead
  • expected Bugzilla bug 1851311 to depend on a bug targeting a release in 4.5.0, 4.5.z and in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1851311: Set the minimum rgw PGs for the metadata pools to the smaller default

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jun 26, 2020
n.fraison @travisn
ceph: remove leading and trailing space from get config
ca963b5 
Value returned by ceph config get can contains some trailing space which leads to
issues when running other ceph command.
For ex.:
2020-04-06 15:32:43.543050 I | exec: Running command: ceph osd pool set test-1.rgw.control pg_num_min 8
 --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/249563089
2020-04-06 15:32:44.005018 I | exec: Error EINVAL: error parsing int value '8

Signed-off-by: n.fraison <n.fraison@criteo.com>
(cherry picked from commit ddadfbc)
@travisn
travisn merged commit a50227d into red-hat-storage:release-4.4 Jun 26, 2020
@openshift-ci-robot

openshift-ci-robot commented Jun 26, 2020

Copy link
Copy Markdown

@travisn: All pull requests linked via external trackers have merged: . Bugzilla bug 1851311 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1851311: Set the minimum rgw PGs for the metadata pools to the smaller default

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@travisn
travisn deleted the backport-rgw-meta-pgs branch June 26, 2020 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@travisn @openshift-ci-robot