New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ceph: add options for cephobjectstore user #8211
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It will be great to have these settings on the object users! Maybe you're already planning on it, but please do add:
- Documentation updates to ceph-object-store-user-crd.md
- Example settings in object-user.yaml, perhaps commented out
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a couple small suggestions...
1a639d7
to
b82b353
Compare
|
The code changes look good and the new integration test will be very nice, thanks. The upgrade and smoke suites just seem to be failing if you could take a look. For example, this failure was seeing from the upgrade suite: |
@travisn : I hope above failures |
|
The smoke suite is passing. Now the upgrade test is just failing. Since the settings are not applied before the upgrade, looks like we just need to check for different values in that case. |
Done. PTAL @travisn |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great to see the tests passing, just one more suggestion...
| if user.Spec.Capabilities != nil { | ||
| if user.Spec.Capabilities.User != "" { | ||
| userConfig.UserCaps += fmt.Sprintf("users=%s;", user.Spec.Capabilities.User) | ||
| } | ||
| if user.Spec.Capabilities.Bucket != "" { | ||
| userConfig.UserCaps += fmt.Sprintf("buckets=%s;", user.Spec.Capabilities.Bucket) | ||
| } | ||
| if user.Spec.Capabilities.MetaData != "" { | ||
| userConfig.UserCaps += fmt.Sprintf("metadata=%s;", user.Spec.Capabilities.MetaData) | ||
| } | ||
| if user.Spec.Capabilities.Usage != "" { | ||
| userConfig.UserCaps += fmt.Sprintf("usage=%s;", user.Spec.Capabilities.Usage) | ||
| } | ||
| if user.Spec.Capabilities.Zone != "" { | ||
| userConfig.UserCaps += fmt.Sprintf("zone=%s;", user.Spec.Capabilities.Zone) | ||
| } | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Similarly to my question about removing quotas, does this allow caps to be revoked? What if a CephObjectStoreUser needs to have some permissions revoked after being given permissions (like the admin made an initial mistake)? Should we always set capabilities even if user.Spec.Capabilities is nil so that permissions can be easily revoked?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@BlaineEXE : Apparently, it looks like either bug in go-ceph or rgw(most likely) or updating caps is not working atm. It can be performed only while CreateUser API() not with ModifyUser(), even though the request is filled with caps details. I have mentioned the same in ceph-object-user-crd.md
| if user.Spec.Quotas != nil && user.Spec.Quotas.MaxBuckets != 0 { | ||
| userConfig.MaxBuckets = &user.Spec.Quotas.MaxBuckets | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
admin.User has a UserQuota spec inside of it which is the same thing set in the above code. Why are we setting bucket quotas at a different point in the code from the rest of the quotas? Is it not better to just set everything all at once upon user creation? (i.e., move the other quota code here)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's how RGW adminOps apis works, maxBucket is not part of quota Spec. And we can set quota(I mean object and size) only after successful user creation, not during the CreateUser(). Hence I moved to setquota() after the success of user creation. So I cannot move the quota code here. The setting MaxBuckets can be moved to another quota code part, but it requires another additional ModifyUser() api call. So I kept the MaxBuckets during the creation of User and other quotas after user creation.
f0755e0
to
73d332b
Compare
1e6a35e
to
e6db4ef
Compare
| // Maximum bucket limit for the ceph user | ||
| // +optional | ||
| // +nullable | ||
| MaxBuckets *int `json:"maxBuckets,omitempty"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
set MaxBuckets to *int and MaxObjects into int64* only based on admin.User Struct form go ceph
e6db4ef
to
ba0c756
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your patience and paying close attention to this.
It's really unfortunate that capabilities can't be modified or removed once they are set due to a Ceph (or go-ceph) bug. Thanks for noting and documenting that.
From a feature perspective, I'm trying to understand what are the necessary steps to take after this PR merges. In my opinion, modifying and removing caps is required for us to call this feature complete. Users expect that of a declarative configuration system like Rook. Some questions to help me:
- Is this for a BZ feature request? If so, we should make sure we have deletion done before that BZ is closed, or we should document the workaround with the docs team
- Is there a tracker in Ceph to fix the issue from the Ceph side so we can follow up later with update/removal features in Rook?
I completely agree.
Quick look didn't give any bugs, I will try to debug further to know whether issue with go-ceph or on rgw. At least on |
ba0c756
to
cea9566
Compare
Create https://tracker.ceph.com/issues/52521 in RGW to track this issue. Currently, RGW code does not handle the |
Create https://tracker.ceph.com/issues/52521 in RGW to track this issue. Currently, RGW code does not handle the
We can use different approach here as well, instead of ModifyUser() API. We can have apis adding/removing for Caps() similar to quota, but it does not exist in go-ceph or not mentioned in RGW upstream adminOps docs. But the supported code exists in ceph-rgw |
|
This pull request has merge conflicts that must be resolved before it can be merged. @thotz please rebase it. https://rook.io/docs/rook/master/development-flow.html#updating-your-fork |
Adding options for quota, bucket limit, caps for the `cephobjectstoreuser`. Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
cea9566
to
ca43800
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am approving this PR with some nit comments to update docs and comments. Feel free to update the nits and merge the PR afterwards, or you can create a new PR to address the nits if you prefer.
| @@ -281,13 +281,44 @@ func (r *ReconcileObjectStoreUser) createorUpdateCephUser(u *cephv1.CephObjectSt | |||
| } else { | |||
| return errors.Wrapf(err, "failed to get details from ceph object user %q", u.Name) | |||
| } | |||
| } else if *user.MaxBuckets != *r.userConfig.MaxBuckets { | |||
| // TODO handle update for user capabilities | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit. Let's link to the Ceph issue tracker here so we understand what is blocking Rook currently.
| @@ -149,6 +153,17 @@ func checkCephObjectUser( | |||
| assert.NoError(s.T(), err) | |||
| assert.Equal(s.T(), k8sutil.ReadyStatus, phase) | |||
| } | |||
| if checkQuotaAndCaps { | |||
| // following fields in CephObjectStoreUser CRD doesn't exist before Rook v1.7 | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| // following fields in CephObjectStoreUser CRD doesn't exist before Rook v1.7 | |
| // following fields in CephObjectStoreUser CRD doesn't exist before Rook v1.7.3 |
| * `maxBuckets`: The maximum bucket limit for the user. | ||
| * `maxSize`: Maximum size limit of all objects across all the user's buckets. | ||
| * `maxObjects`: Maximum number of objects across all the user's buckets. | ||
| * `capabilities`: Ceph allows users to be given additional permissions(support added from onwards v1.7.3). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| * `capabilities`: Ceph allows users to be given additional permissions(support added from onwards v1.7.3). | |
| * `capabilities`: Ceph allows users to be given additional permissions (supported in Rook v1.7.3 and up). |
| * `maxSize`: Maximum size limit of all objects across all the user's buckets. | ||
| * `maxObjects`: Maximum number of objects across all the user's buckets. | ||
| * `capabilities`: Ceph allows users to be given additional permissions(support added from onwards v1.7.3). | ||
| P.S this setting can used only during the creation of the object store user, not afterwards. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| P.S this setting can used only during the creation of the object store user, not afterwards. | |
| Due to a bug in Ceph, this setting can currently only be used during the creation of the object store user. If a user's capabilities need modified, the user must be deleted and re-created. |
| @@ -33,3 +40,17 @@ spec: | |||
|
|
|||
| * `store`: The object store in which the user will be created. This matches the name of the objectstore CRD. | |||
| * `displayName`: The display name which will be passed to the `radosgw-admin user create` command. | |||
| * `quotas`: This represents quota limitation can be set on the user(support added from onwards v1.7.3). | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| * `quotas`: This represents quota limitation can be set on the user(support added from onwards v1.7.3). | |
| * `quotas`: Quota limitations to be set on the user (supported in Rook v1.7.3 and up). |
@BlaineEXE : May I address the nits after the merge of this PR as a separate one? |
Addressing remaining nits from the PR rook#8211 Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
ceph: add options for cephobjectstore user (backport #8211)
ceph: addressing nits from #8211
Addressing remaining nits from the PR rook#8211 Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Description of your changes:
Adding options for quota, bucket limit, caps for the
cephobjectstoreuser.Signed-off-by: Jiffin Tony Thottan thottanjiffin@gmail.com
Which issue is resolved by this Pull Request:
Resolves #7906
Checklist:
make codegen) has been run to update object specifications, if necessary.