rgw: Admin API Support for bucket quota change #18324

Jeegn-Chen · 2017-10-16T10:22:00Z

Allow admin to change the quota of a individual bucket through Admin API

Fixes: http://tracker.ceph.com/issues/21811
Signed-off-by: Jeegn Chen jeegnchen@gmail.com

Jeegn-Chen · 2017-10-16T10:27:56Z

Here are the examples of API calls

host=127.0.0.1:8000
resource="/admin/bucket"
param="?format=json&bucket=test-a&quota&uid=testid"
contentType="application/json"
dateValue=$(date -R -u)
stringToSign="PUT

${contentType}
${dateValue}
${resource}"
s3Key=ABCDEFGHIJKLMNOPQRST
s3Secret='abcdefghijklmnopqrstuvwxyzabcdefghijklmn'
signature=$(/bin/echo -n "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64)
curl -v -X PUT \
  -H "Date: ${dateValue}" \
  -H "Content-Type: ${contentType}" \
  -H "Authorization: AWS ${s3Key}:${signature}" \
  http://${host}${resource}${param} -d '{
        "enabled": true,
        "max_objects": 1,
        "max_size_kb": 5
    }'

host=127.0.0.1:8000
resource="/admin/bucket"
param="?format=json&bucket=test-a&quota&uid=testid"
contentType="application/json"
dateValue=$(date -R -u)
stringToSign="PUT

${contentType}
${dateValue}
${resource}"
s3Key=ABCDEFGHIJKLMNOPQRST
s3Secret='abcdefghijklmnopqrstuvwxyzabcdefghijklmn'
signature=$(/bin/echo -n "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64)
curl -v -X PUT \
  -H "Date: ${dateValue}" \
  -H "Content-Type: ${contentType}" \
  -H "Authorization: AWS ${s3Key}:${signature}" \
  "http://${host}${resource}${param}&enabled=false"

host=127.0.0.1:8000
resource="/admin/bucket"
param="?format=json&bucket=test-a&quota&uid=testid"
contentType="application/json"
dateValue=$(date -R -u)
stringToSign="PUT

${contentType}
${dateValue}
${resource}"
s3Key=ABCDEFGHIJKLMNOPQRST
s3Secret='abcdefghijklmnopqrstuvwxyzabcdefghijklmn'
signature=$(/bin/echo -n "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64)
curl -v -X PUT \
  -H "Date: ${dateValue}" \
  -H "Content-Type: ${contentType}" \
  -H "Authorization: AWS ${s3Key}:${signature}" \
  "http://${host}${resource}${param}&max-objects=2"

host=127.0.0.1:8000
resource="/admin/bucket"
param="?format=json&bucket=test-a&quota&uid=testid"
contentType="application/json"
dateValue=$(date -R -u)
stringToSign="PUT

${contentType}
${dateValue}
${resource}"
s3Key=ABCDEFGHIJKLMNOPQRST
s3Secret='abcdefghijklmnopqrstuvwxyzabcdefghijklmn'
signature=$(/bin/echo -n "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64)
curl -v -X PUT \
  -H "Date: ${dateValue}" \
  -H "Content-Type: ${contentType}" \
  -H "Authorization: AWS ${s3Key}:${signature}" \
  "http://${host}${resource}${param}&max-size-kb=10"

host=127.0.0.1:8000
resource="/admin/bucket"
param="?format=json&bucket=test-a&quota&uid=testid"
contentType="application/json"
dateValue=$(date -R -u)
stringToSign="PUT

${contentType}
${dateValue}
${resource}"
s3Key=ABCDEFGHIJKLMNOPQRST
s3Secret='abcdefghijklmnopqrstuvwxyzabcdefghijklmn'
signature=$(/bin/echo -n "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64)
curl -v -X PUT \
  -H "Date: ${dateValue}" \
  -H "Content-Type: ${contentType}" \
  -H "Authorization: AWS ${s3Key}:${signature}" \
  "http://${host}${resource}${param}&max-size-kb=100&max-objects=101&enabled=true"

Jeegn-Chen · 2017-12-04T09:43:34Z

@yuriw Could you take a look at this PR?

yuriw · 2017-12-12T16:04:31Z

wip-yuri-testing-2017-12-12-1602

yuriw · 2017-12-12T16:04:35Z

wip-yuri-testing-2017-12-12-1602

yuriw · 2017-12-12T16:04:38Z

wip-yuri-testing-2017-12-12-1602

yuriw · 2017-12-12T16:04:39Z

wip-yuri-testing-2017-12-12-1602

yuriw · 2017-12-12T16:04:39Z

wip-yuri-testing-2017-12-12-1602

yuriw · 2017-12-12T16:04:41Z

wip-yuri-testing-2017-12-12-1602

yuriw · 2017-12-12T16:04:42Z

wip-yuri-testing-2017-12-12-1602

yuriw · 2017-12-12T16:05:06Z

wip-yuri-testing-2017-12-12-1602

yuriw · 2017-12-12T16:05:07Z

wip-yuri-testing-2017-12-12-1602

yuriw · 2017-12-12T16:05:09Z

wip-yuri-testing-2017-12-12-1602

yuriw · 2017-12-12T16:21:22Z

wip-yuri-testing-2017-12-12-1602

cbodley · 2017-12-12T17:59:44Z

src/rgw/rgw_bucket.cc

+  }
+
+  bucket_info.quota = op_state.quota;
+  r = store->put_bucket_instance_info(bucket_info, false, real_time(), &attrs);


@adamemerson do you think this would be a good candidate for retry_raced_bucket_write()?

Thanks for the comment. I will take a look

cbodley · 2017-12-15T17:12:52Z

@Jeegn-Chen i like the RGWBucketAdminOp::set_quota() stuff you added, so I'd like to find a way to keep using it along with retry_raced_bucket_write(). it looks like its req_state argument makes that difficult though. we might need to change its signature to something more general, like:

int retry_raced_bucket_write(RGWRados* g, RGWBucketInfo& info,
                             std::map<std::string, bufferlist> *pattrs, const F& f)

i'd also like to find a way to share more of this quota admin code between the rest api and the radosgw-admin quota commands, but that's probably out of scope for this pr

cbodley · 2017-12-15T17:14:47Z

src/rgw/rgw_op.h

+  }
+  return r;
+}
+}


since this helper is specific to buckets, i think rgw_bucket.h is a better home for it. also, since it's being moved into a header file, it shouldn't be wrapped in an anonymous namespace - so either remove that, or use namespace rgw instead

p.s. the PS comment isn't necessary! the git history is enough of a paper trail

@cbodley
So maybe it is better for me to remove the commit 'rgw: Leverage retry_raced_bucket_write when updating quota' from this PR and let's address retry_raced_bucket_write signature improvement in an independent PR for better commit isolation. What do you think?

works for me 👍

requesting changes

Jeegn-Chen · 2017-12-18T13:37:37Z

Per discuss with @cbodley, remove the commit related to retry_raced_bucket_write, which is out of the scope of this PR and will be addressed in a separate one.

yuriw · 2017-12-18T16:49:15Z

ready for merge

cbodley · 2018-01-02T15:35:06Z

@Jeegn-Chen would you be willing to add a bit of documentation to doc/radosgw/adminops.rst for this new api?

Jeegn-Chen · 2018-01-04T06:45:42Z

@cbodley Sure

Allow admin to change the quota of a individual bucket through Admin API Fixes: http://tracker.ceph.com/issues/21811 Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

Allow admin to change the quota of a individual bucket through Admin API with params in URL Fixes: http://tracker.ceph.com/issues/21811 Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

Jeegn-Chen · 2018-01-04T08:01:11Z

@cbodley I've just updated adminops.rst

cbodley · 2018-01-04T15:42:34Z

thanks @Jeegn-Chen! the docs build has a minor complaint:

Warning, treated as error:
/home/jenkins-build/build/workspace/ceph-pr-docs/doc/radosgw/adminops.rst:1917: (WARNING/2) Title underline too short.

Set Quota for an Individual Bucket
~~~~~~~~~~~~~~~

Add usage doc for a new Admin API for individual bucket quota management Fixes: http://tracker.ceph.com/issues/21811 Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

Jeegn-Chen · 2018-01-05T03:17:17Z

@cbodley Sorry for not double checking the doc build. The issue should be fixed now.

hrchu · 2019-06-04T02:23:52Z

@cbodley

It's weird that:

The official document shows that the feature is available since luminous:
http://docs.ceph.com/docs/luminous/radosgw/adminops/#set-quota-for-an-individual-bucket
Github shows that #b99dae0 is available since v13.0.2 (mimic)

cbodley · 2019-06-04T11:52:02Z

hi @hrchu, this work was backported to luminous in #20885

dang previously approved these changes Oct 16, 2017

View reviewed changes

liewegas added rgw needs-qa labels Nov 16, 2017

yuriw added the wip-yuri-testing label Dec 12, 2017

cbodley reviewed Dec 12, 2017

View reviewed changes

Jeegn-Chen force-pushed the wip-bucket-quota-api branch from d6bd7dd to b88c6b1 Compare December 15, 2017 04:32

cbodley reviewed Dec 15, 2017

View reviewed changes

Jeegn-Chen force-pushed the wip-bucket-quota-api branch from 7e7adcf to b88c6b1 Compare December 15, 2017 17:41

cbodley approved these changes Jan 2, 2018

View reviewed changes

cbodley added needs-doc and removed needs-qa wip-yuri-testing labels Jan 2, 2018

Jeegn-Chen added 2 commits January 4, 2018 06:56

rgw: Admin API Support for bucket quota change

b2893d1

Allow admin to change the quota of a individual bucket through Admin API Fixes: http://tracker.ceph.com/issues/21811 Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

rgw: Add URL format support for bucket quota modification

46b1186

Allow admin to change the quota of a individual bucket through Admin API with params in URL Fixes: http://tracker.ceph.com/issues/21811 Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

Jeegn-Chen force-pushed the wip-bucket-quota-api branch from b88c6b1 to df5e25d Compare January 4, 2018 07:59

doc: Add usage doc for individual bucket quota Admin API

0cbd26e

Add usage doc for a new Admin API for individual bucket quota management Fixes: http://tracker.ceph.com/issues/21811 Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

Jeegn-Chen force-pushed the wip-bucket-quota-api branch from df5e25d to 0cbd26e Compare January 5, 2018 01:49

cbodley merged commit b99dae0 into ceph:master Jan 5, 2018

hrchu mentioned this pull request Mar 29, 2018

How to save the BucketInfo? twonote/radosgw-admin4j#18

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rgw: Admin API Support for bucket quota change #18324

rgw: Admin API Support for bucket quota change #18324

Jeegn-Chen commented Oct 16, 2017

Jeegn-Chen commented Oct 16, 2017

Jeegn-Chen commented Dec 4, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

cbodley Dec 12, 2017

Jeegn-Chen Dec 14, 2017

cbodley commented Dec 15, 2017

cbodley Dec 15, 2017 •

edited

Jeegn-Chen Dec 15, 2017

cbodley Dec 15, 2017

Jeegn-Chen commented Dec 18, 2017

yuriw commented Dec 18, 2017

cbodley commented Jan 2, 2018

Jeegn-Chen commented Jan 4, 2018

Jeegn-Chen commented Jan 4, 2018

cbodley commented Jan 4, 2018

Jeegn-Chen commented Jan 5, 2018

hrchu commented Jun 4, 2019 •

edited

cbodley commented Jun 4, 2019

rgw: Admin API Support for bucket quota change #18324

rgw: Admin API Support for bucket quota change #18324

Conversation

Jeegn-Chen commented Oct 16, 2017

Jeegn-Chen commented Oct 16, 2017

Jeegn-Chen commented Dec 4, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

yuriw commented Dec 12, 2017

cbodley Dec 12, 2017

Choose a reason for hiding this comment

Jeegn-Chen Dec 14, 2017

Choose a reason for hiding this comment

cbodley commented Dec 15, 2017

cbodley Dec 15, 2017 • edited

Choose a reason for hiding this comment

Jeegn-Chen Dec 15, 2017

Choose a reason for hiding this comment

cbodley Dec 15, 2017

Choose a reason for hiding this comment

Jeegn-Chen commented Dec 18, 2017

yuriw commented Dec 18, 2017

cbodley commented Jan 2, 2018

Jeegn-Chen commented Jan 4, 2018

Jeegn-Chen commented Jan 4, 2018

cbodley commented Jan 4, 2018

Jeegn-Chen commented Jan 5, 2018

hrchu commented Jun 4, 2019 • edited

cbodley commented Jun 4, 2019

cbodley Dec 15, 2017 •

edited

hrchu commented Jun 4, 2019 •

edited