rgw: Multiple Data Pool Support for a Bucket #19890

Jeegn-Chen · 2018-01-10T04:01:04Z

http://tracker.ceph.com/issues/22565
https://etherpad.net/p/multiple-data-pool-support-for-a-bucket

Jeegn-Chen · 2018-01-10T04:10:15Z

@yehudasa @mattbenjamin @robbat2 @fangyuxiangGL
Discussion on https://etherpad.net/p/multiple-data-pool-support-for-a-bucket through ceph-devel mailing list may be somewhat abstract.
I worked out an initial implementation for atomic upload (multi-part upload and obj copy are to be addressed).
Could you take a look and comment?

ZVampirEM77 · 2018-01-11T02:45:43Z

@Jeegn-Chen /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/radosgw-admin/help.t: failed

ZVampirEM77 · 2018-01-11T02:48:44Z

@Jeegn-Chen the command description should also be added into src/test/cli/radosgw-admin/help.t

fangyuxiangGL · 2018-01-11T10:45:24Z

src/rgw/rgw_rados.cc

@@ -10608,6 +10674,9 @@ int RGWRados::get_obj_iterate_cb(RGWObjectCtx *ctx, RGWObjState *astate,
      if (!len)
 	  return 0;
    }
+    effective_io_ctx.dup(d->io_ctx);
+  }else {
+    effective_io_ctx.dup(d->tail_io_ctx);


@Jeegn-Chen

dup here is dangerous for you trigger aio_operate, effective_io_ctx is on the stack, its private io_ctx_impl will be released when function return. You can use effective_io_ctx = d->tail_io_ctx;, which share io_ctx_impl with d->tail_io_ctx

nice catch. thx

mattbenjamin

updated

mattbenjamin · 2018-01-11T14:45:34Z

src/rgw/rgw_common.h

+inline ostream& operator<<(ostream& out, const RGWBucketDataLayoutType &data_layout_type) 
+{
+  switch (data_layout_type) {
+    case RGWDLType_Normal:


I don't think "normal" and "splitted" are the right names for these cases, if only because for the latter you'd write "split." Maybe single-pool and split-pool? There still hasn't been organized discussion in RGW standups, that would be helpful.

mattbenjamin · 2018-01-11T14:48:26Z

src/rgw/rgw_common.h

@@ -2111,6 +2129,39 @@ struct rgw_cache_entry_info {
  rgw_cache_entry_info() : gen(0) {}
 };

+struct rgw_data_placement_volatile_config {


This isn't a typical RGW name for a class/struct type. Could we rename this? More concretely, what makes it volatile, and how does that relate to similar info in RGW?

To tell the truth, I'm also worry about the proper name for this struct. I just use it for now and prefer to change if a proper one is found.

“volatile” here means unlike index_pool, data_pool and data_extra_pool of a bucket, each object in a bucket may have different rgw_data_placement_volatile_config.
And if we change current_tail_pools and upload the same object again, rgw_data_placement_volatile_config of the same object will be changed.

Any recommendation?

mattbenjamin · 2018-01-11T14:49:55Z

src/rgw/rgw_common.h

+  }
+
+  bool empty() const {
+    return data_layout_type == RGWDLType_Normal;


would put boolean expressions in parenthesis, but then, I like style(9)

fangyuxiangGL · 2018-01-17T04:05:55Z

@Jeegn-Chen

I remembered one issue while I did #16325, which was related to multipart upload but I'm not sure if it exist here, you'd better verify it.

Given that we upload an object using multipart, and for one part of it, there are 3 rados objects, which we named as: multipart-object, shadow-1, shadow-2. At last, in RGWPutObjProcessor_Multipart::do_complete, please pay attention to codes:

  RGWRados::Object op_target(store, s->bucket_info, obj_ctx, head_obj);
  op_target.set_versioning_disabled(true);
  RGWRados::Object::Write head_obj_op(&op_target);

  head_obj_op.meta.set_mtime = set_mtime;
  head_obj_op.meta.mtime = mtime;
  head_obj_op.meta.owner = s->owner.get_id();
  head_obj_op.meta.delete_at = delete_at;
  head_obj_op.meta.zones_trace = zones_trace;
  head_obj_op.meta.modify_tail = true;

  int r = head_obj_op.write_meta(obj_len, accounted_size, attrs);
  if (r < 0)
    return r;

we must make sure that write_meta is called at the proper rados object, which should just be the multipart-object mentioned above.

fangyuxiangGL · 2018-01-17T04:09:25Z

ok, seems you have introduced head_obj_op.meta.track_multipart to avoid it!

Fixes: http://tracker.ceph.com/issues/22565 Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

fangyuxiangGL · 2018-06-20T01:31:12Z

src/rgw/rgw_admin.cc

+          if (current_tail_pool) {
+            info.current_tail_pool = *current_tail_pool;
+          }
+          if (tail_pools) {


@Jeegn-Chen
Seems we have no way to remove anything from data_tail_pools.
how about call info.data_tail_pools.clear(); here?

stale · 2018-10-17T23:54:13Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
If you are a maintainer or core committer, please follow-up on this issue to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

stale · 2019-04-22T20:34:31Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

cugwangyang · 2021-02-24T06:18:57Z

The following URL can not be connected!Why was this request closed?Thank you
https://etherpad.net/p/multiple-data-pool-support-for-a-bucket

cugwangyang · 2021-02-24T06:21:28Z

http://tracker.ceph.com/issues/22565
https://etherpad.net/p/multiple-data-pool-support-for-a-bucket

Why was this request closed?There are big bugs in this request? Thank you!

tchaikov · 2021-02-24T07:03:12Z

@cugwangyang it was closed by the bot. i am reopening it in hope it can have more attention from rgw maintainers.

github-actions · 2021-02-24T07:03:44Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

cugwangyang · 2021-02-24T10:03:42Z

This URL can not be connected. Can you give me another URL to share your mind?
https://etherpad.net/p/multiple-data-pool-support-for-a-bucket

stale · 2021-07-21T05:00:37Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2022-07-07T02:16:40Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

liewegas changed the title ~~[WIP] Multiple Data Pool Support for a Bucket~~ rgw: [WIP] Multiple Data Pool Support for a Bucket Jan 10, 2018

liewegas added feature rgw labels Jan 10, 2018

fangyuxiangGL reviewed Jan 11, 2018

View reviewed changes

Jeegn-Chen force-pushed the wip-bucket-multi-pool branch from 2249295 to 3a1ffdb Compare January 11, 2018 14:13

mattbenjamin reviewed Jan 11, 2018

View reviewed changes

Jeegn-Chen force-pushed the wip-bucket-multi-pool branch 3 times, most recently from d761802 to 9a263a1 Compare January 12, 2018 04:12

Jeegn-Chen changed the title ~~rgw: [WIP] Multiple Data Pool Support for a Bucket~~ rgw: Multiple Data Pool Support for a Bucket Jan 16, 2018

Jeegn-Chen added 8 commits February 23, 2018 10:38

rgw: initial multi-pool support for atomic upload

6567f0e

Fixes: http://tracker.ceph.com/issues/22565 Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

rgw: encode/decode for rgw_data_placement_volatile_config

1fd042d

Fixes: http://tracker.ceph.com/issues/22565 Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

rgw: initial multi-pool support for multipart upload

df3edd1

Fixes: http://tracker.ceph.com/issues/22565 Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

rgw: initial multi-pool support for copy obj

f19da69

Fixes: http://tracker.ceph.com/issues/22565 Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

rgw: write meta to correct pool when completing Multipart PUT

556dfcc

Fixes: http://tracker.ceph.com/issues/22565 Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

rgw: check tail pool in get_max_chunk_size

9c6afea

Fixes: http://tracker.ceph.com/issues/22565 Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

rgw: initial multi-pool support for uploads with POST

e84a4bb

Fixes: http://tracker.ceph.com/issues/22565 Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

rgw: initial support for max_head_size config

747f1be

Fixes: http://tracker.ceph.com/issues/22565 Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

Jeegn-Chen force-pushed the wip-bucket-multi-pool branch from 28f0d54 to 747f1be Compare February 23, 2018 03:15

fangyuxiangGL reviewed Jun 20, 2018

View reviewed changes

stale bot added the stale label Oct 17, 2018

stale bot closed this Apr 22, 2019

tchaikov reopened this Feb 24, 2021

stale bot removed the stale label Feb 24, 2021

github-actions bot added the needs-rebase label Feb 24, 2021

github-actions bot added the tests label Feb 24, 2021

stale bot added the stale label Jul 21, 2021

github-actions bot closed this Jul 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rgw: Multiple Data Pool Support for a Bucket #19890

rgw: Multiple Data Pool Support for a Bucket #19890

Jeegn-Chen commented Jan 10, 2018

Jeegn-Chen commented Jan 10, 2018

ZVampirEM77 commented Jan 11, 2018

ZVampirEM77 commented Jan 11, 2018

fangyuxiangGL Jan 11, 2018

Jeegn-Chen Jan 11, 2018

mattbenjamin left a comment

mattbenjamin Jan 11, 2018

mattbenjamin Jan 11, 2018

Jeegn-Chen Jan 11, 2018

mattbenjamin Jan 11, 2018

fangyuxiangGL commented Jan 17, 2018 •

edited

fangyuxiangGL commented Jan 17, 2018

fangyuxiangGL Jun 20, 2018

stale bot commented Oct 17, 2018

stale bot commented Apr 22, 2019

cugwangyang commented Feb 24, 2021

cugwangyang commented Feb 24, 2021

tchaikov commented Feb 24, 2021

github-actions bot commented Feb 24, 2021

cugwangyang commented Feb 24, 2021

stale bot commented Jul 21, 2021

github-actions bot commented Jul 7, 2022

rgw: Multiple Data Pool Support for a Bucket #19890

rgw: Multiple Data Pool Support for a Bucket #19890

Conversation

Jeegn-Chen commented Jan 10, 2018

Jeegn-Chen commented Jan 10, 2018

ZVampirEM77 commented Jan 11, 2018

ZVampirEM77 commented Jan 11, 2018

fangyuxiangGL Jan 11, 2018

Choose a reason for hiding this comment

Jeegn-Chen Jan 11, 2018

Choose a reason for hiding this comment

mattbenjamin left a comment

Choose a reason for hiding this comment

mattbenjamin Jan 11, 2018

Choose a reason for hiding this comment

mattbenjamin Jan 11, 2018

Choose a reason for hiding this comment

Jeegn-Chen Jan 11, 2018

Choose a reason for hiding this comment

mattbenjamin Jan 11, 2018

Choose a reason for hiding this comment

fangyuxiangGL commented Jan 17, 2018 • edited

fangyuxiangGL commented Jan 17, 2018

fangyuxiangGL Jun 20, 2018

Choose a reason for hiding this comment

stale bot commented Oct 17, 2018

stale bot commented Apr 22, 2019

cugwangyang commented Feb 24, 2021

cugwangyang commented Feb 24, 2021

tchaikov commented Feb 24, 2021

github-actions bot commented Feb 24, 2021

cugwangyang commented Feb 24, 2021

stale bot commented Jul 21, 2021

github-actions bot commented Jul 7, 2022

fangyuxiangGL commented Jan 17, 2018 •

edited