[RGW] Wip rgw compression #11494

aclamk · 2016-10-14T10:25:20Z

Continuation of #10932.
I closed old one by mistake.

cbodley · 2016-10-14T18:16:33Z

unlucky timing on the rebase - gitbuilders, which use WITH_LTTNG=ON, hit some unrelated build failures. i had to rebase the ceph/wip_rgw_compression branch on top of 9983cd3 to fix

cbodley · 2016-10-17T20:40:40Z

the teuthology queue is very sluggish lately, but most of the results are in from the latest run. so far all of the failures are unrelated known issues: valgrind, sync agent (HTTPConnectionPool timeouts and 'object not deleted from destination zone'), and a single s3tests failure caused by http://tracker.ceph.com/issues/17465

cbodley · 2016-10-24T17:27:58Z

there's an interesting new merge conflict from #11567, which fixes a bug with zero-length segments in RGWGetObj::read_user_manifest_part(). this pr replaces that loop with a call to RGWRados::Object::Read::iterate(), so we'll need to make sure that function doesn't have to same issue with empty segments. i'll work with @mdw-at-linuxbox to get that tested

cbodley · 2016-10-25T17:49:58Z

src/rgw/rgw_rados.cc

@@ -7572,7 +7605,8 @@ int RGWRados::copy_obj_data(RGWObjectCtx& obj_ctx,
    }
  }

-  ret = processor.complete(etag, mtime, set_mtime, attrs, delete_at);
+  // XXX: need to copy over compression attr and its orig_size here?
+  ret = processor.complete(ofs, etag, mtime, set_mtime, attrs, delete_at);


@aclamk could you look into this when you get a chance? we're passing the swift copy tests as is, so it may be correct as is. in that case, we should just update the comment

It is OK.

Reasoning:
Here we copy object, including attrs. Copying process is compression agnostic. However, all attrs are preserved, I assume including RGW_ATTR_COMPRESSION. Basing on that object can be read.

(discussed briefly during standup)
i agree that the compression attribute is preserved. i just worry about ofs as the first argument to complete() - my changes involved passing the uncompressed size here, so the bucket could track both compressed and uncompressed sizes

i added some code to decode the uncompressed size from the compression attribute here, and included the commit cbodley@8e0d9a5 in https://github.com/cbodley/ceph/commits/wip-rgw-compression - feel free to cherry-pick that (it's passing s3 and swift tests)

cbodley · 2016-10-25T17:54:19Z

@yehudasa could you please do a final review? only two remaining blockers: my XXX comment in RGWRados::copy_obj_data(), and the merge conflict with the zero-length segment fix from @mdw-at-linuxbox

cbodley · 2016-10-26T19:50:44Z

@aclamk with help from @mdw-at-linuxbox, I was able to run his test successfully on a rebased version of this branch. this means that, when rebasing, you can resolve the conflicts in RGWGetObj::read_user_manifest_part() normally. the conflict in cls_rgw.cc is also trivial

i pushed my rebased branch to https://github.com/cbodley/ceph/commits/wip-rgw-compression for comparison if that helps

cbodley · 2016-11-01T16:11:28Z

would love to see this make the kraken freeze

@aclamk could you please do the rebase and pull in my fix for copy_obj_data() at cbodley@8e0d9a5?

@yehudasa could you please review my commits related to tracking accounted_size in the bucket stats? specifically, that I'm not missing anything w.r.t. backward compatibility?

Signed-off-by: Alyona Kiseleva <akiselyova@mirantis.com>

Changed RGWPutObjProcessor interface with regard to hash calculation. Signed-off-by: Adam Kupczyk (akupczyk@mirantis.com) Conflicts: src/rgw/rgw_op.cc src/rgw/rgw_rados.cc

…sion filters. Signed-off-by: Adam Kupczyk (akupczyk@mirantis.com)

…so processed. Allows to implement flush-like operations on data. Signed-off-by: Adam Kupczyk (akupczyk@mirantis.com) Conflicts: src/rgw/rgw_op.cc

Allows to plug in decompression in future. Signed-off-by: Adam Kupczyk (a.kupczyk@mirantis.com) Conflicts: src/rgw/rgw_op.cc

Signed-off-by: Adam Kupczyk (akupczyk@mirantis.com)

Signed-off-by: Alyona Kiseleva <akiselyova@mirantis.com>