rgw: implement GetObjectAttributes #55259

mattbenjamin · 2024-01-20T22:44:21Z

Fixes: https://tracker.ceph.com/issues/64109

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows
jenkins test rook e2e

dang · 2024-01-22T14:47:17Z

src/rgw/rgw_rest_s3.cc

+   <Checksum>
+      <ChecksumCRC32>string</ChecksumCRC32>
+      <ChecksumCRC32C>string</ChecksumCRC32C>
+      <ChecksumSHA1>string</ChecksumSHA1>


Are these for future implementation?

yes, I do have a whole checksum PR, I wasn't sure if I wanted this one to go before or after it; the checksum part also explicitly requires part enumeration

cbodley · 2024-01-22T14:28:45Z

src/rgw/rgw_op.cc

+    /* XXXX the following conjunction should be &&--but iam_action2 is currently not
+     * hooked up and always fails (but should succeed if the requestor has READ
+     * acess to the object) */


try adding the new s3GetObjectAttributes/s3GetObjectVersionAttributes values to the mapping in

ceph/src/rgw/rgw_iam_policy.cc

Lines 64 to 65 in 62be5c7

static const actpair actpairs[] =

{{ "s3:AbortMultipartUpload", s3AbortMultipartUpload },

thank you, that sounds right

cbodley · 2024-01-22T15:42:44Z

src/rgw/rgw_op.cc

+void RGWGetObjAttrs::execute(optional_yield y)
+{
+  RGWGetObj::execute(y);
+  /* XXXX now we need logic from list parts */


#50148 added some stuff that should be useful for this

in rgw_rados.cc, get_part_obj_state() interprets the manifest for the total parts_count, seeks to the requested part, then reads the part's head object

this functionality is exposed by sal::Object::ReadOp with an optional part_num input param, and parts_count output param

for your purposes here, i'd start with a ReadOp::prepare() for the part_num corresponding to x-amz-part-number-marker + 1. that'll give you the first part's size and xattrs for stuff like checksums. it also returns the total parts_count that you can continue to loop over

just note that there's a special case for part_num=1 on non-multipart uploads described in #50148 (comment)

thanks, casey

github-actions · 2024-03-24T01:26:11Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2024-04-12T18:12:37Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

mattbenjamin · 2024-04-12T18:23:30Z

(will rebase this next week)

Among other things, provides XXH128 and XXH3. Includes compile fixes for gcc-13. Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Fast, cryptographic hash functions suitable for block checksums. BLAKE3 is the faster, more portable successor to Blake2(b,s), now with 4x throughput. Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Adds new Blake3 digest format (native), a concrete type to represent digests, and static_visitor machinery to unify varying checksum computations. This framework, together with new trailing checksum header support, is used to implement S3 additional checksum verification. Parts of the AWS content checksum API work build on a prior contribution from imtzw <tongzhiwei_yewu@cmss.chinamobile.com>. Thank you! Fixes: https://tracker.ceph.com/issues/42080 Fixes: https://tracker.ceph.com/issues/63951 squashed commits: * rgw_cksum: add trival test vectors for sha-format digests Computed digests match those produced by sha1sum, sha256sum, and sha512sum utilities. * rgw_cksum: add test vectors for blake3 Tests the same input strings with digests validated by b3sum (https://crates.io/crates/b3sum). * rgw_ckum: switch to accel crc32c The internal Ceph convention appears to be to omit a final xor where ceph_crc32c is used, but it's required for compatibility with AWS implementations. * rgw_cksum: add XXH3 digest * rgw_cksum: write class encoder for rgw::digest::Cksum * rgw_cksum: also reverse crc32c (REBASEME) Mark noticed that the crc32c output was being tested against a byteswapped value (crc32c also needs byteswap on LE). * rgw_cksum: add digest::Cksum serde tests * rgw_cksum: fix main(...) linkage (so we run our main unit and not the one in gmock * rgw_cksum: convenience extensions for integration with RGW/S3 * introduce rgw_cksum unique_ptr factory * rgw_cksum: mark string transform accessors const * rgw_cksum: fixup unittest_rgw_chksum compilation--all existing tests pass * rgw_cksum: hook up put-object checksum workflow * tweaks to report on content checksum mismatch * rgw_cksum: match SDK as well as general checksum header * make it more efficient * initialize RGWPutObj_Cksum::digest * rgw_cksum: write parse_cksum_type w/const char* arg * initialize _type correctly; doing armored wrong * fix expected checksum header name, clean up verify * fix output on checksum verify fail, cleanup * introduce Cksum::to_armor(); all AWS cases pass * oops, extra 0-byte at end of to_armor() result * use to_armor() with decoded checksums (i.e., for all S3 presentation) * remove unnecessary finalize() in RGWPutObj_Cksum dtor * RGWPutObj_Cksum::Factory fixes * fixes test_object_checksum_sha256 * choose preferred checksum algorithm header if both are present * log verified checksums in RGWPutObj::execute at 16 * checksum not needed in policy condition * fix checksum trailing header format Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>