rgw: support full object encryption stack on compression

[Zhiwei-Dai]

replace #36539
add implementation for full object encryption stack on compression. Compressing first and then encrypting.

Security and trustworthiness framework of ceph is important for data storage. Now, cephx and https base on openssl enhance the security of transport layer, and rgw encrytion implementation supports data protection for object store.
Nevertheless, as issues said, issues19988, space usage and data security are exclusive.
New compression and encryption filters do not stack, but logically could do so. Users must choose between at rest data reduction and data security.
The pr solves this problem. Why compressing first and then encrypting, discussion about data encryption and compression has answer
https://crypto.stackexchange.com/questions/33737/is-it-better-to-encrypt-before-compression-or-vice-versa

Fixed: https://tracker.ceph.com/issues/19988
Signed-off-by: Dai Zhiwei daizhiwei3@huawei.com
Signed-off-by: luo rixin luorixin@huawei.com

Contribution Guidelines

• To sign and title your commits, please refer to Submitting Patches to Ceph.

• If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

• Tracker (select at least one)
  • References tracker ticket
  • Very recent bug; references commit where it was introduced
  • New feature (ticket optional)
  • Doc update (no ticket needed)
  • Code cleanup (no ticket needed)
• Component impact
  • Affects Dashboard, opened tracker ticket
  • Affects Orchestrator, opened tracker ticket
  • No impact that needs to be tracked
• Documentation (select at least one)
  • Updates relevant documentation
  • No doc update is appropriate
• Tests (select at least one)
  • Includes unit test(s)
  • Includes integration test(s)
  • Includes bug reproducer
  • No tests

Show available Jenkins commands

• jenkins retest this please
• jenkins test classic perf
• jenkins test crimson perf
• jenkins test signed
• jenkins test make check
• jenkins test make check arm64
• jenkins test submodules
• jenkins test dashboard
• jenkins test dashboard cephadm
• jenkins test api
• jenkins test docs
• jenkins render docs
• jenkins test ceph-volume all
• jenkins test ceph-volume tox
• jenkins test windows

[cbodley]

@Zhiwei-Dai thanks for the contribution, this is a great feature! i'm seeing some test failures though, could you please take a look?

this is with a vstart cluster with rgw compression enabled:

~/ceph/build $ MON=1 OSD=1 RGW=1 MDS=0 MGR=0 ../src/vstart.sh -n -d --rgw_compression lz4

and running the ceph-master branch of ceph's s3-tests for sse encryption:

~ $ git clone https://github.com/ceph/s3-tests.git -b ceph-master
...
~ $ cd s3-tests
~/s3-tests $ ./bootstrap
...
~/s3-tests $ S3TEST_CONF=s3tests.conf.SAMPLE virtualenv/bin/nosetests -v -a 'encryption,!sse-s3' s3tests_boto3
...
s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_upload ... ERROR
...
s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_bad_download ... ERROR
...
s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload ... ERROR

in addition to those 3 test failures, i also found that GET requests hung if i configured rgw with rgw max chunk size = 1048576

[cbodley]

for automated test coverage, we'll want to enable compression in rgw's encryption suite:

diff --git a/qa/suites/rgw/crypt/3-rgw/rgw.yaml b/qa/suites/rgw/crypt/3-rgw/rgw.yaml
index ee8d62af05c..764d216aade 100644
--- a/qa/suites/rgw/crypt/3-rgw/rgw.yaml
+++ b/qa/suites/rgw/crypt/3-rgw/rgw.yaml
@@ -6,6 +6,8 @@ overrides:
         setgroup: ceph
         rgw crypt require ssl: false
         debug rgw: 20
+  rgw:
+    compression type: random
 
 tasks:
 - rgw:

can you please add this to the PR?

[Zhiwei-Dai]

@Zhiwei-Dai thanks for the contribution, this is a great feature! i'm seeing some test failures though, could you please take a look?

in addition to those 3 test failures, i also found that GET requests hung if i configured rgw with rgw max chunk size = 1048576

ths your testing, i'll recheck it.

[Zhiwei-Dai]

can you please add this to the PR?

ok, of course

[Zhiwei-Dai]

@cbodley I have updated the implementation and ceph's s3-tests for sse encryption are all passed.

[Zhiwei-Dai]

@cbodley would you mind review the pr again? Thanks

[cbodley]

passed qa in https://pulpito.ceph.com/cbodley-2022-10-19_23:28:37-rgw-wip-cbodley-testing-distro-default-smithi/ https://pulpito.ceph.com/cbodley-2022-10-20_13:42:11-rgw:crypt-wip-cbodley-testing-distro-default-smithi/

[cbodley]

thanks @Zhiwei-Dai!

[cbodley]

i've opened #48609 to document this new feature in PendingReleaseNotes

[Zhiwei-Dai]

i've opened #48609 to document this new feature in PendingReleaseNotes

Nice work