Enable replication of SSE-C objects

[shtripat]

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers
under the terms of the Apache 2 license.
By creating this pull request I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.

Description

If site replication enabled across sites, replicate the SSE-C objects as well. These objects could be read from target sites using the same client encryption keys.

Motivation and Context

Support replication of client encrypted object to replicated sites.

How to test this PR?

Needs minio/minio-go#1943

1. Set up two MinIO sites with TLS enabled

2. Create a huge (~500MiB) file with known content

for index in {1..10000000}; do echo "$index - The quick brown fox jumps over the lazy dog" >> hugefile;         done

3. Create a bucket in first site and add objects, two with client side encryption and one without any encryption

$ mc cp /etc/hosts ALIAS1/BUCKET --insecure
$ mc cp /etc/issue ALIAS1/BUCKET --encrypt-key "ALIAS1/BUCKET/issue=iliketobecrazybutnotsomuchreally" --insecure`
$ mc cp ./hugefile ALIAS1/BUCKET --encrypt-key "ALIAS1/BUCKET/hugefile=iliketobecrazybutnotsomuchreally" --insecure

4. Setup site replication between the two sites
   mc admin replicate add ALIAS1 ALIAS2 --insecure

5. List and see if bothe objects are present on both the sites

$ mc ls ALIAS1/BUCKET --insecure
[2024-02-28 12:19:43 IST] 1.0KiB STANDARD hosts
[2024-02-28 12:19:38 IST] 514MiB STANDARD hugefile
[2024-02-28 12:19:35 IST]    28B STANDARD issue

$ mc ls ALIAS2/BUCKET --insecure 
[2024-02-28 12:19:43 IST] 1.0KiB STANDARD hosts
[2024-02-28 12:19:38 IST] 514MiB STANDARD hugefile
[2024-02-28 12:19:35 IST]    28B STANDARD issue

6. Stat client encrypted objects from both the sites and make sure it works as expected

$ mc stat ALIAS1/BUCKET/issue --encrypt-key "ALIAS1/BUCKET/issue=iliketobecrazybutnotsomuchreally" --insecure
Name      : issue
Date      : 2024-02-28 12:19:35 IST 
Size      : 28 B   
ETag      : 6b47c0b518de13c30efcc7dc3111d57a 
Type      : file 
Encryption: SSE-C
Metadata  :
  Content-Type: application/octet-stream 
Replication Status: COMPLETED

$ mc stat ALIAS1/BUCKET/hugefile --encrypt-key "ALIAS1/BUCKET/hugefile=iliketobecrazybutnotsomuchreally" --insecure
Name      : hugefile
Date      : 2024-02-28 12:19:38 IST 
Size      : 514 MiB 
ETag      : 2476c8a936a3732a8b046d3376a5de9c-33 
Type      : file 
Encryption: SSE-C
Metadata  :
  Content-Type: application/octet-stream 
Replication Status: COMPLETED  

$ mc stat ALIAS2/BUCKET/issue --encrypt-key "ALIAS2/BUCKET/issue=iliketobecrazybutnotsomuchreally" --insecure
Name      : issue
Date      : 2024-02-28 12:19:35 IST 
Size      : 28 B   
ETag      : 6b47c0b518de13c30efcc7dc3111d57a 
Type      : file 
Encryption: SSE-C
Metadata  :
  Content-Type: application/octet-stream 
Replication Status: REPLICA 

$ mc stat ALIAS2/BUCKET/hugefile --encrypt-key "ALIAS2/BUCKET/hugefile=iliketobecrazybutnotsomuchreally" --insecure
Name      : hugefile
Date      : 2024-02-28 12:19:38 IST 
Size      : 514 MiB 
ETag      : 2476c8a936a3732a8b046d3376a5de9c-33 
Type      : file 
Encryption: SSE-C
Metadata  :
  Content-Type: application/octet-stream 
Replication Status: REPLICA

7. Cat the client encrypted objects from both sites and it should work as expected

$ mc cat ALIAS1/BUCKET/issue --encrypt-key "ALIAS1/BUCKET/issue=iliketobecrazybutnotsomuchreally" --insecure
\S
Kernel \r on an \m (\l)

$ mc cat ALIAS2/BUCKET/issue --encrypt-key "ALIAS2/BUCKET/issue=iliketobecrazybutnotsomuchreally" --insecure
\S
Kernel \r on an \m (\l)

$ mc cat ALIAS1/BUCKET/hugefile --encrypt-key "ALIAS1/BUCKET/issue=iliketobecrazybutnotsomuchreally" --insecure >/dev/null

$ mc cat ALIAS2/BUCKET/hugefile --encrypt-key "ALIAS2/BUCKET/issue=iliketobecrazybutnotsomuchreally" --insecure >/dev/null

$ mc cat ALIAS1/BUCKET/hugefile --encrypt-key "ALIAS1/BUCKET/issue=iliketobecrazybutnotsomuchreally" --insecure | wc -l
10000000

$ mc cat ALIAS2/BUCKET/hugefile --encrypt-key "ALIAS2/BUCKET/issue=iliketobecrazybutnotsomuchreally" --insecure | wc -l
10000000

8. Try the hugefile object to be loaded with different part size using command mc put ./hugefile ALIAS1/BUCKET --encrypt-key "ALIAS1/BUCKET/hugefile=iliketobecrazybutnotsomuchreally" --insecure --part-size 50MiB and replication of this object also should work as expected to other site

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Optimization (provides speedup with no functional changes)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Unit tests added/updated
• Internal documentation updated
• Create a documentation update request here

[poornas]

Did you test multipart replication? You will need additional changes

[shtripat]

Did you test multipart replication? You will need additional changes

I haven't. Will test and do the needful. Thanks for pointing out.

[shtripat]

Did you test multipart replication? You will need additional changes

I haven't. Will test and do the needful. Thanks for pointing out.

I tried with a 500+ MiB object loaded to the bucket with SSE-C and I can see that object gets replicated fine with encryption and I can stat and cat the object from replicated site fine. Even the trace from second site while replication shows a multi part upload happening to it for the object.

[harshavardhana]

@shtripat please update make test-site-replication-minio tests to start using SSE-C as part of the tests. You can also do this after this PR is merged.

[shtripat]

Did you test multipart replication? You will need additional changes

I haven't. Will test and do the needful. Thanks for pointing out.

I tried with a 500+ MiB object loaded to the bucket with SSE-C and I can see that object gets replicated fine with encryption and I can stat and cat the object from replicated site fine. Even the trace from second site while replication shows a multi part upload happening to it for the object.

This was not a correct scenario tested. Still needs fix. Will update the PR soon. Thanks @poornas for pointing this out.

[shtripat]

Did you test multipart replication? You will need additional changes

I haven't. Will test and do the needful. Thanks for pointing out.

I tried with a 500+ MiB object loaded to the bucket with SSE-C and I can see that object gets replicated fine with encryption and I can stat and cat the object from replicated site fine. Even the trace from second site while replication shows a multi part upload happening to it for the object.

This was not a correct scenario tested. Still needs fix. Will update the PR soon. Thanks @poornas for pointing this out.

Updated the PR with multi part replication. Also added steps for verification.

[poornas]

Name      : m
Date      : 2024-02-28 11:11:42 PST
Size      : 666 MiB
ETag      : 4ba9faee8da585edd31820af6783c1f7-42
VersionID : 8b674afc-d080-46b0-8517-10af8f83da41
Type      : file
Encryption: SSE-C
Metadata  :
  Content-Type: application/octet-stream
Replication Status: REPLICA
➜  git:(main) ✗ mc cat siteb/bucket/m --encrypt-key "siteb/bucket/m=iliketobecrazybutnotsomuchreally"
mc: <ERROR> Unable to read from `siteb/bucket/m`. We encountered an internal error, please try again.: cause(sio: unsupported version).

@shtripat , the object is not decrypted properly, your test script is not correct.

@kannappanr - sse-c replication was not implemented initially because of decryption failures around part boundaries. Perhaps @aead can weigh in with suggestions?

[shtripat]

mc stat siteb/bucket/m --encrypt-key "siteb/bucket/m=iliketobecrazybutnotsomuchreally" --versions Name : m Date : 2024-02-28 11:11:42 PST Size : 666 MiB ETag : 4ba9faee8da585edd31820af6783c1f7-42 VersionID : 8b674afc-d080-46b0-8517-10af8f83da41 Type : file Encryption: SSE-C Metadata : Content-Type: application/octet-stream Replication Status: REPLICA ➜ git:(main) ✗ mc cat siteb/bucket/m --encrypt-key "siteb/bucket/m=iliketobecrazybutnotsomuchreally" mc: <ERROR> Unable to read from siteb/bucket/m. We encountered an internal error, please try again.: cause(sio: unsupported version).

@shtripat , the object is not decrypted properly, your test script is not correct.

@kannappanr - sse-c replication was not implemented initially because of decryption failures around part boundaries. Perhaps @aead can weigh in with suggestions?

@poornas to understand better, is it like if we have prefixes under bucket and objects under them and then try with --versions, this problem happens?

[poornas]

mc stat siteb/bucket/m --encrypt-key "siteb/bucket/m=iliketobecrazybutnotsomuchreally" --versions Name : m Date : 2024-02-28 11:11:42 PST Size : 666 MiB ETag : 4ba9faee8da585edd31820af6783c1f7-42 VersionID : 8b674afc-d080-46b0-8517-10af8f83da41 Type : file Encryption: SSE-C Metadata : Content-Type: application/octet-stream Replication Status: REPLICA ➜ git:(main) ✗ mc cat siteb/bucket/m --encrypt-key "siteb/bucket/m=iliketobecrazybutnotsomuchreally" mc: <ERROR> Unable to read from siteb/bucket/m. We encountered an internal error, please try again.: cause(sio: unsupported version).
@shtripat , the object is not decrypted properly, your test script is not correct.
@kannappanr - sse-c replication was not implemented initially because of decryption failures around part boundaries. Perhaps @aead can weigh in with suggestions?

@poornas to understand better, is it like if we have prefixes under bucket and objects under them and then try with --versions, this problem happens?

just try with your test script - $(./mc cat minio2/test-bucket/custpartsize --encrypt-key "minio2/test-bucket/custpartsize=iliketobecrazybutnotsomuchreally" --insecure should be readable

[shtripat]

mc stat siteb/bucket/m --encrypt-key "siteb/bucket/m=iliketobecrazybutnotsomuchreally" --versions Name : m Date : 2024-02-28 11:11:42 PST Size : 666 MiB ETag : 4ba9faee8da585edd31820af6783c1f7-42 VersionID : 8b674afc-d080-46b0-8517-10af8f83da41 Type : file Encryption: SSE-C Metadata : Content-Type: application/octet-stream Replication Status: REPLICA ➜ git:(main) ✗ mc cat siteb/bucket/m --encrypt-key "siteb/bucket/m=iliketobecrazybutnotsomuchreally" mc: <ERROR> Unable to read from siteb/bucket/m. We encountered an internal error, please try again.: cause(sio: unsupported version).
@shtripat , the object is not decrypted properly, your test script is not correct.
@kannappanr - sse-c replication was not implemented initially because of decryption failures around part boundaries. Perhaps @aead can weigh in with suggestions?

@poornas to understand better, is it like if we have prefixes under bucket and objects under them and then try with --versions, this problem happens?

just try with your test script - $(./mc cat minio2/test-bucket/custpartsize --encrypt-key "minio2/test-bucket/custpartsize=iliketobecrazybutnotsomuchreally" --insecure should be readable

Yes, able to hit the issue while reading the whole object. @aead kindly check and suggest.

[harshavardhana]

There is some persistent failure here looks like due to the changes in this PR @shtripat please investigate.

[shtripat]

There is some persistent failure here looks like due to the changes in this PR @shtripat please investigate.

Sure, let me check and fix this. Thanks!