[WIP] Add support of objects encryption

[vadmeste]

This change is not complete yet.

Streaming encryption produces encrypted data with a length equals to the plain version of the data. We need streaming so mirroring can work effectively: mirroring needs to know the size of the source and target object size to compare and decide if it should mirror or not, but in some encryption algorithms like CBC, it is not the same (though close) so it would be difficult to decide if we should upload or not.

Streaming has its own disadvantages.. I am investigation..

Fixes #2074

[vadmeste]

My finding about possible encryption schemes:

Long version:

AES + CBC: we already use it in minio-go, it is a block cipher and not stream cipher, it means the size of cipher text is different to the plain text. With CBC, we cannot calculate exactly the size of plain text given the size of cipher text. But we can have an approximate number.

AES + OFB or CTR: both are streaming cipher, which means the length of the cipher text is equal to the length of plain text. The problem with both algorithms is that they don't detect malicious or accidental data corruption. The solution is to use HMAC in addition to OFB or CTR (https://leanpub.com/gocrypto/read)

AES + GCM: already supported in AWS SDK, also the maximum length of data that it can encrypt is 64 GB (https://aws.amazon.com/blogs/developer/amazon-s3-client-side-authenticated-encryption/). GCM in go api is not streaming, but I tested GCM with aws-sdk-java, and I noticed that uploading a 5gb file didn't consume more than 250MB RAM. So we can follow what aws-sdk-java does.

Short version
AES + CBC: has an issue with mirroring. If we add a byte in an object, we can't be sure that the encrypted object needs to be updated.

AES + OFB/CTR is the easiest to implement and should be secure.

AES + GCM: it is an opportunity if we want to implement GCM in minio-go since it has known performance reputation.

[harshavardhana]

AES + OFB/CTR is the easiest to implement and should be secure.

AES + GCM: it is an opportunity if we want to implement GCM in minio-go since it has known performance reputation.

We can proceed to do OFB/CTR (HMAC) - AES+GCM @aead is already working on a package for server side encryption which we can bring this later. In addition he is also adding chachapoly as well.

[aead]

I don't know the exact (S3 compatibility) requirements but I would only recommend AEAD ciphers.
There basically two widely used ciphers:

• AES-GCM
• ChaCha20Poly1305

Everything else is not "state-oft-the-art". E.g. AES-(CTR, OFB, CBC, etc.) won't prevent data modification. Depending on the stored objects it is trivial to send malicious (but valid) content to clients.
Further it isn't a good (crypto) design to support algorithms just because they exist - because if they are used, you have to support them.

@vadmeste May take a look at chacha20poly1305 - I'm still working on a (more or less) "streamable" AES-GCM implementation - but this will take some time.

• AES-CTR + HMAC/Keyed-BLAKE2b (over ciphertext) is also fine, but much slower than AEAD.

[deekoder]

closing after discussing with @vadmeste