-
Notifications
You must be signed in to change notification settings - Fork 24.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Client Side Encrypted Snapshot Repositories #41910
Comments
Pinging @elastic/es-security |
Pinging @elastic/es-distributed |
We discussed this today, in our weekly team meeting, but got into extra time pondering the alternatives. Yet we settled that we don't need to support moving snapshots between repositories. I would like to kindly ask the distributed team for any input. |
what do you mean by
It seems to me for the first option we could "simply" pass a secure setting for the current encryption key to That said, I like the first option much better than doing some SDK specific thing just for S3. In the end it seems like that is probably less effort maintenance-wise long term since relying on the SDK's implementations of this completely puts us at the mercy of whatever changes happen with that. Plus, as you point out, working with the SDKs only will be tricky to test and not cover the FS repository. I would point out one thing though (sorry if this was already discussed, just ignore this if it was :)): The snapshot mechanism uses blob names as part of it's logic somewhat extensively. Even if we client side encrypt every blob, we'd still be leaking the following information:
Not sure if that's a compliance problem, but that would certainly be something that would be challenging to not leak via the blob names. That's all I have for now. Happy to help review you work though :) |
@original-brownbear
Yes, that's the first option I was trying to describe. What I mean when I say we duplicate code, is that the "crypto logic" (the envelope encryption, AES algorithm, all that) will most likely be very similar (on purpose) to what the SDK already does.
I think that's a very thoughtful observation, and that it should definitely get in the docs. I don't believe there are regulations for that, and we are not aiming for a specific compliance target, but I'm no expert either. Maybe @joshbressers is more knowledgeable in this regard? I propose we clearly acknowledge this limitation in the docs and act on it only if we get specific requests.
Glad to hear! Thank you! |
Ideally we don't want to leak any metadata, but I know sometimes it's unavoidable. We probably won't run afoul of any compliance standards here. We could see some interest from certain sensitive customers, but generally their concern revolves around leaking names more than this sort of metadata. |
Thank you for the answer @joshbressers ! I merely wish to reinforce this position by highlighting that leaking this type of metadata tips off cluster configuration but no information on the actual data. |
I think it would be preferable to implement this ourselves and not rely on the blob-store libraries to do it. Ultimately, we need this for multiple repository types, and we could use the cloud SDKs for it, but we would still need to build & verify it for each provider, which wouldn't gain us very much over just building it ourselves. |
Here is the 10 thousand feet view of the currently favored approach. We create a new type of blob store repository, the Encryption uses the "envelope" strategy. This means that there is a data encryption key (DEK) and a key encryption key (KEK). The DEK is generated randomly for each "blob", and it encrypts the blob (the actual data). The KEK is a secret parameter of the "encrypted" repository, and it encrypts every DEK. In this case, the encrypted repository is comprised of "blobs" encrypted with DEKs, and of DEKs encrypted with the KEK. The KEK is not stored publicly. The encryption algorithm for the DEK encrypting blobs is AES in the GCM mode using a 256 bit key length. For the DEK encryption with the KEK the same AES algorithm is used but with the ECB mode (this is intrinsic in the One un-encrypted blob "generates" one encrypted blob, and another blob containing the encrypted DEK. Ideally we should have a versioned format for an "encryption metadata" blob which contains the encrypted DEK and other description type of data, such as the IV. The metadata blob should be hashed but plain , aside from the encrypted DEK. The plan is to use the We have a plethora of KEK storage/generation. The code in the current POC generates it from a text password from the keystore. Other alternatives include to store the binary key in the keystore or a separate file, or source it via KMIP, Amazon KMP, or delegate the "key unwrap" operation via KMIP . We don't have to limit it to a single method but we need to decide what the method is for the first iteration (CC @bytebilly). Key rotation happens via an ES API . Key rotation implies the use of the old KEK to decrypt all the DEKs and re-encrypt them with the new KEK. Key rotation is depended upon the way we store the KEK. With the current approach of storing a text password in the keystore, the keystore reload call can also rotate keys. But it might not be desirable to hog the reload API call for the reload operation. Another approach is defer the rotation until the next snapshot (which could be an empty one) rather than creating a new API for it. In any case, because of the failure situations we would have juggle both the old and the new KEKs inside the keystore at least for some time, but the precise flow is yet TBD. Here is the POC were I've explored these choices #46170 . |
Here is what it still needs to be done/investigated, the order is somehow important:
|
We discussed the list above and prioritized some items as follows, so as to be sure we resolve all unknowns as soon as possible:
This last point requires input from the product team @bytebilly . |
@albertzaharovits I totally agree with the client-side management of encryption keys.
We can consider using a password to seed the KEK, and store it in the node keystore. This is a viable first iteration, and I don't see blockers in current enhancement requests to suggest something different. Further support for cloud-specific key management systems could be added in a second step. Storing the binary KEK in the keystore is not very useful in my opinion, since it doesn't increase security (KEK can be directly used to decrypt DEK, so it's not a safer option). The text password is easier to use in command line tools we can eventually provide to manually manage encrypted snapshots. A possible flow could have a flexible configuration that defines the password source. It can define if the password is in the keystore, or if it should be retrieved from an external source. The first part is what we can ship first. Password rotation could occur automatically every new snapshot, and in addition we can provide a specific API for that. I expect customers may need to guarantee rotation within a well-defined range for regulations. I'd rather avoid coupling rotation with keystore reload. With this approach, we provide an out-of-the-box key rotation for everyone (every new snapshot), but we also allow rotation on-demand for customers with specific needs. What I'm still missing, is who defines this password. Is it user defined, or automatically generated by the system? In the first case, how do we deal with key rotation, since it would replace the user-defined value? |
Thanks for looking into it @bytebilly !
Good to hear that.
As far as "passwords" are concerned I think they should reside in the keystore only. Subsequent iterations on this feature could "source" the secret to seed the KEK from external systems, but in this case I think it makes more sense to get the actual KEK, and not do any alterations to that.
Note that the current design aims for a single KEK per repository not per snapshot. Adding a new API to perform the rotation is better compared with coupling this operation with the keystore reload. If the old and the new KEKs during rotation are self-descriptive (meaning the rotation API can tell which one supersedes the other, eg last modified date for keystore entries) then it is possible to do without a new API and trigger the rotation of the whole repository on the next snapshot (which could be empty). But I'm getting ahead of myself, I'll plan for the API not for self-descriptive keys, where you explicitly name in the API the old and the new keys (assumes all keys are "nameable").
If keys are nameable it should work with the API that's to be introduced in the first iteration. Names will look like URIs, I think this is how they are referred to by all cloud providers.
User defined. I'm open to suggestion to have it seeded by a random value in the keystore, although it doesn't sound too useful to me, but if it enhances UX I am open to it, but it's not consequential at this stage.
Yeah, both values should be available in the keystore simultaneously. They should have "similar" names (under the same namespace).
Yes, on every node. We have infrastructure to be assured that they are all equal on all nodes (not used as of right now, but the infra is there). |
In #53352 I've raised a PR with an implementation of the encrypted Encrypted snapshots are implemented as a new repository type, under a module of the x-pack plugin. Encrypted snapshots are available for the following existing repository types: S3, Azure, GCS and FS. An encrypted snapshot encrypts all the data that is part of the snapshot before it is uploaded to the storage service. Snapshots are encrypted when an ordinary snapshot operation is performed against a repository of the new encrypted type. It is not possible to encrypt the snapshots in an existing regular repository. An encrypted repository is created similarly to creating the regular repository of the same type (eg S3). The same APIs are used, but, in addition, creating encrypted repositories require a new repository setting (cluster state) which names the secure setting holding the repository password which is used to derive the encryption keys (See the example at #53352 (comment) for how to create an encrypted FS repository). More technically, encrypted snapshots work by encrypting (AES-256) all data at the blob level, that's uploaded to the storage service. The data encryption keys (DEK) are generated randomly by every node. A generated DEK is reused locally by the node at most for the lifetime of repository (until the repository is deleted or the node is shut down), but the exact details on when a DEK is reused is an implementation detail (it is deliberated starting at #50846 (comment)). The association of the encrypted blob to its DEK is realized by prefixing a DEK name to the encrypted blob. DEKs themselves are encrypted (AES Wrap) and stored under a location, which contains the DEK name, in the storage service as well. The key encryption keys (KEK), used to encrypt the DEKs, are generated starting from the repository password using the PBKDF2 algorithm. The association between the KEK and the DEKs it wraps is realized by storing the wrapped DEK under a path location that requires knowledge of the password (it's again an implementation detai). Theoretically, there could only be one KEK in existence ever (for the lifetime of the repository password), but it is combersome to ensure all the participants use the same KEK, so a relaxed approach has been adopted which derives the KEK by using the DEK name as a salt in the PBKDF2 function (because the DEK name is generated randomly for the purpose of uniqueness anyway). |
The client-side encrypted repository is a new type of snapshot repository that internally delegates to the regular variants of snapshot repositories (of types Azure, S3, GCS, FS, and maybe others but not yet tested). After the encrypted repository is set up, it is transparent to the snapshot and restore APIs (i.e. all snapshots stored in the encrypted repository are encrypted, no other parameters required). The encrypted repository is protected by a password stored on every node's keystore (which must be the same across the nodes). The password is used to generate a key encrytion key (KEK), using the PBKDF2 function, which is used to encrypt (using the AES Wrap algorithm) other symmetric keys (referred to as DEK - data encryption keys), which themselves are generated randomly, and which are ultimately used to encrypt the snapshot blobs. For example, here is how to set up an encrypted FS repository: ------ 1) make sure that the cluster runs under at least a "platinum" license (simplest test configuration is to put `xpack.license.self_generated.type: "trial"` in the elasticsearch.yml file) 2) identical to the un-encrypted FS repository, specify the mount point of the shared FS in the elasticsearch.yml conf file (on all the cluster nodes), e.g. `path.repo: ["/tmp/repo"]` 3) store the repository password inside the elasticsearch.keystore, *on every cluster node*. In order to support changing password on existing repository (implemented in a follow-up), the password itself must be names, e.g. for the "test_enc_key" repository password name: `./bin/elasticsearch-keystore add repository.encrypted.test_enc_pass.password` *type in the password* 4) start up the cluster and create the new encrypted FS repository, named "test_enc", by calling: ` curl -X PUT "localhost:9200/_snapshot/test_enc?pretty" -H 'Content-Type: application/json' -d' { "type": "encrypted", "settings": { "location": "/tmp/repo/enc", "delegate_type": "fs", "password_name": "test_enc_pass" } } ' ` 5) the snapshot and restore APIs work unmodified when they refer to this new repository, e.g. ` curl -X PUT "localhost:9200/_snapshot/test_enc/snapshot_1?wait_for_completion=true"` Related: #49896 #41910 #50846 #48221 #65768
The client-side encrypted repository is a new type of snapshot repository that internally delegates to the regular variants of snapshot repositories (of types Azure, S3, GCS, FS, and maybe others but not yet tested). After the encrypted repository is set up, it is transparent to the snapshot and restore APIs (i.e. all snapshots stored in the encrypted repository are encrypted, no other parameters required). The encrypted repository is protected by a password stored on every node's keystore (which must be the same across the nodes). The password is used to generate a key encrytion key (KEK), using the PBKDF2 function, which is used to encrypt (using the AES Wrap algorithm) other symmetric keys (referred to as DEK - data encryption keys), which themselves are generated randomly, and which are ultimately used to encrypt the snapshot blobs. For example, here is how to set up an encrypted FS repository: ------ 1) make sure that the cluster runs under at least a "platinum" license (simplest test configuration is to put `xpack.license.self_generated.type: "trial"` in the elasticsearch.yml file) 2) identical to the un-encrypted FS repository, specify the mount point of the shared FS in the elasticsearch.yml conf file (on all the cluster nodes), e.g. `path.repo: ["/tmp/repo"]` 3) store the repository password inside the elasticsearch.keystore, *on every cluster node*. In order to support changing password on existing repository (implemented in a follow-up), the password itself must be names, e.g. for the "test_enc_key" repository password name: `./bin/elasticsearch-keystore add repository.encrypted.test_enc_pass.password` *type in the password* 4) start up the cluster and create the new encrypted FS repository, named "test_enc", by calling: ` curl -X PUT "localhost:9200/_snapshot/test_enc?pretty" -H 'Content-Type: application/json' -d' { "type": "encrypted", "settings": { "location": "/tmp/repo/enc", "delegate_type": "fs", "password_name": "test_enc_pass" } } ' ` 5) the snapshot and restore APIs work unmodified when they refer to this new repository, e.g. ` curl -X PUT "localhost:9200/_snapshot/test_enc/snapshot_1?wait_for_completion=true"` Related: elastic#49896 elastic#41910 elastic#50846 elastic#48221 elastic#65768
The client-side encrypted repository is a new type of snapshot repository that internally delegates to the regular variants of snapshot repositories (of types Azure, S3, GCS, FS, and maybe others but not yet tested). After the encrypted repository is set up, it is transparent to the snapshot and restore APIs (i.e. all snapshots stored in the encrypted repository are encrypted, no other parameters required). The encrypted repository is protected by a password stored on every node's keystore (which must be the same across the nodes). The password is used to generate a key encrytion key (KEK), using the PBKDF2 function, which is used to encrypt (using the AES Wrap algorithm) other symmetric keys (referred to as DEK - data encryption keys), which themselves are generated randomly, and which are ultimately used to encrypt the snapshot blobs. For example, here is how to set up an encrypted FS repository: ------ 1) make sure that the cluster runs under at least a "platinum" license (simplest test configuration is to put `xpack.license.self_generated.type: "trial"` in the elasticsearch.yml file) 2) identical to the un-encrypted FS repository, specify the mount point of the shared FS in the elasticsearch.yml conf file (on all the cluster nodes), e.g. `path.repo: ["/tmp/repo"]` 3) store the repository password inside the elasticsearch.keystore, *on every cluster node*. In order to support changing password on existing repository (implemented in a follow-up), the password itself must be names, e.g. for the "test_enc_key" repository password name: `./bin/elasticsearch-keystore add repository.encrypted.test_enc_pass.password` *type in the password* 4) start up the cluster and create the new encrypted FS repository, named "test_enc", by calling: ` curl -X PUT "localhost:9200/_snapshot/test_enc?pretty" -H 'Content-Type: application/json' -d' { "type": "encrypted", "settings": { "location": "/tmp/repo/enc", "delegate_type": "fs", "password_name": "test_enc_pass" } } ' ` 5) the snapshot and restore APIs work unmodified when they refer to this new repository, e.g. ` curl -X PUT "localhost:9200/_snapshot/test_enc/snapshot_1?wait_for_completion=true"` Related: #49896 #41910 #50846 #48221 #65768
I've created the following two diagrams on how keys are generated and used internally: They are sketchy and not very professional, but they are accurate as the code currently stands, and, I hope, informative. |
@albertzaharovits hello. The idea with server side encryption is very cool. Can you tell if there are elasticsearch builds with the |
This concerns the encryption of snapshot data before it leaves the nodes.
We have 3 types of cloud snapshot repository types: Google Cloud Storage, Azure Storage and Amazon S3. Amazon and Azure support client side encryption for their java clients, but Google does not.
Amazon and Azure, which support client side encryption, allow the keys to be managed by the client (us) or by their Key Management Service (Vault-like). They both use the Envelope Encryption method; each blob is individually AES-256 encrypted with a randomly generated (locally) key, and this key (Data/Content Encryption Key) is also encrypted with another Master Key (locally or by the Vault service) and then stored alongside the blob in its metadata. The envelope encryption facilitates Master Key rotation because only the small (D/C)EK key has to be re-encrypted, rather than the complete blob.
On the ES side we discussed on having a single fixed URN key handler at the repository settings level.
This URN identifies the Master Key; for example this could point to a key on the Amazon Vault Service or the Keys on each node's keystore. In this alternative it is not possible to rotate the keys via the repository API (it might be possible to do it outside ES, which is obviously preferable, but see below).
I believe this is the rough picture of the puzzle that we need to put together.
We oscillated between implementation alternatives, and I will lay out the one which I think is favorable. Whatever solution we initially implement, given that the Master Key identifier is an URN we can multiplex multiple implementations for the same repository type.
We mirror the Envelope Encryption algorithm, employed by Amazon and Azure, at the
BlobContainer
level. The key is stored on each node's keystore (and is pointed to by the repository level URN reference).Advantages:
EncryptedBlobContainer
and end-to-end integration tests in only one of the implementation (Amazon or FS), where we can decrypt the data on the service fixture.Disadvantages:
In the opposite corner, there could be this alternative:
We use the AWS cloud library facility to implement it only for the S3 repository type. The key is stored either on the node's keystore or on the AWS Key Management Service.
Advantages:
Disadvantages:
Relates #34454 #40416
EDITED 28.01.2021 Backlog:
The encrypted repository must already be configured with the correct password. The API iterates over all the associated
wrapped DEKs (contained inside a dedicated blob container under the repository's base path), and proceeds to unwrap and re-wrap all the DEKs with the new password.
Finally, the old DEKs are removed, so that the old password cannot be used any longer.
This mainly requires implementing the AbstractBlobContainer#readBlob interface. This is slightly problematic because the association id between an encrypted blob and its DEK is prepended at the beginning of the blob, so that decryption at an internal position currently requires a seek at the beginning. Double check that the definition structure is reasonable (ie. is it searchable and encrypted or vice-versa, ping David about this).
two operations that can fail independently. Make sure testing covers this.
true
and then reading to fail because of decryption problemsOn-premise repository passwords are cached in memory when the node starts, usually requiring a node restart when configuring a new snapshot repository. The security settings implementation on cloud is different, so that maybe we can read the repository passwords immediately after they've been added, changed, without requiring a restart.
The text was updated successfully, but these errors were encountered: