Client-side encrypted snapshot repository (feature flag) #66773
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This builds upon the data encryption streams from #49896 to create an encrypted snapshot repository. The repository encryption works with the following existing repository types: FS, Azure, S3, GCS (possibly works with HDFS and URL, but these are not tested). The encrypted repository is protected by a password stored on every node's keystore. The repository keys (KEK - key encryption key) are generated from the password using the PBKDF2 function, and are used to encrypt (using the AES Wrap algorithm) other symmetric keys (referred to as DEK - data encryption keys) which are themselves used to encrypt the blobs of the regular snapshot. The platinum or enterprise licenses are required to snapshot to the encrypted repository, but no license is required to list or restore already encrypted snapshots.
The encrypted repository plugin only adds the new repository plugin factory if the feature flag `es.encrypted_repository_feature_flag_registered` is toggled.
Encrypted snapshots are available on active PLATINUM and ENTERPRISE license modes. Listing encrypted repositories, as well as restoring and deleting encrypted snapshots is still permitted after the license expires or is downgraded, to avoid a data hostage situation. This PR updates the last usage time stamp of the encrypted snapshots feature to the creation time of the most recent encrypted snapshot.
albertzaharovits
changed the title
albertzaharovits
added
:Security/Security
elasticmachine
added
Team:Distributed (Obsolete)
|
Pinging @elastic/es-distributed (Team:Distributed) |
|
Pinging @elastic/es-security (Team:Security) |
|
Since this is a merge of a feature branch, where all the PRs (excluding nits, compilation and test fixes) contained on the feature branch have been independently reviewed, I'm going to merge this without asking for reviews again. Because this is merged behind a feature flag, I'm going to label this as |
albertzaharovits
added a commit
to albertzaharovits/elasticsearch
that referenced
this pull request
Dec 23, 2020
The client-side encrypted repository is a new type of snapshot repository that
internally delegates to the regular variants of snapshot repositories (of types
Azure, S3, GCS, FS, and maybe others but not yet tested). After the encrypted
repository is set up, it is transparent to the snapshot and restore APIs (i.e. all
snapshots stored in the encrypted repository are encrypted, no other parameters
required).
The encrypted repository is protected by a password stored on every node's
keystore (which must be the same across the nodes).
The password is used to generate a key encrytion key (KEK), using the PBKDF2
function, which is used to encrypt (using the AES Wrap algorithm) other
symmetric keys (referred to as DEK - data encryption keys), which themselves
are generated randomly, and which are ultimately used to encrypt the snapshot
blobs.
For example, here is how to set up an encrypted FS repository:
------
1) make sure that the cluster runs under at least a "platinum" license
(simplest test configuration is to put `xpack.license.self_generated.type: "trial"`
in the elasticsearch.yml file)
2) identical to the un-encrypted FS repository, specify the mount point of the
shared FS in the elasticsearch.yml conf file (on all the cluster nodes),
e.g. `path.repo: ["/tmp/repo"]`
3) store the repository password inside the elasticsearch.keystore, *on every cluster node*.
In order to support changing password on existing repository (implemented in a follow-up),
the password itself must be names, e.g. for the "test_enc_key" repository password name:
`./bin/elasticsearch-keystore add repository.encrypted.test_enc_pass.password`
*type in the password*
4) start up the cluster and create the new encrypted FS repository, named "test_enc", by calling:
`
curl -X PUT "localhost:9200/_snapshot/test_enc?pretty" -H 'Content-Type: application/json' -d'
{
"type": "encrypted",
"settings": {
"location": "/tmp/repo/enc",
"delegate_type": "fs",
"password_name": "test_enc_pass"
}
}
'
`
5) the snapshot and restore APIs work unmodified when they refer to this new repository, e.g.
` curl -X PUT "localhost:9200/_snapshot/test_enc/snapshot_1?wait_for_completion=true"`
Related: elastic#49896 elastic#41910 elastic#50846 elastic#48221 elastic#65768
albertzaharovits
added a commit
that referenced
this pull request
Dec 28, 2020
The client-side encrypted repository is a new type of snapshot repository that
internally delegates to the regular variants of snapshot repositories (of types
Azure, S3, GCS, FS, and maybe others but not yet tested). After the encrypted
repository is set up, it is transparent to the snapshot and restore APIs (i.e. all
snapshots stored in the encrypted repository are encrypted, no other parameters
required).
The encrypted repository is protected by a password stored on every node's
keystore (which must be the same across the nodes).
The password is used to generate a key encrytion key (KEK), using the PBKDF2
function, which is used to encrypt (using the AES Wrap algorithm) other
symmetric keys (referred to as DEK - data encryption keys), which themselves
are generated randomly, and which are ultimately used to encrypt the snapshot
blobs.
For example, here is how to set up an encrypted FS repository:
------
1) make sure that the cluster runs under at least a "platinum" license
(simplest test configuration is to put `xpack.license.self_generated.type: "trial"`
in the elasticsearch.yml file)
2) identical to the un-encrypted FS repository, specify the mount point of the
shared FS in the elasticsearch.yml conf file (on all the cluster nodes),
e.g. `path.repo: ["/tmp/repo"]`
3) store the repository password inside the elasticsearch.keystore, *on every cluster node*.
In order to support changing password on existing repository (implemented in a follow-up),
the password itself must be names, e.g. for the "test_enc_key" repository password name:
`./bin/elasticsearch-keystore add repository.encrypted.test_enc_pass.password`
*type in the password*
4) start up the cluster and create the new encrypted FS repository, named "test_enc", by calling:
`
curl -X PUT "localhost:9200/_snapshot/test_enc?pretty" -H 'Content-Type: application/json' -d'
{
"type": "encrypted",
"settings": {
"location": "/tmp/repo/enc",
"delegate_type": "fs",
"password_name": "test_enc_pass"
}
}
'
`
5) the snapshot and restore APIs work unmodified when they refer to this new repository, e.g.
` curl -X PUT "localhost:9200/_snapshot/test_enc/snapshot_1?wait_for_completion=true"`
Related: #49896 #41910 #50846 #48221 #65768
|
What happened to this feature? Trying it on v8 seems to indicate its not implemented, and I cannot find any references in the documentation.
|
|
Unfortunately this was abandoned before reaching the maturity level for functionality required for general availability. Could you maybe share any details on the setup you were looking to implement, in order to help us design for it when we will resume work? Information such as:
Lastly, what are you going to use instead given that ES does not support it today? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The client-side encrypted repository is a new type of snapshot repository that
internally delegates to the regular variants of snapshot repositories (of types
Azure, S3, GCS, FS, and maybe others but not yet tested). After the encrypted
repository is set up, it is transparent to the snapshot and restore APIs (i.e. all
snapshots stored in the encrypted repository are encrypted, no other parameters
required).
The encrypted repository is protected by a password stored on every node's
keystore (which must be the same across the nodes).
The password is used to generate a key encrytion key (KEK), using the PBKDF2
function, which is used to encrypt (using the AES Wrap algorithm) other
symmetric keys (referred to as DEK - data encryption keys), which themselves
are generated randomly, and which are ultimately used to encrypt the snapshot
blobs.
For example, here is how to set up an encrypted FS repository:
(simplest test configuration is to put
xpack.license.self_generated.type: "trial"in the elasticsearch.yml file)
shared FS in the elasticsearch.yml conf file (on all the cluster nodes),
e.g.
path.repo: ["/tmp/repo"]In order to support changing password on existing repository (implemented in a follow-up),
the password itself must be names, e.g. for the "test_enc_key" repository password name:
./bin/elasticsearch-keystore add repository.encrypted.test_enc_pass.passwordtype in the password
curl -X PUT "localhost:9200/_snapshot/test_enc?pretty" -H 'Content-Type: application/json' -d' { "type": "encrypted", "settings": { "location": "/tmp/repo/enc", "delegate_type": "fs", "password_name": "test_enc_pass" } } 'curl -X PUT "localhost:9200/_snapshot/test_enc/snapshot_1?wait_for_completion=true&pretty"Related: #49896 #41910 #50846 #48221 #65768