BACKPORT 7x Client-side encrypted snapshot repository (feature flag) (#66773) #66809
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport #66773
The client-side encrypted repository is a new type of snapshot repository that
internally delegates to the regular variants of snapshot repositories (of types
Azure, S3, GCS, FS, and maybe others but not yet tested). After the encrypted
repository is set up, it is transparent to the snapshot and restore APIs (i.e. all
snapshots stored in the encrypted repository are encrypted, no other parameters
required).
The encrypted repository is protected by a password stored on every node's
keystore (which must be the same across the nodes).
The password is used to generate a key encrytion key (KEK), using the PBKDF2
function, which is used to encrypt (using the AES Wrap algorithm) other
symmetric keys (referred to as DEK - data encryption keys), which themselves
are generated randomly, and which are ultimately used to encrypt the snapshot
blobs.
For example, here is how to set up an encrypted FS repository:
(simplest test configuration is to put
xpack.license.self_generated.type: "trial"
in the elasticsearch.yml file)
shared FS in the elasticsearch.yml conf file (on all the cluster nodes),
e.g.
path.repo: ["/tmp/repo"]
In order to support changing password on existing repository (implemented in a follow-up),
the password itself must be names, e.g. for the "test_enc_key" repository password name:
./bin/elasticsearch-keystore add repository.encrypted.test_enc_pass.password
type in the password
curl -X PUT "localhost:9200/_snapshot/test_enc?pretty" -H 'Content-Type: application/json' -d' { "type": "encrypted", "settings": { "location": "/tmp/repo/enc", "delegate_type": "fs", "password_name": "test_enc_pass" } } '
curl -X PUT "localhost:9200/_snapshot/test_enc/snapshot_1?wait_for_completion=true"
Related: #49896 #41910 #50846 #48221 #65768