Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BACKPORT 7x Client-side encrypted snapshot repository (feature flag) (#66773) #66809

Conversation

albertzaharovits
Copy link
Contributor

Backport #66773

The client-side encrypted repository is a new type of snapshot repository that
internally delegates to the regular variants of snapshot repositories (of types
Azure, S3, GCS, FS, and maybe others but not yet tested). After the encrypted
repository is set up, it is transparent to the snapshot and restore APIs (i.e. all
snapshots stored in the encrypted repository are encrypted, no other parameters
required).
The encrypted repository is protected by a password stored on every node's
keystore (which must be the same across the nodes).
The password is used to generate a key encrytion key (KEK), using the PBKDF2
function, which is used to encrypt (using the AES Wrap algorithm) other
symmetric keys (referred to as DEK - data encryption keys), which themselves
are generated randomly, and which are ultimately used to encrypt the snapshot
blobs.

For example, here is how to set up an encrypted FS repository:

  1. make sure that the cluster runs under at least a "platinum" license
    (simplest test configuration is to put xpack.license.self_generated.type: "trial"
    in the elasticsearch.yml file)
  2. identical to the un-encrypted FS repository, specify the mount point of the
    shared FS in the elasticsearch.yml conf file (on all the cluster nodes),
    e.g. path.repo: ["/tmp/repo"]
  3. store the repository password inside the elasticsearch.keystore, on every cluster node.
    In order to support changing password on existing repository (implemented in a follow-up),
    the password itself must be names, e.g. for the "test_enc_key" repository password name:
    ./bin/elasticsearch-keystore add repository.encrypted.test_enc_pass.password
    type in the password
  4. start up the cluster and create the new encrypted FS repository, named "test_enc", by calling:
    curl -X PUT "localhost:9200/_snapshot/test_enc?pretty" -H 'Content-Type: application/json' -d' { "type": "encrypted", "settings": { "location": "/tmp/repo/enc", "delegate_type": "fs", "password_name": "test_enc_pass" } } '
  5. the snapshot and restore APIs work unmodified when they refer to this new repository, e.g.
    curl -X PUT "localhost:9200/_snapshot/test_enc/snapshot_1?wait_for_completion=true"

Related: #49896 #41910 #50846 #48221 #65768

The client-side encrypted repository is a new type of snapshot repository that
internally delegates to the regular variants of snapshot repositories (of types
Azure, S3, GCS, FS, and maybe others but not yet tested). After the encrypted
repository is set up, it is transparent to the snapshot and restore APIs (i.e. all
snapshots stored in the encrypted repository are encrypted, no other parameters
required).
The encrypted repository is protected by a password stored on every node's
keystore (which must be the same across the nodes).
The password is used to generate a key encrytion key (KEK), using the PBKDF2
function, which is used to encrypt (using the AES Wrap algorithm) other
symmetric keys (referred to as DEK - data encryption keys), which themselves
are generated randomly, and which are ultimately used to encrypt the snapshot
blobs.

For example, here is how to set up an encrypted  FS repository:
------
 1) make sure that the cluster runs under at least a "platinum" license
(simplest test configuration is to put `xpack.license.self_generated.type: "trial"`
in the elasticsearch.yml file)
 2) identical to the un-encrypted FS repository, specify the mount point of the
shared FS in the elasticsearch.yml conf file (on all the cluster nodes),
e.g. `path.repo: ["/tmp/repo"]`
 3) store the repository password inside the elasticsearch.keystore, *on every cluster node*.
In order to support changing password on existing repository (implemented in a follow-up),
the password itself must be names, e.g. for the "test_enc_key" repository password name:
`./bin/elasticsearch-keystore add repository.encrypted.test_enc_pass.password`
*type in the password*
4) start up the cluster and create the new encrypted FS repository, named "test_enc", by calling:
`
curl -X PUT "localhost:9200/_snapshot/test_enc?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "encrypted",
  "settings": {
    "location": "/tmp/repo/enc",
    "delegate_type": "fs",
    "password_name": "test_enc_pass"
  }
}
'
`
5) the snapshot and restore APIs work unmodified when they refer to this new repository, e.g.
` curl -X PUT "localhost:9200/_snapshot/test_enc/snapshot_1?wait_for_completion=true"`

Related: elastic#49896 elastic#41910 elastic#50846 elastic#48221 elastic#65768
@albertzaharovits albertzaharovits merged commit bbbfa24 into elastic:7.x Dec 28, 2020
@albertzaharovits albertzaharovits deleted the backport-7x-repository-encrypted-client-side branch December 28, 2020 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant