Draft: encryption at rest #3193

Some rudimentary description of encryption at rest, including full example to: * generate keys * enable encryption * change encryption There is much left to do. The section on the admin UI stores page requires cockroachdb/cockroach#26040

cockroach-teamcity · 2018-05-29T17:32:00Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/1f124cbb4363944e1c8b698586790610aa75cb2d/

cockroach-teamcity · 2018-05-29T17:36:48Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/d66cf0a498c58b89138108908c757edb0ef4740c/

)

cockroach-teamcity · 2018-05-29T17:46:27Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/b6a97a9ce3b746c00fb2774f4161a91138ad52a8/

rmloveland · 2018-05-29T17:55:17Z

_includes/sidebar-data-v2.1.json

@@ -330,6 +330,12 @@
                "urls": [
                  "/${VERSION}/roles.html"
                ]
+              },
+              {
+                "title": "Encryption At Rest",


It would be good to add "(Alpha)" after the page title (or the appropriate feature status if not alpha)

rmloveland · 2018-05-29T17:55:59Z

v2.1/encryption.md

@@ -0,0 +1,127 @@
+---
+title: Encryption At Rest


Same as previous comment - page title should end with "(Alpha)" or similar.

rmloveland · 2018-05-29T17:58:00Z

v2.1/encryption.md

+| Level | Description |
+|-|-|
+| Store keys | Provided by the user in a file. They are used to encrypt the list of data keys (see below).<br><br>This is known as a **key encryption key**: it's only purpose is to encrypt other keys.<br><br>Store keys are never persisted by CockroachDB.<br><br>Since very little data is encrypted using this key, it can have a very long lifetime without risk of reuse. |
+| Data keys | Automatically generated by CockroachDB. They are used to encrypt all files on disk.<br><br> This is known as a **data encryption key**.<br><br>Data keys are persisted in a key registry file, encrypted using the store key.<br><br>The key has a short lifetime to avoid key/IV reuse. |


Does IV mean this? https://en.wikipedia.org/wiki/Initialization_vector

If so I suggest that could be a link for non-wizards.

mberhault · 2018-05-29T18:18:22Z

Thanks for the comments Rich, but I just want to make sure we're on the same page: this is not even close to finished, and it's definitely not meant to be the final docs, more of a brain-dump on my part and as such will probably require some serious rewriting by people better at explaining than I am :)

nstewart · 2018-05-29T18:24:16Z

Yes, at this point we just need enough docs to unblock pm acceptance testing and eng qa

jseldess · 2018-05-29T18:26:16Z

@nstewart, do you still want some initial docs for next Monday's alpha release as well? Or is that no longer a priority?

nstewart · 2018-05-29T18:27:29Z

Sorry for the confusion -- just wanted to make sure that we weren't expecting Marc's output to be user-facing today. We want people to be able to use this by the time the alpha comes around for early feedback

rmloveland · 2018-05-29T18:50:56Z

Thanks for the comments Rich, but I just want to make sure we're on the same page: this is not even close to finished, and it's definitely not meant to be the final docs, more of a brain-dump on my part and as such will probably require some serious rewriting by people better at explaining than I am :)

I understand. Sorry to comment too early, I wasn't sure how we usually work on these draft PRs. I will stand back and wait for the go-ahead to review later or take over the PR when ready (or however it's done). :-)

mberhault · 2018-05-29T19:00:05Z

No worries. I'll let you know when I've done the majority of the page. Either today or tomorrow.

mberhault · 2018-05-30T16:46:36Z

The draft is pretty much ready. It will most likely need some serious improvements, but this should be a good start to understand how encryption at rest works, including examples and recommendations.

cockroach-teamcity · 2018-05-30T16:47:40Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/08113eb6fb7172ffd465a942a7ef71b5ba76e5f0/

rmloveland · 2018-05-30T20:38:13Z

Review status: 0 of 2 files reviewed at latest revision, 3 unresolved discussions, all commit checks successful.

v2.1/encryption.md, line 20 at r4 (raw file):

Encryption is performed in the [storage layer](architecture/storage-layer.html) and configured per store.
All files used by the store, regarless of contents, are encrypted with the desired algorithm.

s/regarless/regardless/

v2.1/encryption.md, line 43 at r4 (raw file):

* prevent key reuse with the same encryption parameters (after many files)

Markdown isn't rendering properly for this list. Maybe try an empty line before the list?

v2.1/encryption.md, line 50 at r4 (raw file):

Data keys will automatically be rotated at startup if any of the following conditions are met:
* the active store key changed

Same comment as above re: Markdown list rendering

v2.1/encryption.md, line 72 at r4 (raw file):

## Recommendations

There are a number of considerations to keep in mind when running with encryption:

s/:/./ since no list follows

v2.1/encryption.md, line 75 at r4 (raw file):

Key management is the most dangerous aspect of encryption. The following rules should be kept in mind:
* make sure only the unix user running the `cockroach` process has access to the keys

Markdown list render issue (same as above)

v2.1/encryption.md, line 81 at r4 (raw file):

A few other recommendations apply for best security practices:
* do not switch from encrypted to plaintext, this leaks data keys. Once transitioned to plaintext, all data must be considered reachable.

Markdown list render issue (same as above)

v2.1/encryption.md, line 168 at r4 (raw file):

key=/path/to/my/old-aes-128.key

do you mean old-key=/path/to/my/old-aes-128.key?

v2.1/encryption.md, line 173 at r4 (raw file):

## See Also

TODO(mberhault): links to external resources, report page details, flag descriptions.

Could replace this with something simple for now e.g.

- [Enterprise Licensing](enterprise-licensing.html)

Comments from Reviewable

rmloveland · 2018-05-30T20:51:04Z

@mberhault if you prefer, I can take over the PR and finish making edits

mberhault · 2018-05-30T21:21:53Z

Yes please, I'm deep into debugging an issue with encryption, I won't have time to look at this again today or tomorrow.

nstewart · 2018-05-31T13:25:00Z

v2.1/encryption.md, line 84 at r4 (raw file):

* do not copy the encrypted files as the data keys are not easily available.
* if encryption is desired, start a node with it enabled from first run without ever running in plaintext.

@mberhault -- we need to include a sentence or two for dealing with backups. What happens when I take a backup of a secure cluster? If I want to encrypt backups, what would be our recommendation? Just use encrypted s3? zip up the files and then encrypt that?

cc @rmloveland

Comments from Reviewable

mberhault · 2018-05-31T13:28:54Z

Review status: 0 of 2 files reviewed at latest revision, 12 unresolved discussions, all commit checks successful.

v2.1/encryption.md, line 84 at r4 (raw file):

Previously, nstewart (Nate) wrote…

@mberhault -- we need to include a sentence or two for dealing with backups. What happens when I take a backup of a secure cluster? If I want to encrypt backups, what would be our recommendation? Just use encrypted s3? zip up the files and then encrypt that?

cc @rmloveland

Backup (as in our distributed backup) still works as expected and does not encrypt anything. Encrypting the actual backups will be a completely different feature that likely reuses nothing from this. Ditto for import/export/dump.

Copying raw files from disk when encryption is turned on is now impossible (it's mentioned in the doc).

Comments from Reviewable

nstewart · 2018-05-31T13:36:11Z

v2.1/encryption.md, line 84 at r4 (raw file):

Previously, mberhault (marc) wrote…

Backup (as in our distributed backup) still works as expected and does not encrypt anything. Encrypting the actual backups will be a completely different feature that likely reuses nothing from this. Ditto for import/export/dump.

Copying raw files from disk when encryption is turned on is now impossible (it's mentioned in the doc).

Right -- I'm not talking about copying raw files, I'm talking about working with the files created by the distributed (unencrypted) backup. My question is, even if we don't have a feature for encrypting backups, what would be our recommendation for people that want to encrypt the backed up files?

Comments from Reviewable

mberhault · 2018-05-31T13:40:58Z

Review status: 0 of 2 files reviewed at latest revision, 12 unresolved discussions, all commit checks successful.

v2.1/encryption.md, line 84 at r4 (raw file):

Previously, nstewart (Nate) wrote…

Right -- I'm not talking about copying raw files, I'm talking about working with the files created by the distributed (unencrypted) backup. My question is, even if we don't have a feature for encrypting backups, what would be our recommendation for people that want to encrypt the backed up files?

It depends entirely on whether they trust the storage used as the backup destination. If they don't trust AWS/GCE/etc..., they'll need to encrypt the backups themselves. I don't think we want to provide more guidance than that though.

Comments from Reviewable

nstewart · 2018-05-31T13:43:43Z

v2.1/encryption.md, line 84 at r4 (raw file):

Previously, mberhault (marc) wrote…

It depends entirely on whether they trust the storage used as the backup destination. If they don't trust AWS/GCE/etc..., they'll need to encrypt the backups themselves. I don't think we want to provide more guidance than that though.

I think that's enough. Basically just saying: hey, backups/exports/etc aren't encrypted; you will have to encrypt those using your favorite encryption method.

cc @rmloveland

Comments from Reviewable

rmloveland · 2018-05-31T16:20:00Z

Review status: 0 of 2 files reviewed at latest revision, 12 unresolved discussions, all commit checks successful.

v2.1/encryption.md, line 84 at r4 (raw file):

I think that's enough. Basically just saying: hey, backups/exports/etc aren't encrypted; you will have to encrypt those using your favorite encryption method.

Just added a note to that effect. Thanks Nate!

Comments from Reviewable

cockroach-teamcity · 2018-05-31T16:24:30Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/891e113901daa9b6ff468ea063af9f98c19ae31b/

rmloveland · 2018-05-31T16:28:43Z

Review status: 0 of 2 files reviewed at latest revision, 12 unresolved discussions, all commit checks successful.

Comments from Reviewable

jseldess

Definitely LGTM for alpha docs, with some nits. Thanks, @rmloveland and @mberhault!

jseldess · 2018-05-31T21:19:00Z

v2.1/encryption.md

+| AES-192 | 192 bits (24 bytes) | 56 bytes |
+| AES-256 | 256 bits (32 bytes) | 64 bytes |
+
+Generating a key file can be done using the `cockroach` CLI::


Remove extra :

jseldess · 2018-05-31T21:20:51Z

v2.1/encryption.md

+$ cockroach start --store=cockroach-data --enterprise-encryption=path=cockroach-data,key=/path/to/my/aes-128.key,old-key=plain
+~~~
+
+**WARNING**: once specified for a given store, the `--enterprise-encryption` flag must always be present.


Let's convert this into a danger callout:

{{site.data.alerts.callout_danger}}Once specified for a given store, the <code>--enterprise-encryption</code> flag must always be present.{{site.data.alerts.end}}

jseldess · 2018-05-31T21:22:06Z

v2.1/encryption.md

+
+<div id="toc"></div>
+
+## Background


nit: We use Overview for sections like this in other docs.

jseldess · 2018-05-31T21:22:34Z

v2.1/encryption.md

+
+CockroachDB does not currently force re-encryption of older files but instead relies on normal RocksDB churn to slowly rewrite all data with the desired encryption.
+
+## Key rotation


nit: Rotating keys would make this parallel in form to Changing encryption type

cockroach-teamcity · 2018-05-31T21:52:41Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/f65f7673ab884425b0ebb10919ed08b959f76d9d/

jseldess requested a review from rmloveland May 25, 2018 14:20

mberhault mentioned this pull request May 29, 2018

security: encryption at rest cockroachdb/cockroach#19783

Closed

29 tasks

Draft: encryption at rest

20912bd

Some rudimentary description of encryption at rest, including full example to: * generate keys * enable encryption * change encryption There is much left to do. The section on the admin UI stores page requires cockroachdb/cockroach#26040

mberhault force-pushed the marc/draft_encryption_at_rest branch 2 times, most recently from 9a3ab6b to 1f124cb Compare May 29, 2018 17:30

Improve background section

d66cf0a

mberhault force-pushed the marc/draft_encryption_at_rest branch from 1f124cb to d66cf0a Compare May 29, 2018 17:34

Mention cockroach CLI to generate key (needs cockroachdb/cockroach#26167

b6a97a9

)

rmloveland reviewed May 29, 2018

View reviewed changes

Fill in key rotation and recommendations

08113eb

rmloveland added 2 commits May 31, 2018 12:17

Fix some typos and formatting issues

f1b7f7b

Note: backups are not encrypted by this feature

891e113

rmloveland requested a review from jseldess May 31, 2018 16:28

jseldess approved these changes May 31, 2018

View reviewed changes

Update based on reviewer feedback

f65f767

rmloveland merged commit dec77b3 into master May 31, 2018

rmloveland deleted the marc/draft_encryption_at_rest branch May 31, 2018 21:53


		CockroachDB does not currently force re-encryption of older files but instead relies on normal RocksDB churn to slowly rewrite all data with the desired encryption.

		## Key rotation

Draft: encryption at rest #3193

Draft: encryption at rest #3193

Conversation

mberhault commented May 25, 2018

cockroach-teamcity commented May 25, 2018

cockroach-teamcity commented May 25, 2018

mberhault commented May 25, 2018

mberhault commented May 25, 2018

mberhault commented May 29, 2018

mberhault commented May 29, 2018

cockroach-teamcity commented May 29, 2018

cockroach-teamcity commented May 29, 2018

cockroach-teamcity commented May 29, 2018

cockroach-teamcity commented May 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mberhault commented May 29, 2018

nstewart commented May 29, 2018

jseldess commented May 29, 2018

nstewart commented May 29, 2018 • edited Loading

rmloveland commented May 29, 2018

mberhault commented May 29, 2018

mberhault commented May 30, 2018

cockroach-teamcity commented May 30, 2018

rmloveland commented May 30, 2018

rmloveland commented May 30, 2018

mberhault commented May 30, 2018

nstewart commented May 31, 2018

mberhault commented May 31, 2018

nstewart commented May 31, 2018

mberhault commented May 31, 2018

nstewart commented May 31, 2018

rmloveland commented May 31, 2018

cockroach-teamcity commented May 31, 2018

rmloveland commented May 31, 2018

jseldess left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cockroach-teamcity commented May 31, 2018

nstewart commented May 29, 2018 •

edited

Loading