ZFS Encryption #4329

tcaputi · Feb 11, 2016

Native encryption in zfsonlinux (See issue #494)

The change incorporates 2 major pieces:

The first feature is a keystore that manages wrapping and encryption keys for encrypted datasets. The commands are similar to that of Solaris but with a few key enhancements to make it more predictable, more consistent, and require less manual maintenance. It is fully integrated with the existing zfs create functions and zfs clone functions. It also exposes a new set of commands via zfs key for managing the keystore. For more info on the issues with the Solaris implementation see my comments here and here. The keystore operates on a few rules.

All wrapping keys are 32 bytes (256 bits), even for 128 and 192 bit encryption types.
Encryption must be specified at dataset creation time.
Specifying a keysource while creating a dataset causes the dataset to become the root of an encryption tree.
All members of an encryption tree share the same wrapping key.
Each dataset can have up to 1 keychain (if it is encrypted) that is not shared with anybody.

The second feature is the actual data and metadata encryption. All user data in an encrypted dataset is stored encrypted on-disk. User-provided metadata is also encrypted, but metadata structures have been left plain so that scrubbing and resilvering still works without the keys loaded. The design was originallly inspired by this article but has been changed fairly significantly since.

Implementation details that should be looked at

Encrypting data going to disk requires creating a key_mapping_t during dsl_dataset_tryown(). I added a flag to this function for code that wishes to own the dataset, but that does not require encrypted data, such as the scrub functions. I did my best to confirm that all owners set this flag correctly, but someone should confirm them, just to be sure.
zfs send and zfs recv do not currently do anything special with regards to encryption. The format of the send file has not changed and zfs send requires the keys to be loaded in order to work. At some point there should probably be a way to do raw sends.
I altered the prototype of lzc_create() and lzc_clone() to support hidden arguments. I understand that the purpose of libzfs_core is to have a stable api interacting with the ZFS ioctls. However, these functions need to accept wrapping keys separately from the rest of their parameters because they need to use the (new) hidden_args framework to support hiding arguments from the logs. Without this, the wrapping keys would get printed to the zpool history.

EDIT 5/4/16: Updated to reflect the current state of the PR
EDIT 1/3/17: Updated to reflect the current state of the PR

I see my builds all failed because I left in a few bad asserts and I didn't have debugging enabled so they didn't cause an issue on my end. Oops. I will fix this tonight.

tcaputi · Feb 12, 2016

I see my builds all failed because I left in a few bad asserts and I didn't have debugging enabled so they didn't cause an issue on my end. Oops. I will fix this tonight.

I also see that the style issues are causing the builds to fail. I will fix that tonight as well.

tcaputi · Feb 12, 2016

I also see that the style issues are causing the builds to fail. I will fix that tonight as well.

Does none of this belong in spl?

ilovezfs · Feb 12, 2016

Does none of this belong in spl?

All of the code in module/icp (with the exception of the algs directory which has specific cipher implementations) acts as a single frameowrk and should be kept together. Personally, I view it as a standalone module that relies on the spl which is why I put it where it is. To me it seemed like it was along the same lines as the nvpair module. I can move it if the community feels there is a better place for it.

tcaputi · Feb 12, 2016

All of the code in module/icp (with the exception of the algs directory which has specific cipher implementations) acts as a single frameowrk and should be kept together. Personally, I view it as a standalone module that relies on the spl which is why I put it where it is. To me it seemed like it was along the same lines as the nvpair module. I can move it if the community feels there is a better place for it.

@ilovezfs Putting any of this into the SPL would require implementing new encryption routines because the Linux kernel GPL symbol exports its encryption routines and the SPL is not allowed to touch that.

ryao · Feb 13, 2016

@ilovezfs Putting any of this into the SPL would require implementing new encryption routines because the Linux kernel GPL symbol exports its encryption routines and the SPL is not allowed to touch that.

@ryao Then perhaps SPL needs to be renamed LPL "License Porting Layer" since a piece of Solaris porting cannot go in the SPL. Ahem.

ilovezfs · Feb 13, 2016

@ryao Then perhaps SPL needs to be renamed LPL "License Porting Layer" since a piece of Solaris porting cannot go in the SPL. Ahem.

@ryao Thank you for mentioning that. I had forgotten about the licensing aspect.

tcaputi · Feb 13, 2016

@ryao Thank you for mentioning that. I had forgotten about the licensing aspect.

does anyone know how to re-trigger the automated build tests? The issues from above should have been fixed by my reformatting commit, but I'd like to be sure.

tcaputi · Feb 16, 2016

does anyone know how to re-trigger the automated build tests? The issues from above should have been fixed by my reformatting commit, but I'd like to be sure.

@tcaputi if you force update your branch on Github it will rerun all the builds and tests. Usually, I just rebase my work on master and then force update it.

As for putting this work in the SPL I'd prefer to keep in the ZFS source tree since it's all CDDL and originated from illumos. The SPL itself will in due course get moved in to a directory of the ZFS source tree for convenience so let's try and avoid adding more to it if we don't have too.

behlendorf · Feb 17, 2016

@tcaputi if you force update your branch on Github it will rerun all the builds and tests. Usually, I just rebase my work on master and then force update it.

As for putting this work in the SPL I'd prefer to keep in the ZFS source tree since it's all CDDL and originated from illumos. The SPL itself will in due course get moved in to a directory of the ZFS source tree for convenience so let's try and avoid adding more to it if we don't have too.

@behlendorf It doesn't look like the tests ran. I did a merge against upstream/master and then git push -f origin master to push to my master branch. Is this correct?

tcaputi · Feb 18, 2016

@behlendorf It doesn't look like the tests ran. I did a merge against upstream/master and then git push -f origin master to push to my master branch. Is this correct?

@tcaputi: You need to rebase instead of merge, so assuming your index and working tree are clean:

git reset --hard f702c83
git fetch upstream
git rebase -i upstream/master
git push -f origin master

l1k · Feb 18, 2016

@tcaputi: You need to rebase instead of merge, so assuming your index and working tree are clean:

git reset --hard f702c83
git fetch upstream
git rebase -i upstream/master
git push -f origin master

I looked at the logs here. The builds are failing now because zfs requires the (few) changes I made to the SPL. I will make a PR for that too, but I suppose this won't work on the automated builds until then (which will probably take a little while)

tcaputi · Feb 18, 2016

I looked at the logs here. The builds are failing now because zfs requires the (few) changes I made to the SPL. I will make a PR for that too, but I suppose this won't work on the automated builds until then (which will probably take a little while)

I made a (small) corresponding PR against the spl: zfsonlinux/spl#533

tcaputi · Feb 18, 2016

I made a (small) corresponding PR against the spl: zfsonlinux/spl#533

@tcaputi if you add the line Requires-spl: refs/pull/533/head to commit message of the top patch in this stack then the builtbot will use that PR instead of master when checking out the spl for testing.

behlendorf · Feb 19, 2016

@tcaputi if you add the line Requires-spl: refs/pull/533/head to commit message of the top patch in this stack then the builtbot will use that PR instead of master when checking out the spl for testing.

@behlendorf I will do that and fix the SPL PR tonight. Thanks for the advice and patience.

tcaputi · Feb 19, 2016

@behlendorf I will do that and fix the SPL PR tonight. Thanks for the advice and patience.

I see that there are a few problems on non x86_64 architectures. I will fix these soon, but I don't have any local machines to test against. I hope its ok if I end up hitting the build system a few more times for testing.

tcaputi · Feb 21, 2016

I see that there are a few problems on non x86_64 architectures. I will fix these soon, but I don't have any local machines to test against. I hope its ok if I end up hitting the build system a few more times for testing.

@tcaputi by all means keep submitting things until the buildbot is happy. That's what it's there for.

behlendorf · Feb 25, 2016

@tcaputi by all means keep submitting things until the buildbot is happy. That's what it's there for.

ryao · Feb 26, 2016

ryao reviewed Feb 26, 2016

View changes

include/sys/zio_crypt.h

@ryao Ok. Thanks. I just copies the license from an existing file. I'lll make the change.

tcaputi · Feb 26, 2016

@ryao Ok. Thanks. I just copies the license from an existing file. I'lll make the change.

@ryao fixed licensing in the latest commit.

tcaputi · Mar 7, 2016

@ryao fixed licensing in the latest commit.

Unconstructive comment: I cannot express how exciting this is!

sammcj · May 15, 2016

Unconstructive comment: I cannot express how exciting this is!

@tcaputi This have been out for a few months now, how does it look? Does it work, is there any got-ya's, risks etc?

I think that all boils down to is: Would you trust it with production data?

FransUrbo · May 16, 2016

@tcaputi This have been out for a few months now, how does it look? Does it work, is there any got-ya's, risks etc?

I think that all boils down to is: Would you trust it with production data?

@FransUrbo

This have been out for a few months now, how does it look?

Well, this PR was a WIP for a few months now. I just changed it to a real PR a few days ago. That said, I have not been able to produce a scenario where I have lost data in quite some time. I have a lot of confidence in the encryption code because I stole all of it from Illumos, and after that the problem of not corrupting data is actually pretty simple, since really you just need to make sure you're not losing track of any of the encryption parameters.

I have looked into the 2 buildbot failures that are here (I don't know why github is showing 2 runs as incomplete when they succeeded). I was able to reproduce all of them using ZoL master. Of course, this makes me a little nervous about my code, since the tests are still failing and I'm not sure if I have introduced any new bugs.

The only real gotcha is that performance with many files may be a bit reduced because I force spill zfs to use spill blocks instead of bonus buffers for DMU_OT_PLAIN_FILE_CONTENTS. That should be it, as far as I'm aware.

I think that all boils down to is: Would you trust it with production data?

All of that said, for a project as widely used as zfs, I would not trust ANY code with production data until it has at least been reviewed by somebody else from the project, so I would hold off on using it for anything but testing until then. Today I plan to finish integrating my tests into the zfs-test framework and then I will post a message to the mailing list asking the wider OpenZFS community for review.

tcaputi · May 16, 2016

@FransUrbo

This have been out for a few months now, how does it look?

Well, this PR was a WIP for a few months now. I just changed it to a real PR a few days ago. That said, I have not been able to produce a scenario where I have lost data in quite some time. I have a lot of confidence in the encryption code because I stole all of it from Illumos, and after that the problem of not corrupting data is actually pretty simple, since really you just need to make sure you're not losing track of any of the encryption parameters.

I have looked into the 2 buildbot failures that are here (I don't know why github is showing 2 runs as incomplete when they succeeded). I was able to reproduce all of them using ZoL master. Of course, this makes me a little nervous about my code, since the tests are still failing and I'm not sure if I have introduced any new bugs.

The only real gotcha is that performance with many files may be a bit reduced because I force spill zfs to use spill blocks instead of bonus buffers for DMU_OT_PLAIN_FILE_CONTENTS. That should be it, as far as I'm aware.

I think that all boils down to is: Would you trust it with production data?

All of that said, for a project as widely used as zfs, I would not trust ANY code with production data until it has at least been reviewed by somebody else from the project, so I would hold off on using it for anything but testing until then. Today I plan to finish integrating my tests into the zfs-test framework and then I will post a message to the mailing list asking the wider OpenZFS community for review.

@tcaputi Sounds good, thanx. I'll setup some test machine with this then.

FransUrbo · May 16, 2016

@tcaputi Sounds good, thanx. I'll setup some test machine with this then.

It did occur to me that if it's at all possible, could you do TWO PRs instead of one big one?

As in, PR1 which adds all the fundamentals (commit1 with the direct port of the Illumos code, without any changes, commit2 with the porting stuff) and then PR2 which enables the whole thing?

That way it's easier to review and if it's easier, there's a bigger likelihood to get it accepted [this year :)]. And since it doesn't DO anything, it's safe to add into master.

Then everyone that want to test it, just apply PR2 to enable the whole thing. Also, since PR2 depends on PR1, PR2 will contain the commit(s) of PR1, so before it's accepted, one could just apply PR2 to test..

FransUrbo · May 16, 2016

It did occur to me that if it's at all possible, could you do TWO PRs instead of one big one?

As in, PR1 which adds all the fundamentals (commit1 with the direct port of the Illumos code, without any changes, commit2 with the porting stuff) and then PR2 which enables the whole thing?

That way it's easier to review and if it's easier, there's a bigger likelihood to get it accepted [this year :)]. And since it doesn't DO anything, it's safe to add into master.

Then everyone that want to test it, just apply PR2 to enable the whole thing. Also, since PR2 depends on PR1, PR2 will contain the commit(s) of PR1, so before it's accepted, one could just apply PR2 to test..

@FransUrbo I did the PR as 2 commits at the request of @behlendorf. I would be fine splitting it if that is what everyone wants. Just to clarify, you would just want me to split the PR so that commit 1 is one PR, and commit 2 is another?

tcaputi · May 16, 2016

@FransUrbo I did the PR as 2 commits at the request of @behlendorf. I would be fine splitting it if that is what everyone wants. Just to clarify, you would just want me to split the PR so that commit 1 is one PR, and commit 2 is another?

@tcaputi I was thinking THREE commits:

Full Illumos code, no changes
Porting of the Illumos code
Enabling of native crypto (code to use it in zfs create etc).

Commit one and two in one PR and commit three in a second. But maybe get @behlendorf opinion on that? @behlendorf, wouldn't that help making your life easier as well? If we could get the core in faster, wouldn't that be better for everyone?

We all know that 10k+ lines of PR isn't going to be looked at any time soon :)

FransUrbo · May 16, 2016

@tcaputi I was thinking THREE commits:

Full Illumos code, no changes
Porting of the Illumos code
Enabling of native crypto (code to use it in zfs create etc).

Commit one and two in one PR and commit three in a second. But maybe get @behlendorf opinion on that? @behlendorf, wouldn't that help making your life easier as well? If we could get the core in faster, wouldn't that be better for everyone?

We all know that 10k+ lines of PR isn't going to be looked at any time soon :)

PS. The whole point in doing it this way is that the majority of the code can be accepted without any risk - it's never used/accessed! So technically, @behlendorf only need to review point/commit two this time and once it's been in there "X number of months/weeks", he/we can start looking at commit three without feeling stressed.

At work they're talking about "feature flags" (although that would probably clash with Z filesystem flags :) - write as much of new stuff as possible, but keep it disabled. Then enable it slowly on selected machines.

FransUrbo · May 16, 2016

PS. The whole point in doing it this way is that the majority of the code can be accepted without any risk - it's never used/accessed! So technically, @behlendorf only need to review point/commit two this time and once it's been in there "X number of months/weeks", he/we can start looking at commit three without feeling stressed.

At work they're talking about "feature flags" (although that would probably clash with Z filesystem flags :) - write as much of new stuff as possible, but keep it disabled. Then enable it slowly on selected machines.

@FransUrbo Unfortunately, when I created the ICP I did not start with the entirety of the Illumos Crypto code since it is pretty tightly coupled to the Illumos kernel. Most of what I did involved copying in the API functions I needed, getting them to compile (with linking errors) and then resolving dependencies. As a result, many files are almost perfectly identical to Illumos, but there are quite a few that are missing significant pieces since they were not needed by the functions I ported.

In order to make PR 1 look as you described, I would need to go back and do a lot of extra work to re-port everything starting with the full ICP. I'm also not sure it would be nicer to look at since there would be literally thousands of compilation errors in the first commit.

I agree, though, that it might make sense to have a separate PR for the ICP since it is functionally complete, but doesn't do anything on its own.

tcaputi · May 16, 2016

@FransUrbo Unfortunately, when I created the ICP I did not start with the entirety of the Illumos Crypto code since it is pretty tightly coupled to the Illumos kernel. Most of what I did involved copying in the API functions I needed, getting them to compile (with linking errors) and then resolving dependencies. As a result, many files are almost perfectly identical to Illumos, but there are quite a few that are missing significant pieces since they were not needed by the functions I ported.

In order to make PR 1 look as you described, I would need to go back and do a lot of extra work to re-port everything starting with the full ICP. I'm also not sure it would be nicer to look at since there would be literally thousands of compilation errors in the first commit.

I agree, though, that it might make sense to have a separate PR for the ICP since it is functionally complete, but doesn't do anything on its own.

To explain what I mean, taking a little part of one of the files in one of the commits:

diff --git a/cmd/zfs/zfs_main.c b/cmd/zfs/zfs_main.c
index 7525afc..37dab84 100644
--- a/cmd/zfs/zfs_main.c
+++ b/cmd/zfs/zfs_main.c
@@ -103,6 +103,7 @@ static int zfs_do_holds(int argc, char **argv);  <- This you add to PR1/commit1
 static int zfs_do_release(int argc, char **argv);
 static int zfs_do_diff(int argc, char **argv);
 static int zfs_do_bookmark(int argc, char **argv);
+static int zfs_do_crypto(int argc, char **argv);

 /*
  * Enable a reasonable set of defaults for libumem debugging on DEBUG builds.
@@ -150,6 +151,7 @@ typedef enum {  <- This you add to PR2/commit1
        HELP_RELEASE,
        HELP_DIFF,
        HELP_BOOKMARK,
+       HELP_CRYPTO,
 } zfs_help_t;

 typedef struct zfs_command {
@@ -203,6 +205,7 @@ static zfs_command_t command_table[] = {  <- This you add to PR2/commit1
        { "holds",      zfs_do_holds,           HELP_HOLDS              },
        { "release",    zfs_do_release,         HELP_RELEASE            },
        { "diff",       zfs_do_diff,            HELP_DIFF               },
+       { "key",        zfs_do_crypto,          HELP_CRYPTO             },
 };

 #define        NCOMMAND        (sizeof (command_table) / sizeof (command_table[0]))
@@ -319,6 +322,9 @@ get_usage(zfs_help_t idx)  <- This you add to PR2/commit1
                    "[snapshot|filesystem]\n"));
        case HELP_BOOKMARK:
                return (gettext("\tbookmark <snapshot> <bookmark>\n"));
+       case HELP_CRYPTO:
+               return (gettext("\tkey [-luK] <filesystem|volume>\n"
+                   "\tkey -c [-o keysource=value] <filesystem|volume>\n"));
        }

        abort();
@@ -640,7 +646,7 @@ static int  <- This you add to PR1/commit1
 zfs_do_clone(int argc, char **argv)
 {
        zfs_handle_t *zhp = NULL;
-       boolean_t parents = B_FALSE;
+       boolean_t parents = B_FALSE, add_key = B_FALSE;
        nvlist_t *props;
        int ret = 0;
        int c;
@@ -649,7 +655,7 @@ zfs_do_clone(int argc, char **argv)  <- This you add to PR2/commit1
                nomem();

        /* check options */
-       while ((c = getopt(argc, argv, "o:p")) != -1) {
+       while ((c = getopt(argc, argv, "o:pK")) != -1) {
                switch (c) {
                case 'o':
                        if (parseprop(props, optarg) != 0)
@@ -658,6 +664,9 @@ zfs_do_clone(int argc, char **argv)  <- This you add to PR1/commit1
                case 'p':
                        parents = B_TRUE;
                        break;
+               case 'K':
+                       add_key = B_TRUE;
+                       break;
                case '?':
                        (void) fprintf(stderr, gettext("invalid option '%c'\n"),
                            optopt);
@@ -703,7 +712,7 @@ zfs_do_clone(int argc, char **argv)  <- This you add to PR1/commit1
        }

        /* pass to libzfs */
-       ret = zfs_clone(zhp, argv[1], props);
+       ret = zfs_clone(zhp, argv[1], props, add_key);

        /* create the mountpoint if necessary */
        if (ret == 0) {

You see? That way, PR1/commit1 can be added/accepted without any/little review (because it's already been reviewed by Illumos) and PR2/commit1 can be reviewed at "a later date".

FransUrbo · May 16, 2016

To explain what I mean, taking a little part of one of the files in one of the commits:

diff --git a/cmd/zfs/zfs_main.c b/cmd/zfs/zfs_main.c
index 7525afc..37dab84 100644
--- a/cmd/zfs/zfs_main.c
+++ b/cmd/zfs/zfs_main.c
@@ -103,6 +103,7 @@ static int zfs_do_holds(int argc, char **argv);  <- This you add to PR1/commit1
 static int zfs_do_release(int argc, char **argv);
 static int zfs_do_diff(int argc, char **argv);
 static int zfs_do_bookmark(int argc, char **argv);
+static int zfs_do_crypto(int argc, char **argv);

 /*
  * Enable a reasonable set of defaults for libumem debugging on DEBUG builds.
@@ -150,6 +151,7 @@ typedef enum {  <- This you add to PR2/commit1
        HELP_RELEASE,
        HELP_DIFF,
        HELP_BOOKMARK,
+       HELP_CRYPTO,
 } zfs_help_t;

 typedef struct zfs_command {
@@ -203,6 +205,7 @@ static zfs_command_t command_table[] = {  <- This you add to PR2/commit1
        { "holds",      zfs_do_holds,           HELP_HOLDS              },
        { "release",    zfs_do_release,         HELP_RELEASE            },
        { "diff",       zfs_do_diff,            HELP_DIFF               },
+       { "key",        zfs_do_crypto,          HELP_CRYPTO             },
 };

 #define        NCOMMAND        (sizeof (command_table) / sizeof (command_table[0]))
@@ -319,6 +322,9 @@ get_usage(zfs_help_t idx)  <- This you add to PR2/commit1
                    "[snapshot|filesystem]\n"));
        case HELP_BOOKMARK:
                return (gettext("\tbookmark <snapshot> <bookmark>\n"));
+       case HELP_CRYPTO:
+               return (gettext("\tkey [-luK] <filesystem|volume>\n"
+                   "\tkey -c [-o keysource=value] <filesystem|volume>\n"));
        }

        abort();
@@ -640,7 +646,7 @@ static int  <- This you add to PR1/commit1
 zfs_do_clone(int argc, char **argv)
 {
        zfs_handle_t *zhp = NULL;
-       boolean_t parents = B_FALSE;
+       boolean_t parents = B_FALSE, add_key = B_FALSE;
        nvlist_t *props;
        int ret = 0;
        int c;
@@ -649,7 +655,7 @@ zfs_do_clone(int argc, char **argv)  <- This you add to PR2/commit1
                nomem();

        /* check options */
-       while ((c = getopt(argc, argv, "o:p")) != -1) {
+       while ((c = getopt(argc, argv, "o:pK")) != -1) {
                switch (c) {
                case 'o':
                        if (parseprop(props, optarg) != 0)
@@ -658,6 +664,9 @@ zfs_do_clone(int argc, char **argv)  <- This you add to PR1/commit1
                case 'p':
                        parents = B_TRUE;
                        break;
+               case 'K':
+                       add_key = B_TRUE;
+                       break;
                case '?':
                        (void) fprintf(stderr, gettext("invalid option '%c'\n"),
                            optopt);
@@ -703,7 +712,7 @@ zfs_do_clone(int argc, char **argv)  <- This you add to PR1/commit1
        }

        /* pass to libzfs */
-       ret = zfs_clone(zhp, argv[1], props);
+       ret = zfs_clone(zhp, argv[1], props, add_key);

        /* create the mountpoint if necessary */
        if (ret == 0) {

You see? That way, PR1/commit1 can be added/accepted without any/little review (because it's already been reviewed by Illumos) and PR2/commit1 can be reviewed at "a later date".

@tcaputi Personally, I think PR1/commit1 and all it's compile problems is not a problem. That code have already been reviewed and should be ok generally. It's PR2/commit2 that's "The Magic (™)" and that's what really needs to be reviewed. Yes, they both need to be accepted "in bulk", but reviewing only the stuff that makes it compile should be less than 10k lines :).

And I know that would be a lot of work, but it should be faster second time around :). And it's only if you feel like spending a day or so doing it. Maybe Brian won't have a problem with it as-is, in which case my opinion is moot.

The really "hard core" thing is PR2/commit1. That's when all the really cool things starts to happen for users :D. But I rather have all the functionality in there, but disabled/unusable, for a couple of months to see if we run into some problem than have the whole shebang, 10k lines of code, accepted in one go..

FransUrbo · May 16, 2016

@tcaputi Personally, I think PR1/commit1 and all it's compile problems is not a problem. That code have already been reviewed and should be ok generally. It's PR2/commit2 that's "The Magic (™)" and that's what really needs to be reviewed. Yes, they both need to be accepted "in bulk", but reviewing only the stuff that makes it compile should be less than 10k lines :).

And I know that would be a lot of work, but it should be faster second time around :). And it's only if you feel like spending a day or so doing it. Maybe Brian won't have a problem with it as-is, in which case my opinion is moot.

The really "hard core" thing is PR2/commit1. That's when all the really cool things starts to happen for users :D. But I rather have all the functionality in there, but disabled/unusable, for a couple of months to see if we run into some problem than have the whole shebang, 10k lines of code, accepted in one go..

Also, doing the Illumos + porting in two commits helps with "traceability" - what part is ours and what part is theirs..

FransUrbo · May 16, 2016

Also, doing the Illumos + porting in two commits helps with "traceability" - what part is ours and what part is theirs..

@tcaputi thanks for splitting this in to two commits, that'll make it easier to review and merge in chunks. While splitting it in to three as @FransUrbo suggested would make it easier to review, in this case I don't think it's worth it for a couple of reasons.

Based on past experience, when bringing over large chunks of Illumos code you must have made a ton of trivial changes just to get it to compile. Those changes should all be evident and easily understandable when diff'ing against the original Illumos implementation. This is something I'm going to have to do anyway during the review so splitting them out likely isn't worth the effort.
Merging just the original Illumos code would result in a ZoL commit which doesn't build and can't be tested. I'd prefer to avoid that and always keep the tree in a buildable state. My story here would be different if we were starting with a new git tree where nothing currently worked.

I do think it would be worthwhile to include in the commit for the crypto framework a comment describing in general what kinds of changes needed to be made. Where the changes all trivial C90 fixes? Were there changes to address gcc warnings? Did you identify and fix any real bugs? What large code chucks were dropped? etc.

I agree there's a good chance we can merge the crypto framework first without the additional key store implementation. As for splitting it in to different PRs I'd actually prefer to keep it in one. This way once test cases are added and automated testing runs it will actually exercise that new code. Splitting it in to its own PR logically makes sense but we'd give up the automated testing.

@tcaputi I've resubmitted those 4 failed/pending builders for another test run. I suspect those two never logged success because for some reason those two status updates were dropped when getting posted to Github. It seems to happen on occasion.

behlendorf · May 16, 2016

@tcaputi thanks for splitting this in to two commits, that'll make it easier to review and merge in chunks. While splitting it in to three as @FransUrbo suggested would make it easier to review, in this case I don't think it's worth it for a couple of reasons.

Based on past experience, when bringing over large chunks of Illumos code you must have made a ton of trivial changes just to get it to compile. Those changes should all be evident and easily understandable when diff'ing against the original Illumos implementation. This is something I'm going to have to do anyway during the review so splitting them out likely isn't worth the effort.
Merging just the original Illumos code would result in a ZoL commit which doesn't build and can't be tested. I'd prefer to avoid that and always keep the tree in a buildable state. My story here would be different if we were starting with a new git tree where nothing currently worked.

I do think it would be worthwhile to include in the commit for the crypto framework a comment describing in general what kinds of changes needed to be made. Where the changes all trivial C90 fixes? Were there changes to address gcc warnings? Did you identify and fix any real bugs? What large code chucks were dropped? etc.

I agree there's a good chance we can merge the crypto framework first without the additional key store implementation. As for splitting it in to different PRs I'd actually prefer to keep it in one. This way once test cases are added and automated testing runs it will actually exercise that new code. Splitting it in to its own PR logically makes sense but we'd give up the automated testing.

@tcaputi I've resubmitted those 4 failed/pending builders for another test run. I suspect those two never logged success because for some reason those two status updates were dropped when getting posted to Github. It seems to happen on occasion.

Rats, lost the commit history with rebase, guess I have to report the code again :)

lundman · May 17, 2016

Rats, lost the commit history with rebase, guess I have to report the code again :)

Pawel: encrypted send without loaded keys is already in place, so is scrub. @tcaputi did his and everyone else's homework impeccably in terms of use case requirements. My understanding is we are currently held up by the mostly insurmountable (for open source endeavors) question of whether the on disk/sent structures expose something which may reduce cryptographic strength in offline attacks (today or tomorrow). So long as known vectors such as derivation of state or materiel from cleartext data can be verified as addressed by someone with accredited heavy crypto expertise in storage implementation, we should be good to roll. In ZoL, it works (usability-wise) very well, and apparently faster than using our Linux-native dm-crypt block layer under the vdevs. Given that I don't see BSD systems on anything but zfs anymore, I wonder what GELIs place will be after the port... @tcaputi: could you possibly share the validation plan/actors involved? It may be worth while as a community to scrape together some funds for a proper audit by a well known vendor (or ask them to do it for community PR/exposure - thats a marketable accomplishment) along with some tooling for the test suite to ensure we don't start leaking sensitive content from future additions made by others alongside your code. @behlendorf: can you rope in some of the USG crypto folks? Natsec relevant data will end up being stored this way... While I'm not suggesting exposure of audit methods by orgs who "don't do that," a comprehensive review with "watch for these bits changing moving forward" notes would be a godsend since the on disk format is subject to change down the line, and we want to avoid the FIPS certification nightmare as much as we can since someone, somewhere, will need to get it to use this for work. Far as getting bandwidth for review - tell them that the Russians have already performed several reviews (if they didn't yet, they will now) and probably started building tooling around anything found, so its a tactical boon to cut them off at the pass, and strategic sanity to use the scarce resources (cryptanalysts) for future proofing something as critical as storage crypto for the most advanced fs on earth used across most POSIX systems in play. Let's not do bitlocker all over again in the federal/mil space. @infosec-folks: if this is your area of expertise or you have a graybeard in your shop who can perform ffts and sha ops in their head, please speak up. Without validation this is hard to adopt, and without adoption, real world testing is not going to happen. Spread the word about this work and try to get hands/eyes on it. ZFS has crypto, we now need to ensure its life cycle as a first order member of the functional stack.

…

-------- Original Message -------- From: Pawel Jakub Dawidek <notifications@github.com> Sent: Monday, December 26, 2016 01:41 AM To: zfsonlinux/zfs <zfs@noreply.github.com> Subject: Re: [zfsonlinux/zfs] ZFS Encryption (#4329) CC: RageLtMan <rageltman@sempervictus.com>,Mention <mention@noreply.github.com>

We (at Wheel Systems) will probably work on porting ZFS encryption to FreeBSD, unless someone will beat us to it. For us the most important feature is encrypted zfs send, so we may wait until this is done. Still, there are many active committers working on ZFS in FreeBSD, so I'd expect this is going to happen soon. -- Paweł Jakub Dawidek > On 24 Dec 2016, at 08:06, Graham Perrin ***@***.***> wrote: > > … anyone from BSD-land watching/porting this? … > > @sempervictus I watch with great interest, but I'm not a developer. > > Extending the request for comment, to FreeBSD communities > > freebsd-hackers > > Multi-platform ZFS encryption (ZoL pull #4329 by Tom Caputi): RFC > TrueOS > > In the community lounge, under Approaches to encryption with ZFS: > > https://discourse.trueos.org/t/-/234/4?u=grahamperrin > (Recent efforts are necessarily focused on the transition to OpenRC, so I should not expect a response in the ZFS topic until some time in January.) > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or mute the thread. > -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #4329 (comment)

sempervictus · Dec 26, 2016

Pawel: encrypted send without loaded keys is already in place, so is scrub. @tcaputi did his and everyone else's homework impeccably in terms of use case requirements. My understanding is we are currently held up by the mostly insurmountable (for open source endeavors) question of whether the on disk/sent structures expose something which may reduce cryptographic strength in offline attacks (today or tomorrow). So long as known vectors such as derivation of state or materiel from cleartext data can be verified as addressed by someone with accredited heavy crypto expertise in storage implementation, we should be good to roll. In ZoL, it works (usability-wise) very well, and apparently faster than using our Linux-native dm-crypt block layer under the vdevs. Given that I don't see BSD systems on anything but zfs anymore, I wonder what GELIs place will be after the port... @tcaputi: could you possibly share the validation plan/actors involved? It may be worth while as a community to scrape together some funds for a proper audit by a well known vendor (or ask them to do it for community PR/exposure - thats a marketable accomplishment) along with some tooling for the test suite to ensure we don't start leaking sensitive content from future additions made by others alongside your code. @behlendorf: can you rope in some of the USG crypto folks? Natsec relevant data will end up being stored this way... While I'm not suggesting exposure of audit methods by orgs who "don't do that," a comprehensive review with "watch for these bits changing moving forward" notes would be a godsend since the on disk format is subject to change down the line, and we want to avoid the FIPS certification nightmare as much as we can since someone, somewhere, will need to get it to use this for work. Far as getting bandwidth for review - tell them that the Russians have already performed several reviews (if they didn't yet, they will now) and probably started building tooling around anything found, so its a tactical boon to cut them off at the pass, and strategic sanity to use the scarce resources (cryptanalysts) for future proofing something as critical as storage crypto for the most advanced fs on earth used across most POSIX systems in play. Let's not do bitlocker all over again in the federal/mil space. @infosec-folks: if this is your area of expertise or you have a graybeard in your shop who can perform ffts and sha ops in their head, please speak up. Without validation this is hard to adopt, and without adoption, real world testing is not going to happen. Spread the word about this work and try to get hands/eyes on it. ZFS has crypto, we now need to ensure its life cycle as a first order member of the functional stack.

…

-------- Original Message -------- From: Pawel Jakub Dawidek <notifications@github.com> Sent: Monday, December 26, 2016 01:41 AM To: zfsonlinux/zfs <zfs@noreply.github.com> Subject: Re: [zfsonlinux/zfs] ZFS Encryption (#4329) CC: RageLtMan <rageltman@sempervictus.com>,Mention <mention@noreply.github.com>

We (at Wheel Systems) will probably work on porting ZFS encryption to FreeBSD, unless someone will beat us to it. For us the most important feature is encrypted zfs send, so we may wait until this is done. Still, there are many active committers working on ZFS in FreeBSD, so I'd expect this is going to happen soon. -- Paweł Jakub Dawidek > On 24 Dec 2016, at 08:06, Graham Perrin ***@***.***> wrote: > > … anyone from BSD-land watching/porting this? … > > @sempervictus I watch with great interest, but I'm not a developer. > > Extending the request for comment, to FreeBSD communities > > freebsd-hackers > > Multi-platform ZFS encryption (ZoL pull #4329 by Tom Caputi): RFC > TrueOS > > In the community lounge, under Approaches to encryption with ZFS: > > https://discourse.trueos.org/t/-/234/4?u=grahamperrin > (Recent efforts are necessarily focused on the transition to OpenRC, so I should not expect a response in the ZFS topic until some time in January.) > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or mute the thread. > -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #4329 (comment)

encrypted send without loaded keys is already in place, so is scrub.

To clear up a bit of confusion, raw sends with unloaded keys is NOT actually currently implemented yet. I wanted to make this PR as small as functionally possible to keep it as reviewable as possible. That said, the code has been completely engineered with raw sends in mind and it should not take long to implement (probably a week or 2). I've been a bit busy with some other projects and was waiting for a few initial reviews to come in from the ZFS maintainers (even before cryptogaphic review) before I got started.

Scrubs and resilvers are implemented as these are essential to the way ZFS currently functions so this PR would be incomplete without them.

My understanding is we are currently held up by the mostly insurmountable (for open source endeavors) question of whether the on disk/sent structures expose something which may reduce cryptographic strength in offline attacks (today or tomorrow).

That is certainly the biggest hurdle to overcome. However it is my understanding that review of this patch was also slated to be looked at only after #3656 was merged, so I've just been maintaining it until then.

@tcaputi: could you possibly share the validation plan/actors involved? It may be worth while as a community to scrape together some funds for a proper audit by a well known vendor (or ask them to do it for community PR/exposure - thats a marketable accomplishment) along with some tooling for the test suite to ensure we don't start leaking sensitive content from future additions made by others alongside your code.

I'm not sure how much I'm allowed to say about this due to NDAs I have signed. I believe I can say, however, that Datto would definitely be interested in pooling together some funds to get this properly reviewed. I would also encourage, as you said, anyone who has any contacts at a crypto security shop to see if they might be interested in helping out.

tcaputi · Dec 26, 2016

encrypted send without loaded keys is already in place, so is scrub.

To clear up a bit of confusion, raw sends with unloaded keys is NOT actually currently implemented yet. I wanted to make this PR as small as functionally possible to keep it as reviewable as possible. That said, the code has been completely engineered with raw sends in mind and it should not take long to implement (probably a week or 2). I've been a bit busy with some other projects and was waiting for a few initial reviews to come in from the ZFS maintainers (even before cryptogaphic review) before I got started.

Scrubs and resilvers are implemented as these are essential to the way ZFS currently functions so this PR would be incomplete without them.

My understanding is we are currently held up by the mostly insurmountable (for open source endeavors) question of whether the on disk/sent structures expose something which may reduce cryptographic strength in offline attacks (today or tomorrow).

That is certainly the biggest hurdle to overcome. However it is my understanding that review of this patch was also slated to be looked at only after #3656 was merged, so I've just been maintaining it until then.

@tcaputi: could you possibly share the validation plan/actors involved? It may be worth while as a community to scrape together some funds for a proper audit by a well known vendor (or ask them to do it for community PR/exposure - thats a marketable accomplishment) along with some tooling for the test suite to ensure we don't start leaking sensitive content from future additions made by others alongside your code.

I'm not sure how much I'm allowed to say about this due to NDAs I have signed. I believe I can say, however, that Datto would definitely be interested in pooling together some funds to get this properly reviewed. I would also encourage, as you said, anyone who has any contacts at a crypto security shop to see if they might be interested in helping out.

@tcaputi: Thanks for clarification on the feature list. Re NDAs - i would float the question of public review to the peer entities, as you may even be able to leverage the corporate project management resources to help accomplish it (i've found that tapping those folks for a few hours to organize open source work can be a massive win). The prospective exposure of supporting entities in terms of security context is largely irrelevant once the review work is completed - it will be an even playing field with everyone using it, all they need to differentiate is run their code with a multiplied hardcoded in for the pbkdf iters and off they go being "special" without breaking compatibility in on-disk structure. If they're using it prior to a full review, they may be sitting on holes they dont have the resources to find or fix (back to the original "infosec on OSS is hell on earth" concern). Far as keeping this PR as small as possible, that's kinda like saying you're trying to pack godzilla on a freight ship and shipping his luggage separately to save space :-). If you have the resources available in the coming weeks to implement raw send, i'd say do it in this PR and have the full set of functionality for testing. People wont touch it until its feature complete, i'm sure you've heard the "its not even ready to be tested yet" line regarding many other projects before... I'll be adding this code to our RAP/KASLR-enabled KVM kernel build in Jan, and will see what we can scrub from external memory prints with volatility and other such tools (online attack vectors against a hosted VM) since that use case applies to our client-facing objectives. Time permitting, other evaluations will be performed as well (i need to get better at zdb/zstreamdump/zhack anyway)

…

On Mon, Dec 26, 2016 at 2:12 PM, Tom Caputi ***@***.***> wrote: encrypted send without loaded keys is already in place, so is scrub. To clear up a bit of confusion, raw sends with unloaded keys is NOT actually currently implemented yet. I wanted to make this PR as small as functionally possible to keep it as reviewable as possible. That said, the code has been completely engineered with raw sends in mind and it should not take long to implement (probably a week or 2). I've been a bit busy with some other projects and was waiting for a few initial reviews to come in from the ZFS maintainers (even before cryptogaphic review) before I got started. Scrubs and resilvers are implemented as these are essential to the way ZFS currently functions so this PR would be incomplete without them. My understanding is we are currently held up by the mostly insurmountable (for open source endeavors) question of whether the on disk/sent structures expose something which may reduce cryptographic strength in offline attacks (today or tomorrow). That is certainly the biggest hurdle to overcome. However it is my understanding that review of this patch was also slated to be looked at only after #3656 <#3656> was merged, so I've just been maintaining it until then. @tcaputi <https://github.com/tcaputi>: could you possibly share the validation plan/actors involved? It may be worth while as a community to scrape together some funds for a proper audit by a well known vendor (or ask them to do it for community PR/exposure - thats a marketable accomplishment) along with some tooling for the test suite to ensure we don't start leaking sensitive content from future additions made by others alongside your code. I'm not sure how much I'm allowed to say about this due to NDAs I have signed. I believe I can say, however, that Datto would definitely be interested in pooling together some funds to get this properly reviewed. I would also encourage, as you said, anyone who has any contacts at a crypto security shop to see if they might be interested in helping out. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4329 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABRPjI7ktblnBcEsB-FmK3T11Id56DRNks5rMBGIgaJpZM4HYjZG> .

sempervictus · Dec 29, 2016

@tcaputi: Thanks for clarification on the feature list. Re NDAs - i would float the question of public review to the peer entities, as you may even be able to leverage the corporate project management resources to help accomplish it (i've found that tapping those folks for a few hours to organize open source work can be a massive win). The prospective exposure of supporting entities in terms of security context is largely irrelevant once the review work is completed - it will be an even playing field with everyone using it, all they need to differentiate is run their code with a multiplied hardcoded in for the pbkdf iters and off they go being "special" without breaking compatibility in on-disk structure. If they're using it prior to a full review, they may be sitting on holes they dont have the resources to find or fix (back to the original "infosec on OSS is hell on earth" concern). Far as keeping this PR as small as possible, that's kinda like saying you're trying to pack godzilla on a freight ship and shipping his luggage separately to save space :-). If you have the resources available in the coming weeks to implement raw send, i'd say do it in this PR and have the full set of functionality for testing. People wont touch it until its feature complete, i'm sure you've heard the "its not even ready to be tested yet" line regarding many other projects before... I'll be adding this code to our RAP/KASLR-enabled KVM kernel build in Jan, and will see what we can scrub from external memory prints with volatility and other such tools (online attack vectors against a hosted VM) since that use case applies to our client-facing objectives. Time permitting, other evaluations will be performed as well (i need to get better at zdb/zstreamdump/zhack anyway)

…

On Mon, Dec 26, 2016 at 2:12 PM, Tom Caputi ***@***.***> wrote: encrypted send without loaded keys is already in place, so is scrub. To clear up a bit of confusion, raw sends with unloaded keys is NOT actually currently implemented yet. I wanted to make this PR as small as functionally possible to keep it as reviewable as possible. That said, the code has been completely engineered with raw sends in mind and it should not take long to implement (probably a week or 2). I've been a bit busy with some other projects and was waiting for a few initial reviews to come in from the ZFS maintainers (even before cryptogaphic review) before I got started. Scrubs and resilvers are implemented as these are essential to the way ZFS currently functions so this PR would be incomplete without them. My understanding is we are currently held up by the mostly insurmountable (for open source endeavors) question of whether the on disk/sent structures expose something which may reduce cryptographic strength in offline attacks (today or tomorrow). That is certainly the biggest hurdle to overcome. However it is my understanding that review of this patch was also slated to be looked at only after #3656 <#3656> was merged, so I've just been maintaining it until then. @tcaputi <https://github.com/tcaputi>: could you possibly share the validation plan/actors involved? It may be worth while as a community to scrape together some funds for a proper audit by a well known vendor (or ask them to do it for community PR/exposure - thats a marketable accomplishment) along with some tooling for the test suite to ensure we don't start leaking sensitive content from future additions made by others alongside your code. I'm not sure how much I'm allowed to say about this due to NDAs I have signed. I believe I can say, however, that Datto would definitely be interested in pooling together some funds to get this properly reviewed. I would also encourage, as you said, anyone who has any contacts at a crypto security shop to see if they might be interested in helping out. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4329 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABRPjI7ktblnBcEsB-FmK3T11Id56DRNks5rMBGIgaJpZM4HYjZG> .

Is there a way to make the loaded keys volatile? So that after a reboot the keys must be entered again.

storkone · Jan 3, 2017

Is there a way to make the loaded keys volatile? So that after a reboot the keys must be entered again.

@storkone

Is there a way to make the loaded keys volatile? So that after a reboot the keys must be entered again.

That is exactly how it currently works.

tcaputi · Jan 3, 2017

@storkone

Is there a way to make the loaded keys volatile? So that after a reboot the keys must be entered again.

That is exactly how it currently works.

Is there a way to use encryption with tpm? So one could use this with a coreboot linux payload, like heads (https://github.com/osresearch/heads/commits/master) does. As I want to use buildroot in order to get small
and rather easy to deploy on bios-chip environment.

Kokokokoka · Jan 3, 2017

Is there a way to use encryption with tpm? So one could use this with a coreboot linux payload, like heads (https://github.com/osresearch/heads/commits/master) does. As I want to use buildroot in order to get small
and rather easy to deploy on bios-chip environment.

@tcaputi, sorry about the noise. Didn't actually looked at the code nor build it. But the last paragraph in your comment gave me the impression that the keys were non volatile.

storkone · Jan 3, 2017

@tcaputi, sorry about the noise. Didn't actually looked at the code nor build it. But the last paragraph in your comment gave me the impression that the keys were non volatile.

@Kokokokoka

Is there a way to use encryption with tpm?

As of now, no. The ICP does not currently work with TPMs and we cannot use the Linux crypto frameworks due to licensing problems.

tcaputi · Jan 3, 2017

@Kokokokoka

Is there a way to use encryption with tpm?

As of now, no. The ICP does not currently work with TPMs and we cannot use the Linux crypto frameworks due to licensing problems.

ahrens · Jan 6, 2017

ahrens reviewed Jan 6, 2017

View changes

module/zfs/zio_crypt.c

module/zfs/dsl_crypt.c

@ahrens I've addressed your comments (other than the one about the mapping tree, which I think we need to discuss more) and repushed.

tcaputi · Jan 6, 2017

@ahrens I've addressed your comments (other than the one about the mapping tree, which I think we need to discuss more) and repushed.

ahrens · Jan 14, 2017

ahrens requested changes Jan 14, 2017

View changes

module/zfs/dsl_dataset.c

module/zfs/dmu_objset.c

cmd/zinject/translate.c

ahrens · Jan 14, 2017

man/man8/zpool.8

+.ad
+.sp .6
+.RS 4n
+Indicates that the zpool command will request encryption keys for all encrypted datasets it attempts to mount as it is bringing the pool online. This is equivalent to running \fBzfs mount\fR on each encrypted dataset immediately after the pool is imported. If any datasets have a \fBprompt\fR keysource this command will block waiting for the key to be entered. Otherwise, encrypted datasets will be left unavailable until the keys are loaded.


equivalent to running \fBzfs mount\fR on each encrypted dataset

So, it mounts them? That seems counterintuitive. Seems like this should be equivalent to zfs key -l on all encrypted filesystems.

waiting for the key

I think this should be keys (plural).

Otherwise,

I think you mean, If \fB-l\fR is not specified,, not "If no datasets have a prompt keysource"

Correct on both points, will fix.

Thinking about this some more, on the surface it seems like -l does approximately nothing, because we already mount filesystems when the pool is imported, and presumably that will load keys for the filesystems we mount. I think the exception is filesystems with mountpoint=legacy or canmount=off | noauto. If that's right, I think it would be worth mentioning here - that -l is used to ensure that all keys are loaded, even for filesystems that are not mounted because they have mountpoint=legacy or canmount=off | noauto.

zfs key -l and zpool import -l also will result in encrypted zvols appearing in /dev/, by the way. To me, these commands allow the user to access the dataset normally doing whatever without worrying about encryption anymore.

That said, I can verify the behavior of this command and make the comment more specific accordingly.

Some more context about the reason for zpool import -l after looking back at the code:

Thinking about this some more, on the surface it seems like -l does approximately nothing, because we already mount filesystems when the pool is imported, and presumably that will load keys for the filesystems we mount.

Right now, zpool import (without -l) will actually leave encrypted datasets unmounted. The reason for this was compatibility. I was afraid that someone might add an encrypted dataset to a pool that is mounted via an automated script. When the pool is re-imported, the script would hit the prompt for the encrypted dataset and hang indefinitely. As a result, I wanted people to have to opt into the automatic key loading.

Right now, zpool import (without -l) will actually leave encrypted datasets unmounted.

OK, let's document that in the manpage.

ahrens · Jan 14, 2017

man/man5/zpool-features.5

+hierarchy, file size, file holes, and dedup tables. Key rotation is managed
+internally by the ZFS kernel module and changing the user's key does not
+require re-encrypting the entire dataset. Datasets can be scrubbed, resilvered,
+moved, and deleted without the encryption keys being loaded (see the zfs key


moved

renamed?

zfs key

\fBzfs key\fR

ahrens · Jan 14, 2017

man/man5/zpool-features.5

+provides additional protection against maliciously altered data. Deduplication
+is still possible with encryption enabled but for security, datasets will only
+dedup against themselves, their snapshots, and their clones. Encrypted data
+cannot be embedded via the \fIembedded_data\fR feature.


This paragraph belongs in the zfs.8 manpage.

ahrens · Jan 14, 2017

man/man5/zpool-features.5

+encryption datasets may be vulnerable to a CRIME-like attack if applications
+accessing the data allow for it. Deduplication with encryption will leak
+information about which blocks are equivalent in a dataset and will incur an
+extra CPU cost per block written.


This paragraph belongs in the zfs.8 manpage.

ahrens · Jan 14, 2017

man/man5/zpool-features.5

+internally by the ZFS kernel module and changing the user's key does not
+require re-encrypting the entire dataset. Datasets can be scrubbed, resilvered,
+moved, and deleted without the encryption keys being loaded (see the zfs key
+subcommand for more info).


All the information in here should also be in the zfs.8 manpage, perhaps in a new subsection of the DESCRIPTION (peer to Clones, Mount Points, etc). The zpool-features manpage should primarily document why you would want to enable this feature. See for example the documentation of the bookmarks feature. In this case:

Enabling this property allows setting the \fBencryption\fR property to values other than \fBoff\fR.

Will fix. Would you want me to move some of this information or make a copy of it in zfs.8 (perhaps with different wording)?

I think move it, and leave just the minimal description of what this property does (allows setting encryption), and when it becomes active and enabled.

Sounds good. Will fix.

@ahrens Thanks for helping with the review. I should be able to address all of the comments and make another push early next week.

tcaputi · Jan 14, 2017

@ahrens Thanks for helping with the review. I should be able to address all of the comments and make another push early next week.

ahrens · Jan 14, 2017

ahrens reviewed Jan 14, 2017

View changes

ahrens · Jan 14, 2017

man/man8/zfs.8

+
+.LP
+.nf
+\fB\fBzfs key -l\fR \fIfilesystem\fR | \fIvolume\fR\fR


I think it would be easier to read (e.g. in a script or an email) if we also have long opts for the "verbs" here:

zfs key --load zfs key --unload zfs key --change

ZoL actually doesn't have any longopts handling at all. I can add it though, if it will help.

This (the wording of this command) is actually the biggest complaint I've received about the patch so far. I've heard from several people that they don't like that key isn't a verb like create or destroy. That said, I have been unable to come up with any alternatives that dont seem tedious (like zfs loadkey, zfs unloadkey,zfs changekey`. If you have any thoughts here now would probably be a good time while I'm cleaning up all of this other stuff.

I had the same thoughts... nothing better jumps to mind but I'll keep pondering...

FWIW, I think that the tedious zfs loadkey etc are in keeping with the existing interface, which does have some collections of verbs that all operate on the same concepts (e.g. zfs allow, zfs unallow; and zfs hold, zfs release, zfs holds).

I can change this if you'd like (its a bit tedius to make every one of these a separate funnction in both kernel and userspace. My only other qualm here is that people have asked for a few other key-related subcommands like zfs verifykey which would tell you if a key is correct (even if its already loaded).

Another one I was going to add while doing all these fixups is zfs linkkey which would tell a dataset to inherit the wrapping keys of a parent. Currently, you can break wrapping key inheritance with zfs key -c but there's no way to relink them later so this would add that functionality. So at that point there are now 5 key commands which seems like a lot of bloat in both zfs_cmd.c and zfs_ioctl.c.

I'm fine doing it either way. Let me know what you think is best. Maybe @behlendorf could weigh in here too.

If you do create lots of subcommands, I don't think that makes it necessary to create lots of ioctls; you can keep using one ioctl if that's cleaner/simpler.

zfs linkkey

Too bad key change doesn't really fit in with the property scheme, otherwise zfs inherit would be the obvious choice. I'd suggest that if we want this functionality, it may make sense to borrow the "inherit" terminology here, e.g. zfs inheritkey or zfs key --inherit.

ahrens · Jan 14, 2017

ahrens reviewed Jan 14, 2017

View changes

ahrens · Jan 14, 2017

man/man8/zfs.8

+.ne 2
+.mk
+.na
+\fB\fBkeysource\fR=<\fBraw\fR | \fBhex\fR | \fBpassphrase\fR>,<\fBprompt\fR | \fBfile://\fR\fI<absolute file path\fR>>


I get wanting to use a URI in case we later add extensions, like to contact some kind of keyserver. But it seems like it would be handy if we could specify an absolute path as a shortcut for file://<path>. This seems non-ambiguous since AFAIK a URI can't start with a /.

This is another leftover from trying to stick to the Solaris UI. I actually kind of like the file URI, but that's just me. Maybe @behlendorf could be a tie-breaker here?

ahrens · Jan 14, 2017

man/man8/zfs.8

+.ne 2
+.mk
+.na
+\fB\fBkeysource\fR=<\fBraw\fR | \fBhex\fR | \fBpassphrase\fR>,<\fBprompt\fR | \fBfile://\fR\fI<absolute file path\fR>>


If you do keysource=raw,prompt, is it really possible to type/paste the raw data (including nul characters) into the terminal? If not, I assume this is still a valid setting because you can pipe raw bytes into zfs key -l? That might be worth mentioning in the "STDIN" section of zfs key -l docs.

It actually is possible to type paste raw characters in (at least in my terminal maybe not all). But yes, this is still valid for the use-case you described. I can add a sentence or 2 to the man page.

Back to basics for a moment, and apologies if I read these points out of context:

… can't create an unencrypted fs under an encrypted …

– and from related https://github.com/tcaputi/zfs/blob/2f7c9cb5e27c4feef62971d0c2f802735d60a4e8/man/man8/zfs.8#L1021:

… Regardless, all children of an encrypted dataset must also be encrypted.

Children and creation of children aside, for a moment.

What, if anything, exists – or should exist – to prevent simple mount of a non-encrypted filesystem at a point within an encrypted filesystem?

(Probably treat that as a question about documentation. Not an expression of paranoia.)

grahamperrin · Jan 14, 2017

Back to basics for a moment, and apologies if I read these points out of context:

… can't create an unencrypted fs under an encrypted …

– and from related https://github.com/tcaputi/zfs/blob/2f7c9cb5e27c4feef62971d0c2f802735d60a4e8/man/man8/zfs.8#L1021:

… Regardless, all children of an encrypted dataset must also be encrypted.

Children and creation of children aside, for a moment.

What, if anything, exists – or should exist – to prevent simple mount of a non-encrypted filesystem at a point within an encrypted filesystem?

(Probably treat that as a question about documentation. Not an expression of paranoia.)

ahrens · Jan 14, 2017

ahrens reviewed Jan 14, 2017

View changes

ahrens · Jan 14, 2017

man/man8/zfs.8

+.ad
+.sp .6
+.RS 4n
+Unloads a key from ZFS, removing the ability to access the dataset and all of its children that inherit the \fBencryption\fR property. This requires that the dataset is not currently open or mounted. When a key is unloaded the \fBkeystatus\fR property will be set to \fBunavailable\fR.


how about \fBkeystatus\fR property is \fBunavailable\fR. It seems strange to say that it's "set" because it isn't a settable property.

ahrens · Jan 14, 2017

man/man8/zfs.8

+.ad
+.sp .6
+.RS 4n
+Allows a user to change the encryption key used to access a dataset. This command requires that the existing key for the dataset is already loaded into ZFS. This command may also be used to change the \fBpbkdf2iters\fR and / or \fBkeysource\fR properties as needed. If the dataset was previously inheriting the \fBencryption\fR property when this command is run it will now be locally set, indicating that this dataset must have its key loaded separately from the parent.


What's the rationale behind requiring the key already be loaded for zfs key -c, compared with zfs mount which will load the key for you? I think you mentioned in another comment that there was a concern about what state you're left in (loaded or not) when zfs key -c completes. Maybe it should be the same as if you do zfs mount; zfs unmount, which I think would leave the key loaded.

The philosophy around key loaded-ness seems to be that in general we attempt to load when the keys are needed, and unload only when explicitly requested (zfs key -u). In the current implementation, I'm imaging user interaction:

$ zfs key -c ... Sorry, you can't change keys because they aren't loaded. Run "zfs key -l ..." to load them. # OK fine I'll copy/paste what you said to do, but if you know what I have to do, why didn't you do it for me? $ zfs key -l ... $ zfs key -c ...

Seems like we should have a good reason for annoying them in this case.

Maybe a good middle ground is that the keys will remain in whatever state they were in before the zfs key -c command was run? Is that reasonable?

See this comment I made above.

@grahamperrin

What, if anything, exists – or should exist – to prevent simple mount of a non-encrypted filesystem at a point within an encrypted filesystem?

Nothing prevents a mount like that from being done right now, but as far as I am aware that will not automount when the parent is mounted where as child datasets are.

I can look into adding a sentence or 2 to the docs about this

tcaputi · Jan 14, 2017

@grahamperrin

What, if anything, exists – or should exist – to prevent simple mount of a non-encrypted filesystem at a point within an encrypted filesystem?

Nothing prevents a mount like that from being done right now, but as far as I am aware that will not automount when the parent is mounted where as child datasets are.

I can look into adding a sentence or 2 to the docs about this

if this PR good for testing by unpatient end-users? is the on-disk format finished?

mailinglists35 · Jan 17, 2017

if this PR good for testing by unpatient end-users? is the on-disk format finished?

@mailinglists35 and everyone else:

if this PR good for testing by unpatient end-users? is the on-disk format finished?

I would say patch 89b4e7a is good enough to test. This week I will be addressing @ahrens comments and I will be using buildbot here for testing, so I wouldn't use any of these commits until theyre back to being stable (I will squash them all at that time).

The on-disk format has not been verified yet. We are working to get there.

tcaputi · Jan 17, 2017

@mailinglists35 and everyone else:

if this PR good for testing by unpatient end-users? is the on-disk format finished?

I would say patch 89b4e7a is good enough to test. This week I will be addressing @ahrens comments and I will be using buildbot here for testing, so I wouldn't use any of these commits until theyre back to being stable (I will squash them all at that time).

The on-disk format has not been verified yet. We are working to get there.

I'm still getting patches on a silver-platter once the dust settles, right? :)

lundman · Jan 18, 2017

I'm still getting patches on a silver-platter once the dust settles, right? :)

@lundman Yes. Let me get all of these changes stable first (hopefully sometime this week) and then I'll squash them and I'll send you a nice and tidy patch

tcaputi · Jan 18, 2017

@lundman Yes. Let me get all of these changes stable first (hopefully sometime this week) and then I'll squash them and I'll send you a nice and tidy patch

ahrens · Jan 29, 2017

ahrens reviewed Jan 29, 2017

View changes

ahrens · Jan 29, 2017

include/sys/arc.h

+
+/*
+ * With the advent of encrypted data in the ARC it is now possible for
+ * legitimate errors to arise while transforming data into its desired format.


In general, the style/tone of comments should be to describe the code as it is. Consider someone reading this in 5 years time - the "advent" making it "now possible" will seem misplaced. Instead, consider something like:
Because the ARC can store encrypted data, errors (not due to bugs) may arise while transforming data into its desired format - specifically, when decrypting, the key may not be present, or the HMAC may not be correct, which signifies deliberate tampering with the on-disk state (assuming that the checksum was correct). The "error" parameter will be nonzero in this case, even if there is no associated zio.
Not sure if the technical details above are correct, but hopefully the style makes sense.

That is correct. I will also fix the comment.

ahrens · Jan 29, 2017

include/sys/arc.h

@@ -112,20 +121,21 @@ typedef enum arc_flags
 	ARC_FLAG_L2_WRITING		= 1 << 11,	/* write in progress */
 	ARC_FLAG_L2_EVICTED		= 1 << 12,	/* evicted during I/O */
 	ARC_FLAG_L2_WRITE_HEAD		= 1 << 13,	/* head of write list */
+	ARC_FLAG_ENCRYPT		= 1 << 14,	/* encrypted on disk */


You might add something like may or may not be encrypted in memory, to emphasize this fact (lest anyone wonder if encrypted on disk is what you precisely meant).

ahrens · Jan 29, 2017

include/sys/arc_impl.h

+ * disk as they appear in the main pool. In order for this to work we
+ * need to pass around the encryption parameters so they can be used
+ * to write data to the L2ARC. This struct is only defined in the
+ * arc_buf_hdr_t if the L1 header is defined and the has the


typo: remove extraneous the.

ahrens · Jan 29, 2017

include/sys/arc.h

@@ -144,7 +154,8 @@ typedef enum arc_flags

 typedef enum arc_buf_flags {
 	ARC_BUF_FLAG_SHARED		= 1 << 0,
-	ARC_BUF_FLAG_COMPRESSED		= 1 << 1
+	ARC_BUF_FLAG_COMPRESSED		= 1 << 1,
+	ARC_BUF_FLAG_ENCRYPTED		= 1 << 2


Add a comment here contrasting this with ARC_FLAG_ENCRYPTED.

ahrens · Jan 29, 2017

module/zfs/arc.c

@@ -1331,13 +1391,30 @@ arc_buf_lsize(arc_buf_t *buf)
 	return (HDR_GET_LSIZE(buf->b_hdr));
 }

+boolean_t
+arc_is_encrypted(arc_buf_t *buf)


Add a comment explaining what this means. In particular, this tells if the buf is stored encrypted in the ARC, and therefore ... [can't be read unless X flag passed to arc_read()? unless key loaded via foobar()?]. It returns false if the data is encrypted on disk but decrypted in memory.

(The same should type of comment should probably have been added above arc_get_compression.)

will fix. should i add the one for compression as well while I'm here? Or should that be left to a different PR?

I'm fine with adding the compression comment while you're here.

ahrens · Jan 29, 2017

include/sys/spa.h

@@ -208,7 +209,7 @@ typedef struct zio_cksum_salt {
 * G		gang block indicator
 * B		byteorder (endianness)
 * D		dedup
- * X		encryption (on version 30, which is not supported)
+ * X		encryption


I think this should say set to zero; see diagram below for encrypted blocks, and then there should be a separate diagram that shows the layout for encrypted blocks, similar to how we explain embedded BP's (IIRC, checksum[2-3], fill, and dva[2] have different meanings when the X bit is set). The legend of the new diagram might omit fields that are already described here.

Since I think the in-memory representations of the IV and salt (and MAC?) are treated as byte arrays, be sure to explain how those are encoded into the 64-bit words (e.g. low 8 bits of the word are the low byte of the array, etc). As you probably know, we can't simply treat the byte array as a uint64_t*, because of endianness concerns.

ahrens · Jan 29, 2017

include/sys/spa.h

+}
+
+#define	BP_GET_IV2(bp)			BF64_GET((bp)->blk_fill, 32, 32)
+#define	BP_SET_IV2(bp, iv2)		BF64_SET((bp)->blk_fill, 32, 32, iv2);


These should assert that the BP is encrypted (you can use the comma operator to do the assertion in the GET, see BPE_GET_ETYPE() for an example).

ahrens · Jan 29, 2017

include/sys/spa.h

+}
+
+#define	BP_GET_IV2(bp)			BF64_GET((bp)->blk_fill, 32, 32)
+#define	BP_SET_IV2(bp, iv2)		BF64_SET((bp)->blk_fill, 32, 32, iv2);


It might be helpful (for readability) to add accessor macros for the other encryption-specific fields (even if they are entire words). Those macros could also assert that they are only used on encrypted BP's.

I actually have full functions for these (see zio_crypt_encode_params_bp() in zio_crypt.c for instance). Let me know if you think they should be changed to macros and brought to spa.h

ahrens · Jan 29, 2017

include/sys/spa_impl.h

@@ -269,6 +270,7 @@ struct spa {
 	spa_avz_action_t	spa_avz_action;	/* destroy/rebuild AVZ? */
 	uint64_t	spa_errata;		/* errata issues detected */
 	spa_stats_t	spa_stats;		/* assorted spa statistics */
+	spa_keystore_t	spa_keystore;	/* loaded crypto keys */


align the comment with the adjacent ones.

ahrens · Jan 29, 2017

include/sys/zil.h

@@ -86,7 +87,7 @@ typedef struct zil_header {
 * number passed in the blk_cksum field of the blkptr_t
 */
 typedef struct zil_chain {
-	uint64_t zc_pad;
+	uint64_t zc_mac; /* mac for encryption */


align the comment with the adjacent ones

ahrens · Jan 31, 2017

ahrens reviewed Jan 31, 2017

View changes

ahrens · Jan 31, 2017

man/man8/zpool.8

+.ad
+.sp .6
+.RS 4n
+Indicates that the zpool command will request encryption keys for all encrypted datasets it attempts to mount as it is bringing the pool online. This is equivalent to running \fBzfs mount\fR on each encrypted dataset immediately after the pool is imported. If any datasets have a \fBprompt\fR keysource this command will block waiting for the key to be entered. Otherwise, encrypted datasets will be left unavailable until the keys are loaded.


Right now, zpool import (without -l) will actually leave encrypted datasets unmounted.

OK, let's document that in the manpage.

ahrens · Jan 31, 2017

module/zfs/arc.c

@@ -1331,13 +1391,30 @@ arc_buf_lsize(arc_buf_t *buf)
 	return (HDR_GET_LSIZE(buf->b_hdr));
 }

+boolean_t
+arc_is_encrypted(arc_buf_t *buf)


I'm fine with adding the compression comment while you're here.

ahrens · Jan 31, 2017

include/sys/zio_crypt.h

+/*
+ * After encrypting many blocks with the same salt we may start to run
+ * up against the theoretical limits of how much data can securely be
+ * encrypted a single key using the supported encryption modes. To


typo: encrypted *with* a single key

ahrens · Jan 31, 2017

include/sys/zio_crypt.h

+ * counteract this we generate a new salt after writing
+ * ZIO_CRYPT_MAX_SALT_USAGE blocks of data, tracked by zk_salt_count.
+ * The current value was chosen because it is approximately the number
+ * of blocks that would have to be written in order to acheive a


spelling: achieve

ahrens · Jan 31, 2017

include/sys/zio_crypt.h

+ * 1 / 1 trillion chance of having an IV collision. Developers looking to
+ * change this number should make sure they take into account the
+ * birthday problem in regards to IV generation and the limits of what the
+ * underlying mode can actually handle.


I think this comment should lay out the math behind this number. That would make it clear that the birthday problem is accounted for. Also we should simply say that we account for the birthday problem rather than condescend to future readers. E.g. This protects against a birthday attack. With n = 400 million blocks encrypted with the same key and salt, and d = the number of IV's = 2^96, the probability of two blocks using the same IV is: 1 - 1 * (1 - 1 / d ) * ( 1 - 2 / d ) * … * (1 - (n - 1) / d) which is approximated by ... (according to [citation]), which is 10^-12.

ahrens · Jan 31, 2017

include/sys/zio_crypt.h

+
+/* utility macros */
+#define	BITS_TO_BYTES(x) (((x) + 7) >> 3)
+#define	BYTES_TO_BITS(x) (x << 3)


I think that the compiler can do the optimization for / 8. How about x * NBBY and (x + NBBY - 1) / NBBY.

ahrens · Jan 31, 2017

include/sys/zio_crypt.h

+#define	DSL_CRYPTO_KEY_IV "DSL_CRYPTO_IV"
+#define	DSL_CRYPTO_KEY_MAC "DSL_CRYPTO_MAC"
+#define	DSL_CRYPTO_KEY_MASTER_BUF "DSL_CRYPTO_MASTER"
+#define	DSL_CRYPTO_KEY_HMAC_KEY_BUF "DSL_CRYPTO_HMAC_KEY"


Any reason for the macro names and values to not be consistent, at least DSL_CRYPTO_KEY_XXX -> "DSL_CRYPTO_XXX" (this applies to the last 2 which differ in the _BUF suffix)

ahrens · Jan 31, 2017

include/sys/zio_crypt.h

+extern zio_crypt_info_t zio_crypt_table[ZIO_CRYPT_FUNCTIONS];
+
+/* ZAP entry keys for DSL Encryption Keys stored on disk */
+#define	DSL_CRYPTO_KEY_CRYPT "DSL_CRYPTO_CRYPT"


"crypto crypt" is not the most descriptive name. Maybe CRYPTO_FUNC?

ahrens · Jan 31, 2017

include/sys/spa.h

@@ -208,7 +209,7 @@ typedef struct zio_cksum_salt {
 * G		gang block indicator
 * B		byteorder (endianness)
 * D		dedup
- * X		encryption (on version 30, which is not supported)
+ * X		encryption


Since I think the in-memory representations of the IV and salt (and MAC?) are treated as byte arrays, be sure to explain how those are encoded into the 64-bit words (e.g. low 8 bits of the word are the low byte of the array, etc). As you probably know, we can't simply treat the byte array as a uint64_t*, because of endianness concerns.

ahrens · Feb 4, 2017

ahrens reviewed Feb 4, 2017

View changes

ahrens · Feb 4, 2017

include/sys/zio_impl.h

+ * encrypted, this stage determines how the encryption metadata is stored in
+ * the bp. Decryption is performed during ZIO_STAGE_READ_BP_INIT as a transform
+ * callback. Encryption is also mutually exclusive with nopwrite, because
+ * encrypted blocks with the same plaintext will not have matching ciphertexts.


how about because blocks with the same plaintext will be encrypted with different salts and therefore different IV's (if dedup is off), and therefore have different ciphertexts.

ahrens · Feb 4, 2017

module/zfs/zfeature_common.c

+	zfeature_register(SPA_FEATURE_ENCRYPTION,
+	    "com.datto:encryption", "encryption",
+	    "Support for dataset level encryption",
+	    0, encryption_deps);


The encryption feature is is deactivated when it's no longer used, but it isn't PER_DATASET? I'll look for where this is implemented... it may be simpler to use the PER_DATASET flag here.

From the comments around this flag it looks like this might be the case. I'm not familiar with this flag but it looks like all I need to do is set ds->ds_feature_activation_needed[feature] = B_TRUE when the dataset starts using the feature. Is this correct? Currently, the implementation increments the feature count when it creates a key object in the spa and decrements it when that object is destroyed.

ahrens · Feb 4, 2017

module/zcommon/zfs_prop.c

+	    ZIO_CRYPT_DEFAULT, PROP_ONETIME, ZFS_TYPE_DATASET,
+	    "on | off | aes-128-ccm | aes-192-ccm | aes-256-ccm | "
+	    "aes-128-gcm | aes-192-gcm | aes-256-gcm", "ENCRYPTION",
+	    crypto_table);


I guess you could argue that this property is inherited, but from the prop management infrastructure, it's ONETIME, so it should probably be registered in the "set once index properties" section

ahrens · Feb 4, 2017

module/zcommon/zfs_prop.c

+boolean_t
+zfs_prop_valid_keylocation(const char *str)
+{
+	if (strlen(str) == 6 && strncmp("prompt", str, 6) == 0)


This is equivalent to strcmp("prompt", str) == 0, but more brittle

ahrens · Feb 4, 2017

module/zcommon/zfs_prop.c

+{
+	if (strlen(str) == 6 && strncmp("prompt", str, 6) == 0)
+		return (B_TRUE);
+	else if (strlen(str) > 8 && strncmp("file:///", str, 8) == 0)


Rather than having the literal 8's, how about something like:

#define FILE_URI_PREFIX "file:///" #define FILE_URI_PREFIX_LEN strlen(FILE_URI_PREFIX)

Also I'm not sure how valuable the "len>8" check is, since file:/// is just as wrong as file:///etc, but we aren't checking for that here.

This function is something I was trying to think of ways to improve or possibly remove. Realistically, zfs_prop_valid_keylocation() is just a sanity check. The create, clone, and change-key code all will actually need to load the key from the given keylocation before they can even attempt to talk to the kernel. This is all done in userspace, however, so the kernel can't really be sure if the keylocation it receives is valid. As a result, I added this function as a quick smoke test for the kernel. Can you think of a better way to check this?

ahrens · Feb 4, 2017

module/zfs/dmu_send.c

@@ -1527,13 +1536,14 @@ dmu_recv_begin_sync(void *arg, dmu_tx_t *tx)
 	const char *tofs = drba->drba_cookie->drc_tofs;
 	dsl_dataset_t *ds, *newds;
 	uint64_t dsobj;
+	int flags = DS_HOLD_FLAG_DECRYPT;


maybe name this dsflags for consistency with dmu_recv_begin_check()?

I only called it dsflags there because flags was already being used. Would you like me to change this everywhere?

ahrens · Feb 4, 2017

module/zfs/dmu_send.c

@@ -2454,14 +2466,16 @@ receive_free(struct receive_writer_arg *rwa, struct drr_free *drrf)
 static void
 dmu_recv_cleanup_ds(dmu_recv_cookie_t *drc)
 {
+	int flags = DS_HOLD_FLAG_DECRYPT;


see comment above

ahrens · Feb 4, 2017

module/zfs/dmu_send.c

 	 */
-	dsl_dataset_disown(drc->drc_ds, dmu_recv_tag);
+	(void) spa_keystore_remove_mapping(dmu_tx_pool(tx)->dp_spa,
+	    drc->drc_ds, drc->drc_ds);


this is so that zfs receive unloads the key?

So that it releases the key mapping hold it created in dmu_recv_begin(). Loading and unloading is for wrapping keys, technically.

ahrens · Feb 4, 2017

module/zfs/dmu_traverse.c

@@ -196,8 +197,11 @@ traverse_prefetch_metadata(traverse_data_t *td,
 	if (BP_GET_LEVEL(bp) == 0 && BP_GET_TYPE(bp) != DMU_OT_DNODE)
 		return;

+	if ((td->td_flags & TRAVERSE_NO_DECRYPT) && BP_IS_ENCRYPTED(bp))
+		zio_flags |= ZIO_FLAG_RAW;


It might be worth a comment somewhere that this is all about dnode blocks (since traversal doesn't read any other encrypted blocks).

ahrens · Feb 4, 2017

module/zfs/dmu_traverse.c

 		int32_t i;
 		int32_t epb = BP_GET_LSIZE(bp) >> DNODE_SHIFT;
 		dnode_phys_t *child_dnp;

+		/*
+		 * dnode blocks might have their bonus buffers encrypted, so
+		 * we must be careful to honor TRAVERSE_NO_DECRYPT


should we assert that other cases here are not encrypted (and therefore it's OK to ignore NO_DECRYPT)?

sure. will fix.

Thank you very much to everyone who has looked at and commented on this PR. As some of you may have noticed, this page is getting a bit too big for github to handle effectively so I will be moving it to a new PR in a few minutes. I will link it here when it has been made.

tcaputi · Feb 9, 2017

Thank you very much to everyone who has looked at and commented on this PR. As some of you may have noticed, this page is getting a bit too big for github to handle effectively so I will be moving it to a new PR in a few minutes. I will link it here when it has been made.

The new PR has been opened at #5769. Please leave any future comments and reviews there

tcaputi · Feb 9, 2017

The new PR has been opened at #5769. Please leave any future comments and reviews there

tcaputi referenced this pull request Feb 18, 2016
Closed
Changes needed for ZFS Encryption #533

grahamperrin referenced this pull request Feb 26, 2016
Closed
ZFS Crypto support #494

tcaputi changed the title from [WIP][comments] ZFS Encryption Keystore to ZFS Encryption Keystore May 4, 2016

tcaputi changed the title from ZFS Encryption Keystore to ZFS Encryption May 4, 2016

behlendorf added Feature OpenZFS Review labels May 9, 2016

Native data and metadata encryption for zfs

Loading status checks…

89b4e7a

added assert for sync-to-convergence, small man page fixes, removed Z…
…IL-specific IV derivation

Loading status checks…

2f7c9cb

sempervictus referenced this pull request Jan 11, 2017
Open
ICP module breaks in-tree kernel+zfs builds #5581

tcaputi closed this Feb 9, 2017

tcaputi referenced this pull request Feb 9, 2017
Merged
ZFS Encryption #5769

behlendorf removed this from In Progress in 0.7.0-rc4 Mar 20, 2017

zfsonlinux/zfs

Join GitHub today

ZFS Encryption #4329

Conversation

tcaputi commented Feb 11, 2016 • edited Edited 1 time tcaputi edited Jan 4, 2017 (most recent)

Native encryption in zfsonlinux (See issue #494)

Implementation details that should be looked at

This comment has been minimized.

tcaputi commented Feb 12, 2016

This comment has been minimized.

tcaputi commented Feb 12, 2016

This comment has been minimized.

ilovezfs commented Feb 12, 2016

This comment has been minimized.

tcaputi commented Feb 12, 2016

This comment has been minimized.

ryao commented Feb 13, 2016

This comment has been minimized.

ilovezfs commented Feb 13, 2016

This comment has been minimized.

tcaputi commented Feb 13, 2016

This comment has been minimized.

tcaputi commented Feb 16, 2016

This comment has been minimized.

behlendorf commented Feb 17, 2016

This comment has been minimized.

tcaputi commented Feb 18, 2016

This comment has been minimized.

l1k commented Feb 18, 2016

This comment has been minimized.

tcaputi commented Feb 18, 2016

tcaputi referenced this pull request Feb 18, 2016

This comment has been minimized.

tcaputi commented Feb 18, 2016

This comment has been minimized.

behlendorf commented Feb 19, 2016

This comment has been minimized.

tcaputi commented Feb 19, 2016

This comment has been minimized.

tcaputi commented Feb 21, 2016

This comment has been minimized.

behlendorf commented Feb 25, 2016

ryao reviewed Feb 26, 2016 View changes

This comment has been minimized.

tcaputi commented Feb 26, 2016

grahamperrin referenced this pull request Feb 26, 2016

This comment has been minimized.

tcaputi commented Mar 7, 2016

tcaputi changed the title from [WIP][comments] ZFS Encryption Keystore to ZFS Encryption Keystore May 4, 2016

tcaputi changed the title from ZFS Encryption Keystore to ZFS Encryption May 4, 2016

behlendorf added Feature OpenZFS Review labels May 9, 2016

This comment has been minimized.

sammcj commented May 15, 2016

This comment has been minimized.

FransUrbo commented May 16, 2016

This comment has been minimized.

tcaputi commented May 16, 2016

This comment has been minimized.

FransUrbo commented May 16, 2016

This comment has been minimized.

FransUrbo commented May 16, 2016

This comment has been minimized.

tcaputi commented May 16, 2016

This comment has been minimized.

FransUrbo commented May 16, 2016

This comment has been minimized.

FransUrbo commented May 16, 2016

This comment has been minimized.

tcaputi commented May 16, 2016 • edited Edited 1 time tcaputi edited May 16, 2016 (most recent)

This comment has been minimized.

FransUrbo commented May 16, 2016

This comment has been minimized.

FransUrbo commented May 16, 2016

This comment has been minimized.

FransUrbo commented May 16, 2016

This comment has been minimized.

behlendorf commented May 16, 2016

This comment has been minimized.

lundman commented May 17, 2016

This comment has been minimized.

tcaputi commented Feb 11, 2016 •

edited

Edited 1 time

tcaputi edited Jan 4, 2017 (most recent)

ryao reviewed Feb 26, 2016

View changes

tcaputi commented May 16, 2016 •

edited

Edited 1 time

tcaputi edited May 16, 2016 (most recent)

Kokokokoka commented Jan 3, 2017 •

edited

Edited 1 time

Kokokokoka edited Jan 3, 2017 (most recent)

ahrens reviewed Jan 6, 2017

View changes

ahrens requested changes Jan 14, 2017

View changes