auto-apply --raw send flag (ZREP_SEND_FLAGS) on encrypted datasets

[onlime]

We need some auto-detection of encrypted zfs datasets so that --raw zfs send flag is added without having to explicitly set ZREP_SEND_FLAGS environment variable. As we usually want to replicate all datasets managed by zrep using zrep sync -f all, we could not use different ZREP_SEND_FLAGS (with or without --raw) for individual datasets.

To be on the safe side, this patch only sets ZREP_SEND_FLAGS=--raw if the environment variable was not defined / was empty.

[ppbrown]

Thank you for taking the time to write this patch. however, there are multiple issues with it, which do not fit into my standards. One of which being that it would make more sense to me to check if encryption is enabled, rather than double negative. Please open up a feature request issue for this if y ou wish to see the functionality enabled in zrep. Additionally, it would be very helpful to me, if you could do a little research. In the major versions of ZFS (solaris, freebsd, zfsonlinux) it would be important to learn if the --raw option was released at the same time that the encryption functionality was there. If not, then there is a problem. I cant go automatically adding the option, if the version of zfs doesnt actually SUPPORT the option. so either additional feature checking needs to be done, or an additional flag needs to be set. Perhaps export ZREP_ENCRYPTRAW which would signify to auto add raw, only if encryption is present. And perhaps this is the best feature request to ask for in the first place.

…

On Wed, Dec 11, 2019 at 11:56 PM Philip Iezzi ***@***.***> wrote: We need some auto-detection of encrypted zfs datasets so that --raw zfs send flag is added without having to explicitly set ZREP_SEND_FLAGS environment variable. As we usually want to replicate all datasets managed by zrep using zrep sync -f all, we could not use different ZREP_SEND_FLAGS (with or without --raw) for individual datasets. To be on the safe side, this patch only sets ZREP_SEND_FLAGS=--raw if the environment variable was not defined / was empty. ------------------------------ You can view, comment on, or merge this pull request online at: #144 Commit Summary - auto-apply --raw send flag on encrypted datasets File Changes - *M* zrep <https://github.com/bolthole/zrep/pull/144/files#diff-0> (14) - *M* zrep_snap <https://github.com/bolthole/zrep/pull/144/files#diff-1> (7) - *M* zrep_sync <https://github.com/bolthole/zrep/pull/144/files#diff-2> (7) Patch Links: - https://github.com/bolthole/zrep/pull/144.patch - https://github.com/bolthole/zrep/pull/144.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#144?email_source=notifications&email_token=AANEV6PEIFFD3RHRO65XQA3QYHVBFA5CNFSM4JZ2NKA2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H77EKOA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANEV6O4QNH6ORZCEUWRMITQYHVBFANCNFSM4JZ2NKAQ> .

[darkpixel]

What about the use-case where the local filesystem is encrypted and the remote filesystem uses the exact same key and has it loaded. No need to pass --raw.

[ppbrown]

please describe in detail a comparison of what would happen (at the zfs level only) in that situation, with and without raw.

…

On Thu, Dec 12, 2019 at 11:10 AM Aaron C. de Bruyn ***@***.***> wrote: What about the use-case where the local filesystem is encrypted and the remote filesystem uses the exact same key and has it loaded. No need to pass --raw. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#144?email_source=notifications&email_token=AANEV6N7HRTNL32MKYWO73LQYKEBNA5CNFSM4JZ2NKA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGXWJAA#issuecomment-565142656>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANEV6J472WGFP2TYGA3Y2TQYKEBNANCNFSM4JZ2NKAQ> .

[darkpixel]

Quoth the manpage:

-w, --raw
    For encrypted datasets, send data exactly as it exists on disk. This allows backups to
    be taken >>even if encryption keys are not currently loaded<<. The backup may then
    be received on an untrusted machine since that machine will not have the encryption
    keys to read the protected data or alter it without being detected. Upon being received,
    the dataset will have the same encryption keys as it did on the send side, although the
    keylocation property will be defaulted to prompt if not otherwise provided. For
    unencrypted datasets, this flag will be equivalent to -Lec.  Note that if you do not use
    this flag for sending encrypted datasets, data will be sent unencrypted and may be
    re-encrypted with a different encryption key on the receiving system, which will disable
    the ability to do a raw send to that system for incrementals.

Basically the raw flag lets you send encrypted datasets to the receiver without sending the key.
Without the raw flag, you're sending the decrypted data to the remote. The remote may store it unencrypted, or if it's going to a pool or dataset that has encryption set up, it will encrypt it using the receiver's encryption/key setup.

[ppbrown]

thats about what I thought/ guessed. So to answer your question: if both sides have the key, there's no benefit in NOT using raw. It will just burn time decrypting and recrypting. Seems to me, the only time you would NOT want to use --raw on an encrypted filesystem, would be when you want to store a non-disk-encrypted version on the other side. But that makes no sense to do. If you have some requirement to "encrypt the data at rest" locally.. how would you NOT have the same requirement on the other side?? but at the same time, I dont want to default to using --raw. Since I'd have to make zrep do extra detective work to see if it is supported, and then extra extra to see if the file is encrypted in the first place. Wait... To the original poster... Why EXACTLY dont you just have --raw in your ZREP_SEND_FLAGS all the time? That is not made clear to me.

…

On Thu, Dec 12, 2019 at 9:54 PM Aaron C. de Bruyn ***@***.***> wrote: Quoth the manpage: -w, --raw For encrypted datasets, send data exactly as it exists on disk. This allows backups to be taken >>even if encryption keys are not currently loaded<<. The backup may then be received on an untrusted machine since that machine will not have the encryption keys to read the protected data or alter it without being detected. Upon being received, the dataset will have the same encryption keys as it did on the send side, although the keylocation property will be defaulted to prompt if not otherwise provided. For unencrypted datasets, this flag will be equivalent to -Lec. Note that if you do not use this flag for sending encrypted datasets, data will be sent unencrypted and may be re-encrypted with a different encryption key on the receiving system, which will disable the ability to do a raw send to that system for incrementals. Basically the raw flag lets you send encrypted datasets to the receiver without sending the key. Without the raw flag, you're sending the decrypted data to the remote. The remote may store it unencrypted, or if it's going to a pool or dataset that has encryption set up, it will encrypt it using the receiver's encryption/key setup. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#144?email_source=notifications&email_token=AANEV6KCSH5D64R453ULUHLQYMPP5A5CNFSM4JZ2NKA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGY7K5Q#issuecomment-565310838>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANEV6P4HCSYE27Y7JC37JDQYMPP5ANCNFSM4JZ2NKAQ> .

[onlime]

Wait... To the original poster... Why EXACTLY dont you just have --raw in your ZREP_SEND_FLAGS all the time? That is not made clear to me.

As I tried to explain in my first comment:

As we usually want to replicate all datasets managed by zrep using zrep sync -f all, we could not use different ZREP_SEND_FLAGS (with or without --raw) for individual datasets.

To be more specific: We have like 10 unencrypted and only 2 encrypted datasets. If we use ZREP_SEND_FLAGS=--raw this is applied to all zrep replication, as we want to use the luxury of your all which detects zrep managed datasets. It would be uncool to use zrep list and loop over those datasets by ourselves (in a wrapper script) and only apply --raw on encrypted datasets. Or are you trying to say that using --raw on unencrypted datasets does no harm at all and we shouldn't care about overusing it for all replications?

Our setup looks like this:

# Create encrypted zpool
$ zpool create -O acltype=posixacl -O encryption=on -O keylocation=prompt -O keyformat=passphrase epool sdf
$ zfs get encryption,keylocation,keystatus,keyformat epool
NAME   PROPERTY     VALUE        SOURCE
epool  encryption   aes-256-ccm  -
epool  keylocation  prompt       local
epool  keystatus    available    -
epool  keyformat    passphrase   -

# Create dataset inside epool
$ zfs create epool/subvol-test

epool exists on both host nodes, using the same key (passphrase), being correctly mounted. zrep initializing reports:

hn2$ zrep -i epool/subvol-test hn1 epool/subvol-test
Setting zrep properties on epool/subvol-test
Creating snapshot epool/subvol-test@zrep_000000
Sending initial replication stream to hn1:epool/subvol-test
cannot send epool/subvol-test@zrep_000000: encrypted dataset epool/subvol-test may not be sent with properties without the raw flag
cannot receive: failed to read from stream
Destroying any zrep-related snapshots from epool/subvol-test
Removing zrep-related properties from epool/subvol-test
Error: Error transferring epool/subvol-test@zrep_000000 to hn1:epool/subvol-test. Resetting

There should be no need to pass --raw if encryption keys are the same on both hosts and they are loaded. Besides, this works flawlessly (without passing --raw) on ZFS volumes with fixed volsize and btrfs filesystem inside (which should not make any difference, as ZFS does not know about underlying data).

So maybe we should not develop such a workaround like I have proposed, but rather figure out why zrep asks for --raw while it is actually not required in this use case.

[ppbrown]

“Or are you trying to say that using --raw on unencrypted datasets does no harm at all and we shouldn't care about overusing it for all replications?” that is my GUESS, yes. so it would be good if you tested it.

[darkpixel]

@onlime Yeah, you can use --raw regardless of the dataset being encrypted or not.
From that man page snippet: For unencrypted datasets, this flag will be equivalent to -Lec.

So you can definitely use ZREP_SEND_FLAGS="--raw" for all of them.

@ppbrown I'll give real-world use-cases for each example.

Encrypted source uses --raw to send to the destination to ensure the destination can't access the data. This is how we back up to the cloud and to our datacenter. We don't store the keys in the cloud so no one can get the keys/data.

Encrypted source does not use --raw to send to the destination since they destination has it's own encryption. This is how we back up to our external USB drives since they have a different encryption key.

Not encrypted source does not use --raw to send to encrypted destination. We have a few boxes that don't use ZFS-native encryption (they have encryption at the LUKS layer and those LUKS devices are turned into a zpool) but need to be backed up in an encrypted format to the destination (be it a locally attached USB drive or our cloud/datacenter).

Basically I don't think zrep should be trying to figure out if it needs to use --raw based on encryption properties. Encryption properties can't tell you the whole story. :)