Repair encryption hierarchy of 'send -Rw | recv -d' datasets that do not bring their encryption root

[almereyda]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	20.04
Linux Kernel	5.4.0-72-generic
Architecture	x86_64
ZFS Version	0.8.3-1ubuntu12.7
SPL Version	0.8.3-1ubuntu12.6

Describe the problem you're observing

Following up from #6624 and #9273 it happens that a certain Long-Term Support release from Ubuntu that natively ships with ZFS 0.8.1 (meanwhile updated to 0.8.3) can still produce datasets that have a broken encryption hierarchy, despite all appears to be well at first.

In #6624 (comment) @tcaputi speaks of:

I can probably provide you some code for a small binary that will call the FORCE NEW KEY command (the opposite of FORCE INHERIT) which should allow you to mount the dataset again.

Is it possible to apply this workaround in a way to restore the ability to mount the datasets? For this case, let's consider the source is not available anymore for retransmission. This is due to a juggling of datasets for downsizing a pool in a mirrored setup, where the source has been replaced by a smaller version of itself.

Two things are odd here:

1. The dataset seems to have a valid encryption key and states its key to be available
2. encryptionroot is a read-only value. How is it possible to re-set it?

Describe how to reproduce the problem

1. create two encrypted zpools and fill the first datasets
   • use the same passphrase
2. recursively snapshot the first pool
3. raw send the first level of datasets into the second pool w/ stripping the name of the source
4. recreate the first pool (destroy, create)
   • use the same passphrase
5. raw send the first level of datasets into the first pool
6. the datasets at the target are not mountable

Messages from the logs

zpool create -o ashift=12 -o autotrim=on -O atime=off -O dedup=off -O compression=lz4 -O acltype=posixacl -O xattr=sa -O normalization=formD -O dnodesize=auto -O encryption=on -O keyformat=passphrase -O keylocation=prompt -m /pool1 pool1 /dev/sda3
zpool create -o ashift=12 -o autotrim=on -O atime=off -O dedup=off -O compression=lz4 -O acltype=posixacl -O xattr=sa -O normalization=formD -O dnodesize=auto -O encryption=on -O keyformat=passphrase -O keylocation=prompt -m /pool2 pool2 /dev/sdb3
zfs create pool1/test
zfs create pool1/test/temp
zfs snapshot -r pool1@snap
zfs send -Rw pool1/test@snap | zfs recv -d pool2
zpool destroy pool1
cfdisk /dev/sda # to downsize the partition
zpool create ... pool1 /dev/sda3
zfs send -Rw pool2/test@snap | zfs recv -d pool1

$ zfs get encryption,encryptionroot,keystatus pool1
NAME       PROPERTY        VALUE        SOURCE
pool1  encryption      aes-256-gcm  -
pool1  encryptionroot  pool1    -
pool1  keystatus       available    -

$ zfs get encryption,encryptionroot,keystatus pool1/test
NAME            PROPERTY        VALUE        SOURCE
pool1/test  encryption      aes-256-gcm  -
pool1/test  encryptionroot  pool1    -
pool1/test  keystatus       available    -

$ zfs load-key pool1
Key load error: Key already loaded for 'pool1'.

$ zfs load-key pool1/test
Key load error: Keys must be loaded for encryption root of 'pool1/test' (pool1).

$ zfs mount pool1/test
filesystem 'pool1/test' can not be mounted: Permission denied
cannot mount 'pool1/test': Invalid argument

Additional information

• ZoL 0.8.0 encryption: don't encrypt the pool root! : zfs
  suggests never to encrypt the primary dataset of a pool, thus the pool, but to create encrypted datasets as malleable encryption roots below
• Similar to the comment in Filesystem created via zfs send --raw between encrypted roots fails to mount #6624 (comment) we are in course of a migration and blocked by this issue.
• Am I right in assuming that encryption keys are derived from the passphrase, or did I accidentally remove all possibilities to decrypt the datasets by removing their encryption root?

This is also discussed in:

• Cannot mount raw sent encrypted root dataset | Topicbox

Interestingly running zfs load-key -r pool1 will not try to load the keys for the datasets for which keys are available (via their encryption root pool1), yet still they remain unmountable. Also some of the descendent encryption roots that Docker apparently created are decryptable, but others aren't. This gives hope, next to the key not being rejected for the other datasets, that all USERDATA is still available and recoverable.

Some datasets decrypt, others don't, especially from Docker.

$ zpool export pool1
$ zpool import -l pool1
Enter passphrase for 'pool1': 
Enter passphrase for 'pool1/DOCKER/lib/697917a208c21f64616d09db164318acd0f20e4d1fffa7e4e66f8de95fa47bb2': 
Enter passphrase for 'pool1/DOCKER/lib/79944167f45fa8346086f0ff167a2d423bdb4f7d9ded72e9c401f64a15e49ac8': 
Enter passphrase for 'pool1/DOCKER/lib/33a48ff62d295e54f5d83675ad75b833f75713a7c3d7b56a89c69b08aceb9d0f': 
Enter passphrase for 'pool1/DOCKER/lib/1fb15fd5d836f0dbbe60645d532f4a13225704f790d64510e5785e841a6a9dfb': 
Enter passphrase for 'pool1/DOCKER/lib/f9bc3e5084268da84adc5a2e17e5f894dd1b3d7e5c8ac9bc7f947ce30c4c8a40': 
Enter passphrase for 'pool1/DOCKER/lib/130b413032e79c2a252a83dc7226f3f4106536074c2cc7c2729dc85e2959712a': 
Enter passphrase for 'pool1/DOCKER/lib/eae4f3bd5bea07d25f88c7fc2ffd6324b595ea067162e6ed67d0c5f16097dd56': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/eae4f3bd5bea07d25f88c7fc2ffd6324b595ea067162e6ed67d0c5f16097dd56'.
Enter passphrase for 'pool1/DOCKER/lib/eae4f3bd5bea07d25f88c7fc2ffd6324b595ea067162e6ed67d0c5f16097dd56': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/eae4f3bd5bea07d25f88c7fc2ffd6324b595ea067162e6ed67d0c5f16097dd56'.
Enter passphrase for 'pool1/DOCKER/lib/eae4f3bd5bea07d25f88c7fc2ffd6324b595ea067162e6ed67d0c5f16097dd56': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/eae4f3bd5bea07d25f88c7fc2ffd6324b595ea067162e6ed67d0c5f16097dd56'.
Enter passphrase for 'pool1/DOCKER/lib/683b04ac797d220ea673d705db7246e9254cb24f5f14171f251b3e45012cb93b': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/683b04ac797d220ea673d705db7246e9254cb24f5f14171f251b3e45012cb93b'.
Enter passphrase for 'pool1/DOCKER/lib/683b04ac797d220ea673d705db7246e9254cb24f5f14171f251b3e45012cb93b': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/683b04ac797d220ea673d705db7246e9254cb24f5f14171f251b3e45012cb93b'.
Enter passphrase for 'pool1/DOCKER/lib/683b04ac797d220ea673d705db7246e9254cb24f5f14171f251b3e45012cb93b': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/683b04ac797d220ea673d705db7246e9254cb24f5f14171f251b3e45012cb93b'.
Enter passphrase for 'pool1/DOCKER/lib/291c9a86a0bfdcd8546cff5b6edf7805f6cd548f04ae799ffbea333faf18c8b2': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/291c9a86a0bfdcd8546cff5b6edf7805f6cd548f04ae799ffbea333faf18c8b2'.
Enter passphrase for 'pool1/DOCKER/lib/291c9a86a0bfdcd8546cff5b6edf7805f6cd548f04ae799ffbea333faf18c8b2': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/291c9a86a0bfdcd8546cff5b6edf7805f6cd548f04ae799ffbea333faf18c8b2'.
Enter passphrase for 'pool1/DOCKER/lib/291c9a86a0bfdcd8546cff5b6edf7805f6cd548f04ae799ffbea333faf18c8b2': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/291c9a86a0bfdcd8546cff5b6edf7805f6cd548f04ae799ffbea333faf18c8b2'.
Enter passphrase for 'pool1/DOCKER/lib/968ecf6144a9ac309982de243e23e0bc782f77fd68941248b5f3c827463e5368': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/968ecf6144a9ac309982de243e23e0bc782f77fd68941248b5f3c827463e5368'.
Enter passphrase for 'pool1/DOCKER/lib/968ecf6144a9ac309982de243e23e0bc782f77fd68941248b5f3c827463e5368': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/968ecf6144a9ac309982de243e23e0bc782f77fd68941248b5f3c827463e5368'.
Enter passphrase for 'pool1/DOCKER/lib/968ecf6144a9ac309982de243e23e0bc782f77fd68941248b5f3c827463e5368': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/968ecf6144a9ac309982de243e23e0bc782f77fd68941248b5f3c827463e5368'.
Enter passphrase for 'pool1/DOCKER/lib/dc45835eb73741e53f87fc316342fae82a41f2c7421714ebefc6a2d5b04cebbb': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/dc45835eb73741e53f87fc316342fae82a41f2c7421714ebefc6a2d5b04cebbb'.
Enter passphrase for 'pool1/DOCKER/lib/dc45835eb73741e53f87fc316342fae82a41f2c7421714ebefc6a2d5b04cebbb': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/dc45835eb73741e53f87fc316342fae82a41f2c7421714ebefc6a2d5b04cebbb'.
Enter passphrase for 'pool1/DOCKER/lib/dc45835eb73741e53f87fc316342fae82a41f2c7421714ebefc6a2d5b04cebbb': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/dc45835eb73741e53f87fc316342fae82a41f2c7421714ebefc6a2d5b04cebbb'.
Enter passphrase for 'pool1/DOCKER/lib/3071a1943c5d067abbfea06ff3721dd6aac6ff56c25ad83c9a65281a115326bc': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/3071a1943c5d067abbfea06ff3721dd6aac6ff56c25ad83c9a65281a115326bc'.
Enter passphrase for 'pool1/DOCKER/lib/3071a1943c5d067abbfea06ff3721dd6aac6ff56c25ad83c9a65281a115326bc': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/3071a1943c5d067abbfea06ff3721dd6aac6ff56c25ad83c9a65281a115326bc'.
Enter passphrase for 'pool1/DOCKER/lib/3071a1943c5d067abbfea06ff3721dd6aac6ff56c25ad83c9a65281a115326bc': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/3071a1943c5d067abbfea06ff3721dd6aac6ff56c25ad83c9a65281a115326bc'.
Enter passphrase for 'pool1/DOCKER/lib/52bfa76983a9904e7e3cfbc911c1c2e0e92d69ee26467e46f002a8de614dc6d9': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/52bfa76983a9904e7e3cfbc911c1c2e0e92d69ee26467e46f002a8de614dc6d9'.
Enter passphrase for 'pool1/DOCKER/lib/52bfa76983a9904e7e3cfbc911c1c2e0e92d69ee26467e46f002a8de614dc6d9': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/52bfa76983a9904e7e3cfbc911c1c2e0e92d69ee26467e46f002a8de614dc6d9'.
Enter passphrase for 'pool1/DOCKER/lib/52bfa76983a9904e7e3cfbc911c1c2e0e92d69ee26467e46f002a8de614dc6d9': 
Key load error: Incorrect key provided for 'pool1/DOCKER/lib/52bfa76983a9904e7e3cfbc911c1c2e0e92d69ee26467e46f002a8de614dc6d9'.
7 / 14 keys successfully loaded
filesystem 'pool1/DOCKER/lib' can not be mounted: Permission denied
cannot mount 'pool1/DOCKER/lib': Invalid argument
filesystem 'pool1/USERDATA/bases' can not be mounted: Permission denied
cannot mount 'pool1/USERDATA/bases': Invalid argument
filesystem 'pool1/USERDATA/lake' can not be mounted: Permission denied
cannot mount 'pool1/USERDATA/lake': Invalid argument
filesystem 'pool1/BACKUP/bases' can not be mounted: Permission denied
cannot mount 'pool1/BACKUP/bases': Invalid argument
filesystem 'pool1/ROOT/system' can not be mounted: Permission denied
cannot mount 'pool1/ROOT/system': Invalid argument
filesystem 'pool1/ROOT/shell' can not be mounted: Permission denied
cannot mount 'pool1/ROOT/shell': Invalid argument

Further mentions of similar behaviours are:

• Having Trouble Using Send/Receive on Encrypted Dataset : zfs offers a way to first replicate the encryption root, and then all following
• Verschlüsselte zfs-Datasets übertragen (send/receive) | TechGoat.net does not use -d
• Difficulty replicating native ZFS encrypted pool to new pool. | TrueNAS Community uses -F and also replicates encryption roots distinctively
• linux - Can't mount ZFS or change encryption key after raw receive - Unix & Linux Stack Exchange suggests to use change-key -i
  It is not possible to set a new keylocation for dependant datasets in the encryption hierarchy.

  $ zfs set keylocation=prompt pool1/TRASH
  cannot set property for 'pool1/TRASH': keylocation may only be set on encryption roots
  

Following on from #6847 (comment) also doesn't help in this case, where the encryption root has been replaced:

$ zfs change-key -i pool1/ROOT      
Key change error: Key inheritting can only be performed on encryption roots.

$ zfs change-key -i pool1     
Key change error: Root dataset cannot inherit key.

The pool itself does not report any data errors:

$ zpool status pool1          
  pool: pool1
 state: ONLINE
  scan: resilvered 11.1G in 0 days 00:00:39 with 0 errors on Wed May  5 04:23:34 2021
config:

        NAME                                                     STATE     READ WRITE CKSUM
        pool1                                                ONLINE       0     0     0
          mirror-0                                               ONLINE       0     0     0
            sda3  ONLINE       0     0     0
            sdb3  ONLINE       0     0     0

errors: No known data errors

May it be possible to reintroduce a new encryption root for those datasets that don't mount currently, eventually by decomposing and reconstructing the pool into two and back again, or are there any other known workarounds I am not aware of, yet?

Thank you for your kind help, our users will appreciate it.

Reflection and conclusion

Reading the linked references brings up their reasoning and questioning again:

1. Is it neccessary that the first level of a zpool is also a dataset in itself (that can be encrypted)?
2. Is it considered harmful to directly encrypt the root dataset of a zpool imlicitly upon creation?

Which other good practices are known to omit the side effects and edge cases we produce here, other than creating independently encrypted datasets under the root dataset of the pool that act as encryption roots for their descendants, and leaving that itself unencrypted?

When trying to do send | recv -x encryption, the command output complains about a missing raw flag, while we have migrated datasets from unencrypted pools into encrypted ones before. Now this looks even more bestranging:

$ zfs send pool1/USERDATA/lake@downsize | zfs recv -x encryption pool2/TRASH/lake                                          
warning: cannot send 'pool1/USERDATA/lake@downsize': source key must be loaded
cannot receive: failed to read from stream

$ zfs get keystatus -r pool1/USERDATA/lake                                           
NAME                              PROPERTY   VALUE        SOURCE
pool1/USERDATA/lake           keystatus  available    -
pool1/USERDATA/lake@downsize  keystatus  available    -

Is the source key actually available and loaded, or is it not? From the given output of the commands, for me it is not possible to depict anymore.

---

Reading through zfs-receive.8 makes me guess that we now probably have different initialization vectors (IV) for the AEAD cipher in the pool1 dataset and its descendants, why recreating the encrypted zpool with the same passphrase does not mean it can act as an encryption root for its newly retrieved childs.

• Given this hypothesis is true; would it be possible to extract and reintroduce the IV from the child datasets to the new encryption root, and allow the key (= passphrase + random IV?) to decrypt all datasets again?
• In replicating only descendant datasets below a the zpool's root dataset that also acted as encryption root, did I accidentally omit to also transfer the salt that is used and needed to derive the encryption key from the passphrase?

[secabeen]

I think this is a combination of a documentation shortfall, and an opportunity for a protecting users by testing IVs when receiving.

On the documentation side, the Encryption section of the ZFS manpage should stress that critical encryption information is stored within the encryptionroot, and that destroying all copies of an encryptionroot will cause data loss.

On the protection side, perhaps zfs receive should warn or fail whenever an encryptionroot is discarded using the '-d' argument to zfs receive. Additionally, zfs receive could compare the IV of existing encryptionroots to the received encryptionroot and ensure they were/are the same if encryptionroot inheritance is being preserved.

[bghira]

the handling of -o encryption is woefully inadequate and one of those things that was planned to revisit, but Datto no longer seems to be working on this.

[almereyda]

zfs receive should warn or fail whenever an encryptionroot is discarded

That would be an option that could have helped me assess the situation better, prior to discarding the source pool. Then as a consequence, ultimately it appears to me -d is primarily an option for backing up datasets that will eventually be restored into their source again.

Additionally it now comes to my mind, we also need a mechanism to back up and restore encryption headers, IVs and keys, as we would do with any LUKS device as well. Wasn't this something the kind of binary @tcaputi brought up could have helped with?

[secabeen]

Additionally it now comes to my mind, we also need a mechanism to back up and restore encryption headers, IVs and keys, as we would do with any LUKS device as well. Wasn't this something the kind of binary @tcaputi brought up could have helped with?

I would guess that a zfs send of the encryptionroot dataset would include the headers and IVs necessary for any dataset that used that encryption root, but it would also include all of the data in that dataset. I always keep my encryptionroot datasets empty of data and unmounted, just like my pool root datasets, but not everyone does that.

[brenc]

Just started using ZoL with native encryption and think I have hit the same or a similar bug (related to #6624 as well).

truncate -s 100M /root/src.img
truncate -s 100M /root/replica.img

zpool create src /root/src.img
zpool create replica /root/replica.img

zfs create -o encryption=on -o keyformat=passphrase -o keylocation=prompt src/encrypted
zfs create -o encryption=on -o keyformat=passphrase -o keylocation=prompt replica/encrypted

zfs create src/encrypted/a

dd if=/dev/urandom of=/src/encrypted/a/test1.bin bs=1M count=1

zfs snap src/encrypted/a@test1

zfs send -Rvw src/encrypted/a@test1 | zfs receive -svF replica/encrypted/a

zfs mount -l replica/encrypted

zfs mount -l replica/encrypted/a

zfs change-key -i replica/encrypted/a

zfs umount -u replica/encrypted

zfs mount -l replica/encrypted

zfs mount replica/encrypted/a

All good at this point. Everything works as expected. Now, do an incremental send:

dd if=/dev/urandom of=/src/encrypted/a/test2.bin bs=1M count=1

zfs snap src/encrypted/a@test2

zfs send -RvwI @test1 src/encrypted/a@test2 | zfs receive -svF replica/encrypted/a

# ls -al /replica/encrypted/a/
total 2056
drwxr-xr-x 2 root root       4 Sep 26 03:59 .
drwxr-xr-x 3 root root       3 Sep 26 03:57 ..
-rw-r--r-- 1 root root 1048576 Sep 26 03:55 test1.bin
-rw-r--r-- 1 root root 1048576 Sep 26 03:59 test2.bin

Again, all good. Now unmount/mount:

zfs umount -u replica/encrypted

zfs mount -l replica/encrypted

# zfs get encryptionroot,keystatus -rt filesystem replica/encrypted
NAME                 PROPERTY        VALUE              SOURCE
replica/encrypted    encryptionroot  replica/encrypted  -
replica/encrypted    keystatus       available          -
replica/encrypted/a  encryptionroot  replica/encrypted  -
replica/encrypted/a  keystatus       available          -

# zfs mount -l replica/encrypted/a
cannot mount 'replica/encrypted/a': Permission denied

Yikes! This appears to have corrupted 10TB of backup filesystems. I've been trying to recover from this but no luck so far.

If I don't run change-key then I can send incrementals, unmount, and mount no problem (I just have to enter the password in twice). If I run change-key then unmount/mount still no problem. It's when I run change-key and then send an incremental snapshot that seems to render the filesystem unmountable.

After running change-key and sending an incremental, once the filesystem is unmounted it can't be mounted again. It looks like the encryption root absolutely has to be replicated to prevent this from happening. If I replicate the encryption root then everything works as expected.

I may have also uncovered another bug in trying to recover from this. If I run zfs change-key -o keylocation=prompt -o keyformat=passphrase replica/encrypted/a, after entering the new passwords the command hangs forever due to a panic. I have to completely reset the system.

[ 7080.228309] VERIFY3(0 == spa_keystore_dsl_key_hold_dd(dp->dp_spa, dd, FTAG, &dck)) failed (0 == 13)
[ 7080.228369] PANIC at dsl_crypt.c:1450:spa_keystore_change_key_sync_impl()
[ 7080.228399] Showing stack for process 1120
[ 7080.228403] CPU: 2 PID: 1120 Comm: txg_sync Tainted: P           O      5.11.0-36-generic #40-Ubuntu
[ 7080.228406] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
[ 7080.228408] Call Trace:
[ 7080.228414]  show_stack+0x52/0x58
[ 7080.228424]  dump_stack+0x70/0x8b
[ 7080.228431]  spl_dumpstack+0x29/0x2b [spl]
[ 7080.228448]  spl_panic+0xd4/0xfc [spl]
[ 7080.228459]  ? dsl_wrapping_key_rele.constprop.0+0x12/0x20 [zfs]
[ 7080.228597]  ? spa_keystore_dsl_key_hold_dd+0x1a8/0x200 [zfs]
[ 7080.228687]  spa_keystore_change_key_sync_impl+0x3c0/0x3d0 [zfs]
[ 7080.228776]  ? zap_lookup+0x16/0x20 [zfs]
[ 7080.228899]  spa_keystore_change_key_sync+0x157/0x3c0 [zfs]
[ 7080.228988]  ? dmu_buf_rele+0xe/0x10 [zfs]
[ 7080.229064]  ? dsl_dir_rele+0x30/0x40 [zfs]
[ 7080.229189]  ? spa_keystore_change_key_check+0x178/0x4f0 [zfs]
[ 7080.229324]  dsl_sync_task_sync+0xb5/0x100 [zfs]
[ 7080.229418]  dsl_pool_sync+0x365/0x3f0 [zfs]
[ 7080.229507]  spa_sync_iterate_to_convergence+0xe0/0x1e0 [zfs]
[ 7080.229609]  spa_sync+0x305/0x5b0 [zfs]
[ 7080.229718]  txg_sync_thread+0x26c/0x2f0 [zfs]
[ 7080.229835]  ? txg_dispatch_callbacks+0x100/0x100 [zfs]
[ 7080.229952]  thread_generic_wrapper+0x79/0x90 [spl]
[ 7080.229963]  kthread+0x11f/0x140
[ 7080.229970]  ? __thread_exit+0x20/0x20 [spl]
[ 7080.229980]  ? set_kthread_struct+0x50/0x50
[ 7080.229984]  ret_from_fork+0x22/0x30

I've tested this on (all x64_64):

• Proxmox VE 7 (Debian Bullseye with zfs-2.0.5-pve1, 5.11.22-4-pve)
• Stock Debian Bullseye (zfs-2.0.3-9, 5.10.0-8-amd64)
• Stock Ubuntu 20.04 LTS (zfs-2.0.2-1ubuntu5.1, 5.11.0-36-generic)

[secabeen]

I think you've got the problem mostly identified. While your zfs change-key -i command exposed the problem, the root problem here was originally creating the receive encryption root manually, rather than receiving it from the source:
zfs create -o encryption=on -o keyformat=passphrase -o keylocation=prompt replica/encrypted. Doing this means that your replica/encrypted dataset has a different IV than your src/encrypted dataset.

When you zfs change-key -i replica/encrypted/a, that tells ZFS to inherit, which it happily does (without checking to see if the IVs match). I think ZFS happens to continue to allow you to use the FS until you send over another snapshot because the old IVs for those datasets are still associated with them somehow, and it just happens to keep working. If you rebooted, here, before doing the snapshot, it might stop working.

There's clearly a documentation shortfall, and some opportunities for code to protect users as well as some tooling to repair/import some of this data in disaster recovery situations. In your current situation, you may be able to recover by rolling back your receive dataset to eliminate all of the snapshots made after the change-key -i, then re-sending your incrementals.

[brenc]

Doing this means that your replica/encrypted dataset has a different IV than your src/encrypted dataset.

So it sounds to me like it's OK to have child datasets with a different IV but the tooling should absolutely not be allowing zfs change-key -i on these child datasets?

If you rebooted, here, before doing the snapshot, it might stop working.

After running zfs change-key -i replica/encrypted/a I can still mount the filesystem after a reboot. It's only when I send the first incremental and then reboot (or simply unmount/mount) that I get cannot mount 'replica/encrypted/a': Permission denied.

you may be able to recover by rolling back your receive dataset

Unfortunately this does not work. The encryptionroot is still set to replica/encrypted and I still get a permission denied on mount.

Another thing I noticed: while keys are loaded I tried doing zfs send -Rvw replica/encrypted/a@test1 | zfs receive -svF replica/a_restore. But when I do a zfs mount -l replica/a_restore and enter the password, I get Key load error: Incorrect key provided for 'replica/encrypted/a_restore'.

I tried this a several times and get the same result: the password is no longer accepted.

Then I tried doing zfs change-key -o keylocation=prompt -o keyformat=passphrase replica/encrypted. Before I tried this on the child dataset and got a panic. Same deal:

[  950.488754] VERIFY3(0 == spa_keystore_dsl_key_hold_dd(dp->dp_spa, dd, FTAG, &dck)) failed (0 == 13)
[  950.488858] PANIC at dsl_crypt.c:1450:spa_keystore_change_key_sync_impl()
[  950.488918] Showing stack for process 1131
[  950.488924] CPU: 5 PID: 1131 Comm: txg_sync Tainted: P           O      5.11.0-36-generic #40-Ubuntu
[  950.488930] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
[  950.488933] Call Trace:
[  950.488942]  show_stack+0x52/0x58
[  950.488958]  dump_stack+0x70/0x8b
[  950.488970]  spl_dumpstack+0x29/0x2b [spl]
[  950.489001]  spl_panic+0xd4/0xfc [spl]
[  950.489024]  ? dsl_wrapping_key_rele.constprop.0+0x12/0x20 [zfs]
[  950.489341]  ? spa_keystore_dsl_key_hold_dd+0x1a8/0x200 [zfs]
[  950.489518]  spa_keystore_change_key_sync_impl+0x3c0/0x3d0 [zfs]
[  950.489703]  spa_keystore_change_key_sync_impl+0x12c/0x3d0 [zfs]
[  950.489878]  ? zap_lookup+0x16/0x20 [zfs]
[  950.490121]  spa_keystore_change_key_sync+0x157/0x3c0 [zfs]
[  950.490423]  ? dmu_buf_rele+0xe/0x10 [zfs]
[  950.490557]  ? dsl_dir_rele+0x30/0x40 [zfs]
[  950.490729]  ? spa_keystore_change_key_check+0x178/0x4f0 [zfs]
[  950.490886]  dsl_sync_task_sync+0xb5/0x100 [zfs]
[  950.491066]  dsl_pool_sync+0x365/0x3f0 [zfs]
[  950.491270]  spa_sync_iterate_to_convergence+0xe0/0x1e0 [zfs]
[  950.491473]  spa_sync+0x305/0x5b0 [zfs]
[  950.491686]  txg_sync_thread+0x26c/0x2f0 [zfs]
[  950.491927]  ? txg_dispatch_callbacks+0x100/0x100 [zfs]
[  950.492159]  thread_generic_wrapper+0x79/0x90 [spl]
[  950.492213]  kthread+0x11f/0x140
[  950.492224]  ? __thread_exit+0x20/0x20 [spl]
[  950.492242]  ? set_kthread_struct+0x50/0x50
[  950.492249]  ret_from_fork+0x22/0x30

This happens even when I destroy the corrupted child dataset so it appears that not only is this corrupting the replicated child datasets but it's also corrupting the parent dataset as well.

There's clearly a documentation shortfall

Yeah this was shockingly easy to step in to and pretty scary considering it appears to work just fine until that first reboot or unmount/mount!

[secabeen]

So it sounds to me like it's OK to have child datasets with a different IV but the tooling should absolutely not be allowing zfs change-key -i on these child datasets?

I am not a ZFS expert, but I would guess that that is the case. I believe that change-key -i requires both keys to be loaded, and probably even checks or changes the key for the child dataset as needed, but neglects to do the same for the IVs.

[AttilaFueloep]

@almereyda What you are observing should've been fixed by #9309 which was merged two years ago. Unfortunately it was never included in a 0.8.x release, this looks like an oversight to me. It is included in the 2.0.0 release though. And indeed I can't reproduce this on a somewhat current master.

Regarding your inaccessible data, let's summarize key handling for a better understanding of what 's happening here.

Every dataset has its master key which is shared with all snapshots and clones of that dataset. This is a random key generated on dataset creation and never changes. It is stored encrypted on disk. The encryption keys which are used to encrypt the data are derived from that master key. The on disk master key is encrypted with a wrapping key which in turn is either passed in verbatim
via keyformat=raw|hex or generated by PBKDF2 which uses a passphrase and a salt as inputs (keyformat=passphrase). The encryptionroot property refers to the dataset where the wrapping key information is stored.

You are now in a situation where you can't decrypt the master key of the pool1/test dataset since the wrapping key the encryption root refers to is different from the one used to encrypt the master key. The PBKDF2 salt changed when you recreated the pool1 dataset, meaning the same passphrase will now generate a different wrapping key. Whether or not your data is recoverable depends on the accessibility of the old PBKDF2 salt. It may still be stored in the pool1/test dataset. If this is true a custom binary could make the pool1/test dataset its own encryption root and the data would be accessible again. This is basically what zfs change-key does if you give it the right -o options and run it on a non-encryptionroot dataset. But it requires the keys to be loaded since it has to re-encrypt the master key. So this custom binary needs to be written or the zfs command extended to allow for that. I'm not aware of any other options to recover the data.

Please note that there's a bit of speculation in the above since I have no zfs 0.8.x lying around to reproduce with. If you're desperate enough I can give you some commands to check my assumptions.

[secabeen]

@AttilaFueloep That's interesting backstory. However, no where in @almereyda's listed commands is -d included. The only listed arguments to a zfs receive are -svF. Can this still happen without the -d?

[AttilaFueloep]

Well, this is in the reproducer.

zfs send -Rw pool1/test@snap | zfs recv -d pool2
zpool destroy pool1
cfdisk /dev/sda # to downsize the partition
zpool create ... pool1 /dev/sda3
zfs send -Rw pool2/test@snap | zfs recv -d pool1

recv -d is even mentioned in the subject of this issue.

I think you are referring to the issue @brenc reported here. That one is different since it involves changing the encryptionroot via zfs change-key -i. I haven't found time to look into that yet.

[secabeen]

Ahh yes. We may have two different problems that ended up in the same issue. They could be related, or they could not be. Perhaps we can spin off the @brenc problem into a new issue, and close @AttilaFueloep's as resolved in 2.0?