Ubuntu 20.04 beta encrypted files disappear

[mibolin]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	20.04
Linux Kernel	5.4.0-21-generic
Architecture	x86_64
ZFS Version	0.8.3-1ubuntu8
SPL Version	0.8.3-1ubuntu8

Describe the problem you're observing

For benchmarking I create encrypted datasets and write a file with dd on it.
After shutdown -h now and restarting the computer I have to manually import the pool and load the encryption keys. But, my benchmark file is not there anymore.
It happened to me first on 2020/04/13 but I could reproduce it today (had no time before).

Describe how to reproduce the problem

Some comments about my setup.
It's my old AM2 fileserver with an Athlon X2 cpu and ECC DDR2 memory. Before upgrading to a new ZFS mirror on SATA drives I connected 2xWD Elements 10TB external harddisks to the USB3 PCIe card. My / is on EXT4 on an old ssd.

zpool create -f daten2020 -m /mnt/daten2020 -o ashift=12 mirror /dev/sdg /dev/sdh
zfs set compression=lz4 daten2020

dd if=/dev/random of=/root/20200416_zfs_key.dd bs=32 count=1
zfs create -o encryption=aes-256-ccm -o keyformat=raw -o keylocation=file:///root/20200416_zfs_key.dd daten2020/enc256ccm_2
zfs create -o encryption=aes-256-gcm -o keyformat=raw -o keylocation=file:///root/20200416_zfs_key.dd daten2020/enc256gcm_2
root@fileserver:/home/michael# dd if=/dev/urandom of=/mnt/daten2020/enc256ccm_2/speed_daten2020_rand_bs4k_count10k.dd bs=4k count=10000 conv=fsync
10000+0 records in
10000+0 records out
40960000 bytes (41 MB, 39 MiB) copied, 1.21541 s, 33.7 MB/s
root@fileserver:/home/michael# dd if=/dev/urandom of=/mnt/daten2020/enc256gcm_2/speed_daten2020_rand_bs4k_count10k.dd bs=4k count=10000 conv=fsync
10000+0 records in
10000+0 records out
40960000 bytes (41 MB, 39 MiB) copied, 2.52985 s, 16.2 MB/s
root@fileserver:/home/michael# ll /mnt/daten2020/enc256gcm_2
total 40062
drwxr-xr-x 2 root root        3 Apr 16 21:15 ./
drwxrwxrwx 6 root root       10 Apr 16 21:14 ../
-rw-r--r-- 1 root root 40960000 Apr 16 21:15 speed_daten2020_rand_bs4k_count10k.dd
root@fileserver:/home/michael# ll /mnt/daten2020/enc256ccm_2
total 40062
drwxr-xr-x 2 root root        3 Apr 16 21:15 ./
drwxrwxrwx 6 root root       10 Apr 16 21:14 ../
-rw-r--r-- 1 root root 40960000 Apr 16 21:15 speed_daten2020_rand_bs4k_count10k.dd

So, the benchmark files were there.
Then I shutdown -h now
and restarted this morning.

root@fileserver:/home/michael# zpool import daten2020
root@fileserver:/home/michael# ll /mnt/daten2020/enc256ccm_2/
total 9
drwxr-xr-x 2 root root  2 Apr 16 21:13 ./
drwxrwxrwx 6 root root 10 Apr 16 21:14 ../
root@fileserver:/home/michael# ll /mnt/daten2020/enc256gcm_2/
total 9
drwxr-xr-x 2 root root  2 Apr 16 21:14 ./
drwxrwxrwx 6 root root 10 Apr 16 21:14 ../
root@fileserver:/home/michael# zfs load-key -a
4 / 4 key(s) successfully loaded
root@fileserver:/home/michael# ll /mnt/daten2020/enc256gcm_2/
total 9
drwxr-xr-x 2 root root  2 Apr 16 21:14 ./
drwxrwxrwx 6 root root 10 Apr 16 21:14 ../
root@fileserver:/home/michael# ll /mnt/daten2020/enc256ccm_2/
total 9
drwxr-xr-x 2 root root  2 Apr 16 21:13 ./
drwxrwxrwx 6 root root 10 Apr 16 21:14 ../
root@fileserver:/home/michael# zfs get -p encryption,keystatus,keylocation
NAME                   PROPERTY     VALUE                             SOURCE
daten                  encryption   off                               default
daten                  keystatus    -                                 -
daten                  keylocation  none                              default
daten@backup20190203   encryption   off                               default
daten@backup20190203   keystatus    -                                 -
daten@backup20190203   keylocation  -                                 -
daten@backup20190205   encryption   off                               default
daten@backup20190205   keystatus    -                                 -
daten@backup20190205   keylocation  -                                 -
daten@backup20200405   encryption   off                               default
daten@backup20200405   keystatus    -                                 -
daten@backup20200405   keylocation  -                                 -
daten2020              encryption   off                               default
daten2020              keystatus    -                                 -
daten2020              keylocation  none                              default
daten2020/enc256ccm    encryption   aes-256-ccm                       -
daten2020/enc256ccm    keystatus    available                         -
daten2020/enc256ccm    keylocation  file:///root/20200412_zfs_key.dd  local
daten2020/enc256ccm_2  encryption   aes-256-ccm                       -
daten2020/enc256ccm_2  keystatus    available                         -
daten2020/enc256ccm_2  keylocation  file:///root/20200416_zfs_key.dd  local
daten2020/enc256gcm    encryption   aes-256-gcm                       -
daten2020/enc256gcm    keystatus    available                         -
daten2020/enc256gcm    keylocation  file:///root/20200412_zfs_key.dd  local
daten2020/enc256gcm_2  encryption   aes-256-gcm                       -
daten2020/enc256gcm_2  keystatus    available                         -
daten2020/enc256gcm_2  keylocation  file:///root/20200416_zfs_key.dd  local
root@fileserver:/home/michael#

So, the benchmark files in /mnt/daten2020/enc256ccm_2/ and /mnt/daten2020/enc256gcm_2/ are gone.
Why?

Include any warning/errors/backtraces from the system logs

I haven't seen warnings/errors.
Please ask for more details.

[mibolin]

As Ubuntu 20.04 gets updates I re-tested again.
The problem is still reproducible. The written files are not there after a reboot.
If my testing is wrong - please tell me.

Currently used versions:
`root@fileserver:/home/michael# uname -a
Linux fileserver 5.4.0-25-generic #29-Ubuntu SMP Fri Apr 17 15:06:57 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

root@fileserver:/home/michael# modinfo zfs | grep -iw version
version: 0.8.3-1ubuntu11

root@fileserver:/home/michael# modinfo spl | grep -iw version
version: 0.8.3-1ubuntu11
`

[InsanePrawn]

I suspect your encrypted datasets are just not mounted. Check the output of df -h /mnt/daten2020/enc256ccm_2, mountpoint /mnt/daten2020/enc256ccm_2 or even a simple mount | grep /mnt/daten2020/enc256ccm_2.

If the post-reboot terminal session you posted is complete, that's unsurprising; after loading the keys with zfs load-key -a, you never mount anything. Try a zfs mount -a after loading the keys?

[mibolin]

Thank you very much.
I didn't know that I have to mount the datasets after zfs load-key -a. I thought this is done automatically.
With zfs mount -a the "old" files are still there.

Sorry for the noise. This issue was not a ZFS issue. Closed.