FreeBSD: zfskeys_enable: encryption key not loaded for a file system within a pool that imports automatically at startup

[grahamperrin]

System information

Type	Version/Name
Distribution Name	FreeBSD
Distribution Version	14.0-CURRENT
Kernel Version	1400048
Architecture	AMD64
OpenZFS Version	2.1.99

https://bsd-hardware.info/?probe=f47789d894

Describe the problem you're observing

When the operating system starts:

• pool Transcend imports automatically
• non-encrypted Transcend mounts automatically
• encrypted Transcend/VirtualBox can not be mounted.

% date ; uptime 
Sat 29 Jan 2022 15:30:05 GMT
 3:30p.m.  up 48 mins, 5 users, load averages: 0.77, 0.56, 0.44
% mount | grep Transcend
Transcend on /Volumes/t500 (zfs, local, nfsv4acls)
% su -
Password:
root@mowa219-gjp4-8570p-freebsd:~ # zfs mount Transcend/VirtualBox
cannot mount 'Transcend/VirtualBox': encryption key not loaded
root@mowa219-gjp4-8570p-freebsd:~ # 

Describe how to reproduce the problem

zfskeys_enable="YES" in /etc/rc.conf

Have a pool Transcend on a mobile hard disk drive on USB.

One non-encrypted file system.

One encrypted file system Transcend/VirtualBox with a key that's stored in the home directory, for example:

/usr/home/grahamperrin/Documents/personal/VirtualBox.key

After starting FreeBSD, import the pool, for example:

zpool import Transcend ; zpool status Transcend && zfs load-key Transcend/VirtualBox && zfs mount Transcend/VirtualBox ; mount | grep Transcend ; sleep 5 ; zpool iostat -v

Restart the operating system.

Expected:

• load of the key
• mount of the decrypted file system.

Actual result:

• the expected file system is not mounted, and (above)

cannot mount 'Transcend/VirtualBox': encryption key not loaded

Workaround:

• manually load the key and mount the file system, for example:

root@mowa219-gjp4-8570p-freebsd:~ # zfs load-key Transcend/VirtualBox && zfs mount Transcend/VirtualBox ; zpool online Transcend gpt/cache-transcend && mount | grep Transcend && zpool status -v Transcend && sleep 5 && zpool iostat -v
warning: device 'gpt/cache-transcend' onlined, but remains in faulted state
use 'zpool replace' to replace devices that are no longer present
Transcend on /Volumes/t500 (zfs, local, nfsv4acls)
Transcend/VirtualBox on /Volumes/t500/VirtualBox (zfs, local, nfsv4acls)
  pool: Transcend
 state: ONLINE
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q
  scan: scrub repaired 0B in 01:32:30 with 0 errors on Sun Dec 12 16:25:35 2021
config:

        NAME                   STATE     READ WRITE CKSUM
        Transcend              ONLINE       0     0     0
          gpt/Transcend        ONLINE       0     0     0
        cache
          gpt/cache-transcend  UNAVAIL      0     0     0  cannot open

errors: No known data errors
                         capacity     operations     bandwidth 
pool                   alloc   free   read  write   read  write
---------------------  -----  -----  -----  -----  -----  -----
Transcend               425G  39.2G      0      0  3.03K  1.62K
  gpt/Transcend         425G  39.2G      0      0  3.03K  1.62K
cache                      -      -      -      -      -      -
  gpt/cache-transcend      -      -      0      0      0      0
---------------------  -----  -----  -----  -----  -----  -----
august                  314G   598G      2      9  38.5K   232K
  ada0p3.eli            314G   598G      2      9  38.5K   232K
cache                      -      -      -      -      -      -
  gpt/cache-august     9.89G  18.9G      8      0   256K  37.0K
  gpt/duracell         9.89G  5.53G      9      0   250K  35.3K
---------------------  -----  -----  -----  -----  -----  -----
root@mowa219-gjp4-8570p-freebsd:~ # 

(Please ignore L2ARC cache device gpt/cache-transcend. It's rarely connected.)

Warning/errors/backtraces from the system logs

I see nothing relevant.

Additional information

https://github.com/freebsd/freebsd-src/blob/main/libexec/rc/rc.d/zfskeys

https://openzfs.github.io/openzfs-docs/man/8/zfs-load-key.8.html

Prior to restarting FreeBSD:

% sysrc -f /etc/rc.conf zfsd_enable zfskeys_enable
zfsd_enable: YES
zfskeys_enable: YES
% zfs version
zfs-2.1.99-FreeBSD_gf291fa658
zfs-kmod-2.1.99-FreeBSD_gf291fa658
% zfs get canmount,keylocation Transcend/VirtualBox
NAME                  PROPERTY     VALUE                                                            SOURCE
Transcend/VirtualBox  canmount     on                                                               default
Transcend/VirtualBox  keylocation  file:///usr/home/grahamperrin/Documents/personal/VirtualBox.key  local
% file /usr/home/grahamperrin/Documents/personal/VirtualBox.key
/usr/home/grahamperrin/Documents/personal/VirtualBox.key: ASCII text
% date ; freebsd-version -kru ; uname -aKU
Sat 29 Jan 2022 14:32:26 GMT
14.0-CURRENT
14.0-CURRENT
14.0-CURRENT
FreeBSD mowa219-gjp4-8570p-freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT #1 main-n252531-0ce7909cd0b-dirty: Wed Jan 19 13:29:34 GMT 2022     root@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG  amd64 1400048 1400048
% lsblk
DEVICE         MAJ:MIN SIZE TYPE                                          LABEL MOUNT
ada0             0:150 932G GPT                                               - -
  ada0p1         0:151 260M efi                                    gpt/efiboot0 -
  <FREE>         -:-   1.0M -                                                 - -
  ada0p2         0:152  16G freebsd-swap                              gpt/swap0 SWAP
  ada0p2.eli     1:241  16G freebsd-swap                                      - SWAP
  ada0p3         0:153 915G freebsd-zfs                                gpt/zfs0 <ZFS>
  ada0p3.eli     0:159 915G -                                                 - -
  <FREE>         -:-   708K -                                                 - -
da0              0:154  29G GPT                                               - -
  da0p1          0:160  29G freebsd-zfs                        gpt/cache-august <ZFS>
da1              0:167  15G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da1p1          0:168  15G freebsd-zfs                            gpt/duracell <ZFS>
da2              2:1   466G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da2p1          2:2   466G freebsd-zfs                           gpt/Transcend <ZFS>
% geom disk list da2
Geom name: da2
Providers:
1. Name: da2
   Mediasize: 500107862016 (466G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e3
   descr: StoreJet Transcend
   lunid: 5000000000000001
   ident: (null)
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

% 

[ghost]

The zfskeys script and other zfs rc sripts are part of FreeBSD. not OpenZFS. You'll want to get in touch with whoever wrote that in case they aren't watching the issues on this repo.

[grahamperrin]

@freqlabs thanks for direction. Now:

• FreeBSD bug 262468 – (13038) zfskeys_enable: encryption key not loaded for a file system within a pool that imports automatically at startup

Condensed:

… the key file is found, but not initially; not when required

(When I began here in the OpenZFS repo, I wondered whether something here might also cause the same effect on other platforms.)

[0mp]

I've landed a patch in the FreeBSD src tree to address this issue.

https://cgit.freebsd.org/src/commit/?id=97aeda2243568b386d792514996a06daec55eece

[ghost]

Thanks!

[grahamperrin]

Likewise, thank you!