CXL nvdimm auto-unlock fails on key-description mismatch

[AlisonSchofield]

CXL nvdimm auto-unlock: serial-number key-description mismatch

A passphrase-protected CXL nvdimm may fail to auto-unlock after a reboot,
staying "locked" so that the pmem namespaces it backs do not come up. The
cause is a serial-number formatting mismatch between the kernel and ndctl:
the kernel looked the security key up by a hex-formatted serial, while
ndctl saved and named the key blob by the decimal serial. The two spellings
differ for any serial of 10 or greater, so the key the kernel requests is
never found. A second formatting problem affects serials with bit 63 set
(for example Montage devices): the decimal serial was formatted as signed,
so it rendered as a negative value.

A fix for this is expected in Linux 7.3 and later, that formats the serial as unsigned
decimal consistently in both the key description and the 'id' attribute

This document covers two situations:

• Recovering a dimm on a kernel that still lacks the fix.
• Migrating an already-enrolled key after upgrading to the fixed kernel.

Manual recovery (kernel that lacks the fix)

If a passphrase-protected CXL nvdimm does not auto-unlock after a reboot
-- it stays "locked" and the pmem namespaces it backs do not come up --
use the following to recover on a kernel that lacks the serial fix.

1. Confirm the device is actually locked, and that auto-unlock is what
   failed (not, say, a missing key blob or a hardware/security state
   issue):

   ndctl list -i -d nmemX | jq -r '.[].dimms[0].security'

   This should report "locked". If it reports "disabled" the device has
   no passphrase enrolled and this is a different problem; if it reports
   "unlocked" there is nothing to recover.

2. Confirm the failure is the serial-number format mismatch. The kernel
   looks the key up by a hex-formatted serial, while ndctl saved and
   named it by the decimal serial. They differ for any serial of 10 or
   greater. Compare the two spellings of this device's serial:

   decimal serial, as ndctl named the key:

   cat /sys/bus/nd/devices/nmemX/cxl/id

   hex serial, as the kernel looks it up:

   cat /sys/bus/cxl/devices/memY/serial

   If the decimal 'id' is 10 or greater (the hex and decimal forms are
   not the same string), you are hitting this issue. Confirm the key
   blob ndctl saved exists, named by the decimal serial:

   id=$(cat /sys/bus/nd/devices/nmemX/cxl/id)
   ls /etc/ndctl/keys/nvdimm_${id}_$(hostname).blob

   If that blob is missing, the key was never saved and this recovery
   does not apply -- re-enroll the passphrase instead.

3. Stage the same key blob under the hex-formatted serial so that
   load-keys also installs the key the kernel asks for:

   id=$(cat /sys/bus/nd/devices/nmemX/cxl/id)
   hexid=$(printf '%x' "$id")
   host=$(hostname)
   cp /etc/ndctl/keys/nvdimm_${id}${host}.blob
   /etc/ndctl/keys/nvdimm${hexid}_${host}.blob

4. Reload the keys and bring the device up. load-keys derives the key
   description from the blob file name, so the copy is installed as the
   nvdimm: key the kernel requests:

   ndctl load-keys
   ndctl enable-dimm nmemX

   The device should now report "unlocked":

   ndctl list -i -d nmemX | jq -r '.[].dimms[0].security'

Verified against cxl_test on an unpatched kernel: the dimm unlocks.

Migration after upgrading to the fixed kernel

The recovery above is for a kernel that lacks the fix. After upgrading
to the fixed kernel, one class of already-enrolled key must be migrated.

The fix formats the serial as unsigned decimal (%llu) for both the key
description and the 'id' attribute. The only keys affected on upgrade are
those for a serial with bit 63 set (for example a Montage device): before
the fix id_show() emitted the serial as signed decimal, so ndctl named
the key blob by a negative string, while the fixed kernel now requests
the key by the unsigned decimal. The existing blob is not found and the
dimm fails to auto-unlock.

Serials without bit 63 set need no action. ndctl already named the blob
by the positive decimal the fixed kernel now requests, so they
auto-unlock after the upgrade with no migration.

To migrate a bit-63 serial, stage the existing blob under the unsigned
decimal name the fixed kernel requests. The old blob is already on disk
named by the negative decimal the previous kernel emitted, so locate it
rather than recompute it:

 # unsigned decimal serial, as the fixed kernel now looks it up:
 id=$(cat /sys/bus/nd/devices/nmemX/cxl/id)
 host=$(hostname)

 # the existing blob, named by the old negative decimal:
 ls /etc/ndctl/keys/nvdimm_-*_${host}.blob

 cp /etc/ndctl/keys/nvdimm_<negative-id>_${host}.blob \
    /etc/ndctl/keys/nvdimm_${id}_${host}.blob

Reload the keys and bring the device up. load-keys derives the key
description from the blob file name, so the copy is installed as the
nvdimm: key the fixed kernel requests:

 ndctl load-keys
 ndctl enable-dimm nmemX

The device should now report "unlocked":

 ndctl list -i -d nmemX | jq -r '.[].dimms[0].security'