fix: Resolve missing unlock encrypted wal/db at osd start

[johnramsden]

Description

On node reboot, or if devices are closed encrypted wal/dbs will not be unlocked correctly and will lead to issues on startup.

The relevant issue #655 describes the problem very well:

There is a discrepancy between the OSD Creation logic (Go) and the OSD Startup logic (Shell).

1. Creation (Correct):
   In microceph/ceph/osd.go, the function setupEncryptedOSD correctly handles Block, WAL, and DB using the corresponding "suffix". It formats and opens (unlocks) all of them explicitly.

   • Reference: microceph/ceph/osd.go

2. Startup (Bug):
   In snapcraft/commands/osd.start, the spawn() function iterates through the OSD directories.

   • It checks for ${i}/unencrypted (Main Block) and calls maybe_unlock.
   • It fails to check for or unlock ${i}/unencrypted.wal or ${i}/unencrypted.db.
   • Reference: snapcraft/commands/osd.start

Because the shell script ignores the unencrypted.wal symlink, the mapper is never created, and BlueStore cannot access its write-ahead log.

Proposed Fix

Update snapcraft/commands/osd.start to check for and unlock separate WAL/DB devices if they exist, mirroring the logic for the main block device.

Pseudocode logic missing in osd.start:

if [ -b "${i}/unencrypted.db" ] ; then
maybe_unlock "${i}/unencrypted.db" "${nr}" "$( get_key "${nr}" )"
fi
if [ -b "${i}/unencrypted.wal" ] ; then
maybe_unlock "${i}/unencrypted.wal" "${nr}" "$( get_key "${nr}" )"
fi

Add tests and fix the unlock.

Fixes #655

Type of change

• Bug fix (non-breaking change which fixes an issue)

How has this been tested?

This has been tested locally by executing:

INITIAL_OSD_COUNT=$(microceph.ceph -s -f json | jq -r '.osdmap.num_in_osds // 0')
EXPECTED=$((INITIAL_OSD_COUNT + 1))
tests/scripts/actionutils.sh test_encrypted_wal_db_startup "${EXPECTED}" 

The test which is also been added to CI creates three loop back devices, and then creates an OSD with an encrypted WAL & DB. This happens successfully.

The test then shuts down the OSD, closes the relevant encrypted devices and then starts the OSD back up. This should lead to the OSD being opened successfully. Instead it fails.

Contributor checklist

Please check that you have:

• self-reviewed the code in this PR
• added code comments, particularly in less straightforward areas
• checked and added or updated relevant documentation
• checked and added or updated relevant release notes
• added tests to verify effectiveness of this change reply with my

[sabaini]

Hey @johnramsden thanks lgtm in general, only a minor testing nit!

[johnramsden]

I believe the current errors are not related to this PR

[sabaini]

Good stuff, thx!