OSD failing to start if encrypted and also have metadata device

[SlyngDK]

Is this a bug report or feature request?

• Bug Report

Deviation from expected behavior:
Metadata device is not decrypted, on first pod start, because decryption of data device deletes key, before running decryption of metadata.

Expected behavior:
Expected that both data and metadata devices is decrypted the first time.
How to reproduce it (minimal and precise):

Setup using pvc with encryption enabled, with metadata device.
Use vault as KMS.

Start osd with both data and metadata device encrypted, waiting for decrypt.

File(s) to submit:

• Cluster CR (custom resource), typically called cluster.yaml, if necessary

---
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v18.2.1
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
  dashboard:
    enabled: true
    ssl: false
  mgr:
    count: 2
  storage:
    config:
      encryptedDevice: "true"
    useAllNodes: true
    useAllDevices: false
    storageClassDeviceSets:
      - name: ceph-set-2
        count: 3
        portable: false
        encrypted: true
        volumeClaimTemplates:
          - metadata:
              name: data
            spec:
              resources:
                requests:
                  storage: 1000Gi
              storageClassName: local-device
              volumeMode: Block
              accessModes:
                - ReadWriteOnce
          - metadata:
              name: metadata
            spec:
              resources:
                requests:
                  storage: 200Gi
              storageClassName: local-device-ssd
              volumeMode: Block
              accessModes:
                - ReadWriteOnce
  security:
    keyRotation:
      enabled: false
    kms:
      connectionDetails:
        KMS_PROVIDER: vault
        VAULT_ADDR: https://xxxx.xxx
        VAULT_AUTH_KUBERNETES_ROLE: xxxxxx
        VAULT_AUTH_METHOD: kubernetes
        VAULT_AUTH_MOUNT_PATH: xxxxxxx
        VAULT_BACKEND_PATH: xxxxxx
        VAULT_SECRET_ENGINE: kv

Logs to submit:

• Crashing pod(s) logs, if necessary

➜  ~ kubectl logs -n rook rook-ceph-osd-1-5856d8cfd8-zl5nj -c encryption-open-metadata
+ CEPH_FSID=d76efac7-13da-4b9a-88ac-6d33623561db
+ PVC_NAME=ceph-set-2-metadata-06kdkr
+ KEY_FILE_PATH=/etc/ceph/luks_key
+ BLOCK_PATH=/var/lib/ceph/osd/ceph-1/block.db-tmp
+ DM_NAME=ceph-set-2-metadata-06kdkr-db-dmcrypt
+ DM_PATH=/dev/mapper/ceph-set-2-metadata-06kdkr-db-dmcrypt
+ dmsetup version
Library version:   1.02.181-RHEL8 (2021-10-20)
Driver version:    4.48.0
+ '[' -b /dev/mapper/ceph-set-2-metadata-06kdkr-db-dmcrypt ']'
+ open_encrypted_block
+ echo 'Opening encrypted device /var/lib/ceph/osd/ceph-1/block.db-tmp at /dev/mapper/ceph-set-2-metadata-06kdkr-db-dmcrypt'
Opening encrypted device /var/lib/ceph/osd/ceph-1/block.db-tmp at /dev/mapper/ceph-set-2-metadata-06kdkr-db-dmcrypt
+ cryptsetup luksOpen --verbose --disable-keyring --allow-discards --key-file /etc/ceph/luks_key /var/lib/ceph/osd/ceph-1/block.db-tmp ceph-set-2-metadata-06kdkr-db-dmcrypt
Failed to open key file.
Command failed with code -1 (wrong or missing parameters).

Cluster Status to submit:
N/A

Environment:

• OS (e.g. from /etc/os-release): Ubuntu 22.04
• Kernel (e.g. uname -a): 6.5.0-17-generic
• Cloud provider or hardware configuration:
• Rook version (use rook version inside of a Rook Pod): 1.13.3
• Storage backend version (e.g. for ceph do ceph -v): 18.2.1
• Kubernetes version (use kubectl version): 1.26.4
• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): RKE2
• Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

[cupnes]

I reproducing your situation.

[cupnes]

I deleted spec.security from your cluster.yaml and it did not reproduce. Perhaps it reproduces when spec.security is written.

[SlyngDK]

I deleted spec.security from your cluster.yaml and it did not reproduce. Perhaps it reproduces when spec.security is written.

I think it is only a problem when using encryption, and .spec.security, is part of encryption configuration.

[cupnes]

I succeeded in reproducing your situation at #13773.

[cupnes]

I am busy with another matter and will not be able to investigate until the end of next week.

[cupnes]

The key file is deleted in the shell script commonly used by both encryption-open and encryption-open-metadata initContainers. Failed to open key file. in encryption-open-metadata because the key file has been deleted when encryption-open is run.

The cause has been identified. I will create PR. Instead of deleting the key file here, the container that is just to delete the key file will be added at the end of the sequence of containers that use the key file. (I will work on it from about next Tuesday due to the holiday.)