regression: OSD cannot be deployed on PV backed by LVM Logical Volume

[morimoto-cybozu]

Is this a bug report or feature request?

• Bug Report

Deviation from expected behavior:

When configuring an OSD on PVC with a PV backed by an LVM Logical Volume, the OSD prepare pod fails to initialize the LV device as an OSD.

It fails when I use the master of Rook while it succeeds with Rook v1.2.6. The problem was introduced around the refactor of OSD store.

There are two problems. The more important issue is that the availability check using ceph-volume inventory in inventoryDevice() does not succeed for LVs. I tried /dev/mapper/vg1-lv1, /dev/vg1/lv1, /dev/dm-7, and vg1/lv1 for ceph-volume inventory, and received "Insufficient space (<5GB)" in all cases.

(This is a little off the topic, but the error message of ceph-volume seems odd. I'll send a bug fix PR to the Ceph community.)

As a minor issue, inventoryDevice() passes "/dev/<device.RealName>" to ceph-volume inventory if pvcBacked, but this is not good for LVs. "/dev/<device.RealName>" produces a non-existent device path like /dev/vg1-lv1 instead of /dev/mapper/vg1-lv1.

Expected behavior:

The LV device is initialized as an OSD.

How to reproduce it (minimal and precise):

1. create a k8s cluster
2. deploy the Rook operator by applying "common.yaml" and "operator.yaml" given at cluster/examples/kubernetes/ceph
3. create an LV on a node, e.g. 10 GiB vg1/lv1 on 10.69.0.4
4. register the LV as a PV by applying the attached "pv.yaml"
5. deploy a Ceph cluster by applying the attached "cluster.yaml"
6. watch pods; the OSD prepare pod completes with outputting the attached log and no OSD pod starts

Files:

pv.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: lv1
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  local:
    path: /dev/vg1/lv1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - 10.69.0.4
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  volumeMode: Block

cluster.yaml:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  dataDirHostPath: /var/lib/rook
  mon:
    count: 1
    allowMultiplePerNode: false
  cephVersion:
    image: ceph/ceph:v14.2.8
  dashboard:
    enabled: false
    ssl: true
  network:
    hostNetwork: false
  storage:
    storageClassDeviceSets:
    - name: set1
      count: 1
      volumeClaimTemplates:
      - metadata:
          name: data
        spec:
          resources:
            requests:
              storage: 10Gi
          storageClassName: local-storage
          volumeMode: Block
          accessModes:
            - ReadWriteOnce

OSD prepare pod log:

2020-03-23 08:54:15.606396 I | cephcmd: desired devices to configure osds: [{Name:/mnt/set1-data-0-98jxb OSDsPerDevice:1 MetadataDevice: DatabaseSizeMB:0 DeviceClass: IsFilter:false IsDevicePathFilter:false}]
2020-03-23 08:54:15.611320 I | rookcmd: starting Rook v1.1.0-beta.0.1102.gda3bf49 with arguments '/rook/rook ceph osd provision'
2020-03-23 08:54:15.611336 I | rookcmd: flag values: --cluster-id=60d3a214-5a06-423e-8814-7a3c2b2eab46, --data-device-filter=, --data-device-path-filter=, --data-devices=/mnt/set1-data-0-98jxb, --encrypted-device=false, --force-format=false, --help=false, --location=, --log-flush-frequency=5s, --log-level=DEBUG, --metadata-device=, --node-name=set1-data-0-98jxb, --operator-image=, --osd-database-size=0, --osd-store=, --osd-wal-size=576, --osds-per-device=1, --pvc-backed-osd=true, --service-account=
2020-03-23 08:54:15.611345 I | op-mon: parsing mon endpoints: a=10.68.91.3:6789
2020-03-23 08:54:15.661619 I | op-osd: CRUSH location=root=default host=10-69-0-4 zone=rack0
2020-03-23 08:54:15.662016 I | cephcmd: crush location of osd: root=default host=10-69-0-4 zone=rack0
2020-03-23 08:54:15.835101 I | cephconfig: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-03-23 08:54:15.835153 I | cephconfig: generated admin config in /var/lib/rook/rook-ceph
2020-03-23 08:54:15.835327 D | cephosd: config file @ /etc/ceph/ceph.conf: [global]
fsid                = eeeeb2fe-6dd4-4e3c-8d95-f6031d3287ba
mon initial members = a
mon host            = [v2:10.68.91.3:3300,v1:10.68.91.3:6789]
public addr         = 10.64.0.12
cluster addr        = 10.64.0.12

[client.admin]
keyring = /var/lib/rook/rook-ceph/client.admin.keyring

2020-03-23 08:54:15.835335 I | cephosd: discovering hardware
2020-03-23 08:54:15.835346 D | exec: Running command: lsblk /mnt/set1-data-0-98jxb --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME,NAME
2020-03-23 08:54:15.854616 D | exec: Running command: sgdisk --print /mnt/set1-data-0-98jxb
2020-03-23 08:54:15.875630 I | cephosd: creating and starting the osds
2020-03-23 08:54:15.876106 D | cephosd: desiredDevices are [{Name:/mnt/set1-data-0-98jxb OSDsPerDevice:1 MetadataDevice: DatabaseSizeMB:0 DeviceClass: IsFilter:false IsDevicePathFilter:false}]
2020-03-23 08:54:15.876118 D | cephosd: context.Devices are [0xc000119440]
2020-03-23 08:54:15.876139 D | exec: Running command: ceph-volume inventory --format json /dev/vg1-lv1
2020-03-23 08:54:19.073529 I | cephosd: skipping device "/mnt/set1-data-0-98jxb": ["Insufficient space (<5GB)"].
2020-03-23 08:54:19.130929 I | cephosd: configuring osd devices: {"Entries":{}}
2020-03-23 08:54:19.131114 I | cephosd: no new devices to configure. returning devices already configured with ceph-volume.
2020-03-23 08:54:19.131169 D | exec: Running command: pvdisplay -C -o lvpath --noheadings /mnt/set1-data-0-98jxb
2020-03-23 08:54:19.241503 W | cephosd: failed to retrieve logical volume path for "/mnt/set1-data-0-98jxb". exit status 5
2020-03-23 08:54:19.241524 D | exec: Running command: ceph-volume lvm list  --format json
2020-03-23 08:54:19.863556 I | cephosd: 0 ceph-volume lvm osd devices configured on this node
2020-03-23 08:54:19.863693 D | exec: Running command: ceph-volume raw list /mnt/set1-data-0-98jxb --format json
2020-03-23 08:54:20.373591 I | cephosd: 0 ceph-volume raw osd devices configured on this node
2020-03-23 08:54:20.373681 W | cephosd: skipping OSD configuration as no devices matched the storage settings for this node "set1-data-0-98jxb"

Environment:

• OS (e.g. from /etc/os-release): Container Linux by CoreOS 2345.3.0 (Rhyolite)
• Kernel (e.g. uname -a): Linux rack0-cs4 4.19.106-coreos Monitor bootstrapping with libcephd #1 SMP Wed Feb 26 21:43:18 -00 2020 x86_64 Intel(R) Xeon(R) CPU GenuineIntel GNU/Linux
• Cloud provider or hardware configuration: QEMU VMs on GCP
• Rook version (use rook version inside of a Rook Pod): v1.1.0-beta.0.1102.gda3bf49 (using rook/ceph:master)
• Storage backend version (e.g. for ceph do ceph -v): ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable)
• Kubernetes version (use kubectl version): v1.16.6-beta.0
• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): CKE
• Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): N/A

[leseb]

@morimoto-cybozu are you going to send a PR to fix this? It looks like you have analyzed the issue quite well. Thanks. It would be nice to have this for 1.3.

[morimoto-cybozu]

@leseb
Yes, I'm working on this to send a PR.

[morimoto-cybozu]

I'm trying to use ceph-volume lvm list when checking the availability of LVs. It returns an empty dict if the LV is not used by Ceph.