lvmetad caching not working in flatcar

[jsalatiel]

Description

I have noticed a pretty strange behaviour on lvm tools inside flatcar.
When running any CSI that uses LVM as backend ( in this case I will show an example using openebs ) in ubuntu, alma, redhat , debian when the CSI creates an LVM to expose a PVC to a pod, lvs and pvs in the host shows that volume just fine. But in flatcar, those new LVM do not show in lvs/pvs in the host neither the used space in the VG is correct. Those information will only be updated in the server is rebooted OR the lvmetad service is restarted.

[ 1 paragraph concisely describing the bug ]
That can mislead sysadmins to change the vg size leading to data corruption because the real state of the volumes are outdated.
( It has happened )

Environment and steps to reproduce

1. Install flatcar
2. Add a secondary disk and create a new volumegroup called openebs on it vgcreate openebs /dev/sdb
3. Deploy k8s
4. helm repo add openebs https://openebs.github.io/charts ( we will use openebs as a test )
5. Install a minimum openebs release
   helm install openebs openebs/openebs --set cstor.enabled=false --set nfs-provisioner.enabled=false --set localprovisioner.enabled=false --set zfs-localpv.enabled=false --set lvm-localpv.enabled=true --set ndmOperator.enabled=false --set ndm.enabled=false --namespace kube-system
6. Create a new storage class using the VG from step 2

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-lvm
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
allowVolumeExpansion: true    
volumeBindingMode: WaitForFirstConsumer
parameters:
  storage: "lvm"
  volgroup: "openebs"
  fsType: "xfs"
provisioner: local.csi.openebs.io

7. Deploy the following yaml ( it will create a PVC from the storage class above )

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  selector:
    matchLabels:
      app: redis
  replicas: 1
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis
        resources:
          limits:
            memory: 2Gi
        volumeMounts:
          - name: redis-data
            mountPath: /usr/share/redis
  volumeClaimTemplates:
  - metadata:
      name: redis-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 500Mi

8. After the container is up, run sudo lvs or sudo vgs inside the host and you will noticed that the lvs output does not show the newly create logical volume and the vgs output wont show the space used by that volume.
9. Run sudo systemctl restart lvm2-lvmetad.service OR sudo pvscan --cache or reboot
10. Try again run lvs or vgs and you will noticed that those outputs now reflect correctly what has been created in openebs

Noticed that the behaviour is only on flatcar, not in any other linux distribution i haved tested ( ubuntu, redhat, alma , debian ).

Expected behavior

Volumes should be shown despite restart lvmetad

[jsalatiel]

lvmetad has been deprecated and should not be used. The reason it works in other linux distributions like rhel for example is that lvmetad is not present on newer versions.

The solution for flatcar is add this to /etc/lvm/lvm.conf and restart

global {
  use_lvmetad = 0
}