Error: unable to read omap keys: pool or key missing: key not found: rados: ret=-2, No such file or directory

[reefland]

Rook v1.13.5:

$ ceph -v        
ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)

Using the Helm Chart, I have an existing cephBlockPools (works fine):

  - name: ceph-blockpool
  ...
    storageClass:
      enabled: true
      name: ceph-block
      isDefault: true
      reclaimPolicy: Retain
      allowVolumeExpansion: true
  ...

I wanted a new storageClass with reclaimPolicy: Delete, created a second storageClass:

  - name: csi-ceph-blockpool
  ...
    storageClass:
      enabled: true
      name: csi-ceph-block
      isDefault: false
      reclaimPolicy: Delete
      allowVolumeExpansion: true
  ...

Restarted the rook operator and I see the new storageClass:

$ k get sc | grep block

ceph-block (default)   rook-ceph.rbd.csi.ceph.com      Retain          Immediate              true                   476d
csi-ceph-block         rook-ceph.rbd.csi.ceph.com      Delete          Immediate              true                   4d4h

When I look at Pod csi-rbdplugin-provsioner, csi-rbdplugin container I see errors such as:

W0303 12:00:14.372306       1 voljournal.go:729] ID: 206 Req-ID: 0001-0009-rook-ceph-000000000000000d-e3e27f91-bce6-47fb-9617-bedf0969f3b4 unable to read omap keys: pool or key missing: key not found: rados: ret=-2, No such file or directory
E0303 12:00:14.379509       1 rbd_journal.go:689] ID: 206 Req-ID: 0001-0009-rook-ceph-000000000000000d-e3e27f91-bce6-47fb-9617-bedf0969f3b4 failed to get image id csi-ceph-blockpool/csi-vol-e3e27f91-bce6-47fb-9617-bedf0969f3b4: image not found: RBD image not found
E0303 16:00:23.385346       1 omap.go:79] ID: 215 Req-ID: 0001-0009-rook-ceph-000000000000000d-a6793299-9d83-46e7-831d-29b9322ae9bc omap not found (pool="csi-ceph-blockpool", namespace="", name="csi.volume.a6793299-9d83-46e7-831d-29b9322ae9bc"): rados: ret=-2, No such file or directory
W0303 16:00:23.385396       1 voljournal.go:729] ID: 215 Req-ID: 0001-0009-rook-ceph-000000000000000d-a6793299-9d83-46e7-831d-29b9322ae9bc unable to read omap keys: pool or key missing: key not found: rados: ret=-2, No such file or directory
E0303 16:00:23.391748       1 rbd_journal.go:689] ID: 215 Req-ID: 0001-0009-rook-ceph-000000000000000d-a6793299-9d83-46e7-831d-29b9322ae9bc failed to get image id csi-ceph-blockpool/csi-vol-a6793299-9d83-46e7-831d-29b9322ae9bc: image not found: RBD image not found
E0303 20:00:23.971080       1 omap.go:79] ID: 224 Req-ID: 0001-0009-rook-ceph-000000000000000d-797353de-777d-4377-a128-17ad48ec58f5 omap not found (pool="csi-ceph-blockpool", namespace="", name="csi.volume.797353de-777d-4377-a128-17ad48ec58f5"): rados: ret=-2, No such file or directory
W0303 20:00:23.971124       1 voljournal.go:729] ID: 224 Req-ID: 0001-0009-rook-ceph-000000000000000d-797353de-777d-4377-a128-17ad48ec58f5 unable to read omap keys: pool or key missing: key not found: rados: ret=-2, No such file or directory
E0303 20:00:23.975563       1 rbd_journal.go:689] ID: 224 Req-ID: 0001-0009-rook-ceph-000000000000000d-797353de-777d-4377-a128-17ad48ec58f5 failed to get image id csi-ceph-blockpool/csi-vol-797353de-777d-4377-a128-17ad48ec58f5: image not found: RBD image not found

I see PVCs are being created with the new storage class:

$ k get pv | grep csi-ceph-block

pvc-03409965-c186-45cf-9168-10088da2b49b   50Gi       RWO            Delete           Bound    apt-cacher-ng/apt-cacher-ng                                                                 csi-ceph-block             2d19h
pvc-65e157e8-2064-411d-a82d-2867e21adcc7   1Gi        RWO            Delete           Bound    apt-cacher-ng/volsync-apt-cacher-ng-dst-cache                                               csi-ceph-block             2d19h
pvc-aa8647dd-4e32-4332-845d-4af42dcd9076   50Gi       RWO            Delete           Bound    apt-cacher-ng/volsync-apt-cacher-ng-dst-dest                                                csi-ceph-block             2d19h
pvc-e5b5c3f4-379f-40cf-9b20-3236133c394c   1Gi        RWO            Delete           Bound    apt-cacher-ng/volsync-apt-cacher-ng-cache                                                   csi-ceph-block             2d19h

Trying to understand what the errors are telling me.

[subhamkrai]

@Rakshith-R @iPraveenParihar ^^

[iPraveenParihar]

@reefland, AFAIK these are logged during the DeleteVolume operation.
It seems like when you tried to delete the PVC?, it called GRPC /DeleteVolume. But then, for some reason, /DeleteVolume was called again for the same volume.

The first request deleted the volume and its omap data. In the second request, since the volume and its omap data were already deleted, it resulted in an error indicating that the required data wasn't found, leading to the omap not found / failed to get image id message.

[reefland]

@iPraveenParihar - thank you, I appreciate you taking the time.