Create Prometheus exporter to export PV to RBD/CephFS mappings

[dillaman]

Describe the feature you'd like to have

The logic of the existing tracevol.py tool should be incorporated into a prometheus exporter to facilitate the mapping between PVs and backing RBD image and CephFS subvolumes.

What is the value to the end user? (why is it a priority?)

Tools like the RBD and CephFS top should be able to translate the internal RBD/CephFS subvolume names back to k8s PVs for tracking down noisy neighbours. The RBD "top" stats that are already exposed via its own Prometheus exporter from the MGR and the Ceph Dashboard includes embedded grafana graphs for showing top X image stats but that doesn't help track down the matching PV.

How will we know we have a good solution? (acceptance criteria)

We should be able to trivially map an RBD image / CephFS subvolume to its matching PV.

Additional context

(example Ceph Dashboard showing RBD image stats -- it would be nice to be able to develop something similar that maps images to PVs).

[nixpanic]

See #72

[chenxu1990]

�we can easily get subvolumeName or rbdImageName from pv object #957 . So we just need to list
all pvs and export the mappings ?

[Madhu-1]

�we can easily get subvolumeName or rbdImageName from pv object #957 . So we just need to list
all pvs and export the mappings ?

This will not work for the older PVC which got created prior to it

[chenxu1990]

�we can easily get subvolumeName or rbdImageName from pv object #957 . So we just need to list
all pvs and export the mappings ?

This will not work for the older PVC which got created prior to it

So we should get the mappings following the logic of the existing tracevol.py tool. I would like to take this feature

[Madhu-1]

Thank you sure, assigning this to you @c3y1huang

[chenxu1990]

How about putting the mapping information into metric image_size, such as 'image_size(name=xx, pool=xx, pvc_name=xx, pv_name=xx) 20000' ?

[Madhu-1]

@dillaman this is new to me, do we have any sample format or already existing Prometheus exporters we can follow?

[dillaman]

I don't have any concrete examples. Perhaps something like:

ceph_csi_rbd{pvc_namespace="abc", pvc_name="bcd", rbd_pool="cde", rbd_namespace="", rbd_image="def"} 1

The important thing is we need some type of metric that would allow a PVC to be looked via an RBD image name so that a Grafana dashboard could show the k8s PVC name instead of the RBD image name (i.e. find the k8s noisy neighbour instead just the RBD image and making the admin chase down the associated PVC).

[chenxu1990]

I have two solutions.
One, get all pvc then get related image
Two, get all images then get related pvc
which solution is better and if the data collection will
cause performance problems when the total number is large ?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

[mykaul]

That's a very important feature for us. @Madhu-1 - who can look at this?

[Madhu-1]

@mykaul sure. we are targeting this for the next release.

[mykaul]

1. Can we do in ceph-csi some naming convention, since we cannot rename a PVC after creation, it might be useful?
2. Alternatively, can we use rbd metadata to store the name ('rbd image-meta set') ?

[Madhu-1]

1. Can we do in ceph-csi some naming convention, since we cannot rename a PVC after creation, it might be useful?

Not sure I get the question of what naming you are referring to here?

2. Alternatively, can we use rbd metadata to store the name ('rbd image-meta set') ?

Yes, we can do it for RBD, even we can use rados Omap(for both cephfs and RBD) to set some key-values for PVC.

[mykaul]

I'm trying to answer the above question, on how to do it properly @ scale - clearly looking at all images makes less sense - and I'm also unsure about all PVs (perhaps in a specific class?). Perhaps by using the metadata we can do some filtering.

As for the naming convention - I was wondering if it makes sense to include the PV name (as suffix?) to the rbd volume name.

[Madhu-1]

I'm trying to answer the above question, on how to do it properly @ scale - clearly looking at all images makes less sense - and I'm also unsure about all PVs (perhaps in a specific class?). Perhaps by using the metadata we can do some filtering.

Thanks, @mykaul will check what would be the easy and more convenient way to do this one. example:- One can statically Bind the PVC to an already existing PV(Retained PV) we need to see how to handle that one also.

As for the naming convention - I was wondering if it makes sense to include the PV name (as suffix?) to the rbd volume name.

Currently, we are storing the PV name to RBD image/cephfs subvolume name mapping in the rados omap. This is helpful if we just want to export PV names, RBD image names not PVC names. Even We are marking the image/subvolume names in the PV object itself.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

[nirs]

@Madhu-1 #1894 was closed assuming that it is a dup of this, but this was closed as stale.

Is this solved in current version?

[Madhu-1]

@nirs We dont want to create the prometheus exported in the cephcsi itself, cephcsi adds the required details like (pvc name, pv name and namespace name and clusterid) to the rbd image/cephfs subvolume metadata. you can write your own exporter to list all the images and extract the requried information. In future ceph will expose the metadata on the image/subvolume as part of its metrics.

[ctml91]

In future ceph will expose the metadata on the image/subvolume as part of its metrics.

Where is this being worked on? TIA