mgr: prometheus: set metadata metrics value to '1' (#22717).

[k0ste]

By default we have pool metrics like this:

ceph_pool_objects{pool_id="1"} 1075534
ceph_pool_metadata{name="name_of_pool_1",pool_id="1"} 0

When I was try to join name label from metadata series (to use label with Grafana) like this:

ceph_pool_objects * on (pool_id) group_left(name) ceph_pool_metadata

Result returned without metric, only labels:

{name="name_of_pool_1",pool_id="1"}

This is because metadata metric always have value '0'.
This behavior will be fixed if set value to '1'.

http://tracker.ceph.com/issues/22717

[jcsp]

Hmm, I had tested the ceph_disk_occupation stuff with an and on query, I wonder if that query type just doesn't care about the value (vs normal on?)

Anyway, this change looks reasonable to me

[jan--f]

When I was try to join name label from metadata series (to use label with Grafana) like this:
ceph_pool_objects * on (pool_id) group_left(name) ceph_pool_metadata

Do you execute this query in Grafana or prometheus? I do get a metric value (its 0), in Grafana that is not obvious though.

So the reason why this behaves like this is due to the use of *. This operator multiplies the two metrics where the labels match. Using + should work as expected with the current metadata metric value. Generally its better to use the and operator as described in the documentation since this does not rely on the metadata metric having the right neutral element for the arithmetic operation you choose and the group_left is not needed.

While I don't have a strong opinion whether this metadata metric value should be 1 or 0, the initial motivation was to keep the value neutral in a graph...so 0. I.e. should someone add that metric to a graph (why would be beyond me) it would simply be 0.

[k0ste]

Please, see screenshot at http://tracker.ceph.com/issues/22717
Should answer your questions about Grafana.

Difference between and and * query:

When and: labels from ceph_pool_metadata can't be used:

And when *: label name from ceph_pool_metadata joined to answer:

Now see how it look on graph (14 Jan is when I set metadata metrics to '1'):

Obviously '0' returned instead ceph_pool_objects metric value.

[jcsp]

If you use the + instead of *, does that preserve the labels and do what you want?

[jcsp]

When I look at https://www.robustperception.io/how-to-have-labels-for-machine-roles/ and https://www.robustperception.io/exposing-the-software-version-to-prometheus/, they use a 1 value for all the metadata series, so I think we may find more people wanting to use the * operator like in the prometheus docs examples -- I'm inclined to merge this PR for that reason, even if it isn't strictly necessary. Okay with you @jan--f ?

[k0ste]

If you use the + instead of *, does that preserve the labels and do what you want?

Yes...
Now I even more hate the documentation of Prometheus.

[jan--f]

@jcsp yes agreed

[liewegas]

http://pulpito.ceph.com/sage-2018-01-22_16:28:42-rados-wip-sage3-testing-2018-01-22-0845-distro-basic-smithi/#