mgr/prometheus: fix pool_objects_repaired and daemon_health_metrics format

[banuchka]

I created a bug report on Ceph Tracker - BUG #59505

This PR is a fix for:

• quincy: mgr/prometheus: expose daemon health metrics #49519
• quincy: mgr/prometheus: expose repaired pgs metrics #48204

[pereman2]

Looks great. Thanks @banuchka for taking care of this.

The Signed-off-by is missing from the commit message and the Fixes: too. If you can change that and past the current output of /metrics it would be awesome.

[pereman2]

Btw, just curious, is repeated headers an issue or just cumbersome?

[nizamial09]

btw @banuchka i see you targetted this PR for quincy only. Our normal way of workflow is to first create the PR in main and merge it there. Then cherry-pick those commits to stable branches like quincy or reef. Could you please target this PR to main first?

[banuchka]

Btw, just curious, is repeated headers an issue or just cumbersome?

@pereman2 It is a not valid output format for the Prometheus scrapper, as an example below:
error reading metrics for http://****:***/metrics: reading text format failed: text format parsing error in line 2010: second HELP line for metric name "ceph_pool_objects_repaired"

[banuchka]

btw @banuchka i see you targetted this PR for quincy only. Our normal way of workflow is to first create the PR in main and merge it there. Then cherry-pick those commits to stable branches like quincy or reef. Could you please target this PR to main first?

do I need to create a new PR or is it possible to change the target on the current one? (I cant find how to do that, my bad)

[github-actions]

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

[trociny]

do I need to create a new PR or is it possible to change the target on the current one? (I cant find how to do that, my bad)

You should set the backport field in the tracker ticket (I did it), and then when this PR is merged to the main branch and the ticket status is changed to "Pending backport" the backport tickets will be created automatically.

[idryomov]

@pereman2 Looking at this PR, I see additional issues with the original #47494 and #48843. Please see my comments there.

[idryomov]

@banuchka The update looks good, please squash it into the original commit and:

• amend the commit title to match the (new) title of the PR
• add some details (what is being fixed, perhaps quote error reading metrics for http://****:***/metrics: reading text format failed: text format parsing error in line 2010: second HELP line for metric name "ceph_pool_objects_repaired" error, etc) to the commit message

[banuchka]

@idryomov done
What do you about changing poolid to pool_id here as well? Or is it a separate issue(not an issue at all)?

[idryomov]

What do you about changing poolid to pool_id here as well? Or is it a separate issue(not an issue at all)?

I would say that it's a separate issue (and we still need to hear from @pereman2 on why this metric deviated from the rest in this regard).

[pereman2]

@idryomov I honestly didn't give much thought on it. I remember pushing those PR quickly so poolid was a clear mistake. I think it is great news if this can be fixed in this PR.

[banuchka]

@pereman2 @idryomov just let me know your decision and I'm happy with both ways:

1. change poolid to pool_id and force push here
2. create a new PR with the changes above

[idryomov]

@pereman2 @idryomov just let me know your decision

@pereman2 Could you please respond on #48843 (comment)? Then perhaps @banuchka could address all these issues in this PR and we could backport in one go.

[pereman2]

@idryomov done. Summary: gauge should be ideal there.

@banuchka I think you can add those changes in this PR as ilya pointed out, it will be easier for the backport.

[banuchka]

@pereman2 @idryomov done as we've discussed.

[banuchka]

@idryomov Now it should be better as I hope.

Dismissing since the new version needs to be retested. @pereman2 please test as you did before and ideally post the output here.

[pereman2]

# HELP ceph_pool_objects_repaired Number of objects repaired in a pool
# TYPE ceph_pool_objects_repaired counter
ceph_pool_objects_repaired{pool_id="9"} 0.0
ceph_pool_objects_repaired{pool_id="8"} 0.0
ceph_pool_objects_repaired{pool_id="7"} 0.0
ceph_pool_objects_repaired{pool_id="2"} 0.0
ceph_pool_objects_repaired{pool_id="1"} 0.0
ceph_pool_objects_repaired{pool_id="3"} 0.0
ceph_pool_objects_repaired{pool_id="4"} 0.0
ceph_pool_objects_repaired{pool_id="5"} 0.0
ceph_pool_objects_repaired{pool_id="6"} 0.0
# HELP ceph_daemon_health_metrics Health metrics for Ceph daemons
# TYPE ceph_daemon_health_metrics gauge
ceph_daemon_health_metrics{type="SLOW_OPS",ceph_daemon="mon.a"} 0.0
ceph_daemon_health_metrics{type="SLOW_OPS",ceph_daemon="mon.b"} 0.0
ceph_daemon_health_metrics{type="SLOW_OPS",ceph_daemon="mon.c"} 0.0
ceph_daemon_health_metrics{type="SLOW_OPS",ceph_daemon="osd.0"} 0.0
ceph_daemon_health_metrics{type="PENDING_CREATING_PGS",ceph_daemon="osd.0"} 0.0
ceph_daemon_health_metrics{type="SLOW_OPS",ceph_daemon="osd.1"} 0.0
ceph_daemon_health_metrics{type="PENDING_CREATING_PGS",ceph_daemon="osd.1"} 0.0
ceph_daemon_health_metrics{type="SLOW_OPS",ceph_daemon="osd.2"} 0.0
ceph_daemon_health_metrics{type="PENDING_CREATING_PGS",ceph_daemon="osd.2"} 0.0

looks good.

[idryomov]

jenkins test make check

[idryomov]

jenkins test dashboard

[rzarzynski]

jenkins test dashboard