mgr/Mgr.cc: clear daemon health metrics instead of removing down/out osd from daemon state

[cfsnyder]

Reverts the change from #53993 and directly clears daemon health metrics for down and out OSDs. The former approach of removing down/out OSDs from the daemon state has undesirable consequences for stat output, including the prometheus exporter.

For example, the approach of removing the daemon from the daemon state causes the down/out OSD to not appear in prometheus output at all. An alert for detecting down/out OSDs that triggers when count(ceph_osd_up = 0) > 1 no longer works.

Fixes: https://tracker.ceph.com/issues/66168

Contribution Guidelines

• To sign and title your commits, please refer to Submitting Patches to Ceph.

• If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

• When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

• Tracker (select at least one)
  • References tracker ticket
  • Very recent bug; references commit where it was introduced
  • New feature (ticket optional)
  • Doc update (no ticket needed)
  • Code cleanup (no ticket needed)
• Component impact
  • Affects Dashboard, opened tracker ticket
  • Affects Orchestrator, opened tracker ticket
  • No impact that needs to be tracked
• Documentation (select at least one)
  • Updates relevant documentation
  • No doc update is appropriate
• Tests (select at least one)
  • Includes unit test(s)
  • Includes integration test(s)
  • Includes bug reproducer
  • No tests

Show available Jenkins commands

• jenkins retest this please
• jenkins test classic perf
• jenkins test crimson perf
• jenkins test signed
• jenkins test make check
• jenkins test make check arm64
• jenkins test submodules
• jenkins test dashboard
• jenkins test dashboard cephadm
• jenkins test api
• jenkins test docs
• jenkins render docs
• jenkins test ceph-volume all
• jenkins test ceph-volume tox
• jenkins test windows
• jenkins test rook e2e

[cfsnyder]

jenkins test make check

[cfsnyder]

jenkins test api

[cfsnyder]

jenkins test make check

[cfsnyder]

@neha-ojha ping. Any reason that no one has been assigned to review this PR yet?

[pdvian]

@cfsnyder I agree with your approach here. I tried to solve same problem in a different way with #50543. Your changes looks more appropriate to me.

[rzarzynski]

The commit description reuses https://tracker.ceph.com/issues/63195 (the ticket for the change it reverts) which would create a mess when backporting.

I think we need a new, dedicated ticket for this patch.

[cfsnyder]

The commit description reuses https://tracker.ceph.com/issues/63195 (the ticket for the change it reverts) which would create a mess when backporting.

I think we need a new, dedicated ticket for this patch.

I've created a new ticket and linked it in the commit and PR.

[ljflores]

Rados approved: https://tracker.ceph.com/projects/rados/wiki/MAIN#httpstrackercephcomissues66793

[ljflores]

jenkins test windows