pageserver: label tenant_slots metric by slot type

[jcsp]

Problem

The current tenant_slots metric becomes less useful once we have lots of secondaries, because we can't tell how many tenants are really attached (without doing a sum() on some other metric).

Summary of changes

• Add a mode label to this metric
• Update the metric with slot_added and slot_removed helpers that are called at all the places we mutate the tenants map.
• Add a debug assertion at shutdown that checks the metrics add up to the right number, as a cheap way of validating that we're calling the metric hooks in all the right places.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2880 tests run: 2759 passed, 0 failed, 121 skipped (full report)

---

Flaky tests (1)

Postgres 16

• test_vm_bit_clear_on_heap_lock: debug

Code coverage* (full report)

• functions: 28.0% (6622 of 23608 functions)
• lines: 46.7% (47101 of 100779 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
9f4271a at 2024-05-03T14:22:29.373Z :recycle:}