fix(tenant/timeline metrics): race condition during shutdown + recreation

[problame]

Tenant::shutdown or Timeline::shutdown completes and becomes externally
observable before the corresponding Tenant/Timeline object is dropped.

For example, after observing a Tenant::shutdown to complete, we could
attach the same tenant_id again. The shut down Tenant object might still
be around at the time of the attach.

The race is then the following:

• old object's metrics are still around
• new object uses with_label_values
• old object calls remove_label_values

The outcome is that the new object will have the metric objects (they're
an Arc internall) but the metrics won't be part of the internal registry
and hence they'll be missing in /metrics.

Later, when the new object gets shut down and tries to
remove_label_value, it will observe an error because
the metric was already removed by the old object.

Changes

This PR moves metric removal to shutdown().

An alternative design would be to multi-version the metrics using a
distinguishing label, or, to use a better metrics crate that allows
removing metrics from the registry through the locally held metric
handle instead of interacting with the (globally shared) registry.

refs #7051

[github-actions]

2490 tests run: 2365 passed, 0 failed, 125 skipped (full report)

---

Code coverage* (full report)

• functions: 28.8% (7025 of 24428 functions)
• lines: 47.6% (43445 of 91322 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
89b177e at 2024-03-08T12:30:32.435Z :recycle:}

[koivunej]

Interesting there was no test failures, but I guess that only means we don't have a test creating two tenants/timelines with the same id.

[koivunej]

Ok, this is interesting. We have yet to merge this. Still, on staging, there are a few "forever attaching" tenants in metrics. I will have to dig deeper then.

EDIT: I understood what causes this; we never reset the time series on startup: #7084