storcon: add metric for long running reconciles

[VladLazar]

Problem

We don't have an alert for long running reconciles. Stuck reconciles are problematic
as we've seen in a recent incident.

Summary of changes

Add a new metric storage_controller_reconcile_long_running_total with labels: {tenant_id, shard_number, seq}.
The metric is removed after the long running reconcile finishes. These events should be rare, so we won't break
the bank on cardinality.

Related #9150

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

5022 tests run: 4864 passed, 0 failed, 158 skipped (full report)

---

Flaky tests (3)

Postgres 17

• test_pageserver_compaction_smoke: release-arm64
• test_multi_attach: debug-x86-64

Postgres 14

• test_multixid_wraparound_import: release-x86-64

Code coverage* (full report)

• functions: 31.3% (7492 of 23902 functions)
• lines: 49.6% (60118 of 121310 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
966d775 at 2024-10-01T09:18:27.981Z :recycle:}

[yliang412]

LGTM