storage controller: tenant scheduling policy

[jcsp]

Problem

In the event of bugs with scheduling or reconciliation, we need to be able to switch this off at a per-tenant granularity.

This is intended to mitigate risk of issues with #7181, which makes scheduling more involved.

Closes: #7103

Summary of changes

• Introduce a scheduling policy per tenant, with API to set it
• Refactor persistent.rs helpers for updating tenants to be more general
• Add tests

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2736 tests run: 2598 passed, 0 failed, 138 skipped (full report)

---

Flaky tests (2)

Postgres 16

• test_compute_pageserver_connection_stress: debug

Postgres 14

• test_pageserver_init_node_id: debug

Code coverage* (full report)

• functions: 28.1% (6310 of 22443 functions)
• lines: 46.9% (44318 of 94441 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
b273dce at 2024-03-28T14:33:29.767Z :recycle:}

[VladLazar]

Looks good. The most substantive comment is the one about inflight reconciles.