storage controller: test for large shard counts

[jcsp]

Problem

Storage controller was observed to have unexpectedly large memory consumption when loaded with many thousands of shards.

This was recently fixed:

• controller: limit Reconciler concurrency #7493

...but we need a general test that the controller is well behaved with thousands of shards.

Closes: #7460
Closes: #7463

Summary of changes

• Add test test_storage_controller_many_tenants to exercise the system's behaviour with a more substantial workload. This test measures memory consumption and reproduces storage controller using ~500kb memory per shard #7460 before the other changes in this PR.
• Tweak reconcile_all's return value to make it nonzero if it spawns no reconcilers, but would have spawned some reconcilers if they weren't blocked by the reconcile concurrency limit. This makes the test's reconcile_until_idle behave as expected (i.e. not complete until the system is nice and calm).
• Fix an issue where tenant migrations would leave a spurious secondary location when migrated to some location that was not already their secondary (this was an existing low-impact bug that tripped up the test's consistency checks).

On the test with 8000 shards, the resident memory per shard is about 20KiB. This is not really per-shard memory: the primary source of memory growth is the number of concurrent network/db clients we create.

With 8000 shards, the test takes 125s to run on my workstation.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2955 tests run: 2821 passed, 0 failed, 134 skipped (full report)

---

Flaky tests (2)

Postgres 15

• test_vm_bit_clear_on_heap_lock: debug

Postgres 14

• test_basebackup_with_high_slru_count[github-actions-selfhosted-sequential-10-13-30]: release

Code coverage* (full report)

• functions: 28.1% (6574 of 23355 functions)
• lines: 46.9% (46728 of 99639 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
a2ff93b at 2024-04-30T15:35:30.523Z :recycle:}

[jcsp]

On my workstation:

• Runtime of the test is 1125.35s ( 8000 tenants with 2 shards each)
• Most of that runtime is the reconciles that happen after turning the world upside-down by marking all the nodes as offline then active. I'm not sure exactly where the slowness is coming from: spot checking individual shards' migrations, I see them taking less than a second, so we should be doing them all in ~120 seconds, but it's taking several times longer than that. Not a blocker for this PR, but worth figuring out in future, as the rate at which we execute migrations will impact the rate at which we can drain nodes when we're in a hurry.

[VladLazar]

Nice test!

[jcsp]

When running on slower nodes in CI, this demonstrated that the default heartbeat period is currently not long enough to gracefully handle a pageserver restart.

Opened #7552 to track this.

[VladLazar]

Looks good to me once tests pass.