storage controller: robustness improvements

[jcsp]

Problem

Closes: #6847
Closes: #7006

Summary of changes

• Pageserver API calls are wrapped in timeout/retry logic: this prevents a reconciler getting hung on a pageserver API hang, and prevents reconcilers having to totally retry if one API call returns a retryable error (e.g. 503).
• Add a cancellation token to Node, so that when we mark a node offline we will cancel any API calls in progress to that node, and avoid issuing any more API calls to that offline node.
• If the dirty locations of a shard are all on offline nodes, then don't spawn a reconciler
• In re-attach, if we have no observed state object for a tenant then construct one with conf: None (which means "unknown"). Then in Reconciler, implement a TODO for scanning such locations before running, so that we will avoid spuriously incrementing a generation in the case of a node that was offline while we started (this is the case that tripped up test_basebackup_with_high_slru_count fails due to attachment service reconcile race #7006)
• Refactoring: make Node contents private (and thereby guarantee that updates to availability mode reliably update the cancellation token.)
• Refactoring: don't pass the whole map of nodes into Reconciler (and thereby remove a bunch of .expect() calls)

Some of this was discovered/tested with a new failure injection test that will come in a separate PR, once it is stable enough for CI.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2567 tests run: 2434 passed, 0 failed, 133 skipped (full report)

---

Flaky tests (3)

Postgres 15

• test_pg_regress[None]: debug
• test_empty_tenant_size: debug

Postgres 14

• test_multi_attach: debug

Code coverage* (full report)

• functions: 28.8% (7009 of 24365 functions)
• lines: 47.5% (43162 of 90949 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
d59b3d1 at 2024-03-07T14:52:07.086Z :recycle:}

[VladLazar]

Very nice