Epic: seamless tenant migration between pageservers

[jcsp]

Motivation

This follows on from #5050. Whereas #5050 makes it safe to attach a tenant to a new pageserver without detaching the old one, this epic will make a tenant migration seamless from postgres's point of view.

See RFC: #5029

This ticket is the pageserver side of the work: control plane changes are described elsewhere:

• https://github.com/neondatabase/cloud/issues/7992

DoD

For failover (i.e. unplanned migration), only most recent LSNs become unavailable for reads. They become available again in a time bounded by the time taken for the new pageserver to download the writes since the remote_consistent_lsn on the old pageserver.

For planned migrations, postgres experiences no loss of read availability whatsoever.

Tasks

Beta Give feedback

1. Compute: live reconfiguration of pageserver Do not require compute restart to change pageserver connection string #4578
2. Improved concurrency for TenantsMap updates (pageserver: make TenantsMap lock synchronous, add explicit "busy" state to protect operations #5378)
3. Pageserver: implement flush & AttachedStale notification API, AttachedMulti mode ( pageserver: implement API for secondary locations, attachment modes #5379)
4. tests: start adding tests for secondary mode, live migration #5842 Pageserver: create automated tests that demonstrate the migration process
5. [WIP] Secondary locations heatmaps, downloads & tests #5745
   c/storage/pageserver t/feature +2
6. Deploy more pageservers to accommodate secondary locations
7. Carry out drill of whole node evacution in staging
8. pageserver: clean up code for pre-LocationConf configuration #5388
   a/tech_debt c/storage/pageserver +2
   
9. storage_controller: start adding chaos hooks #7946
   c/storage/controller t/feature +2
10. pageserver: additional live migration regression tests #5396
    a/test c/storage/pageserver +2

[koivunej]

Before we can enable generation usage and these, we will need to have a way to restrict consumption_metrics.rs from a pageserver with previous generation number.

[jcsp]

This is on pause to avoid merging more code until we can deploy existing changes, which will require https://github.com/neondatabase/cloud/issues/6600

[NanoBjorn]

@jcsp do you have an issue somewhere on re-attach and maybe validate modifications? As far as I understood from the RFC, we should return mode in /re-attach response, but generation number there should also become optional for the secondaries then.

And it is also unclear what should we do about /validate? If it is called in the middle of migration by old primary, when new primary is already AttachedMulti, but compute not yet switched, then pageserver will invalidate tenant, what will happen then with not yet switched compute?

[jcsp]

I just responded to the similar ping on https://github.com/neondatabase/cloud/issues/9614#issuecomment-1911064489 . There's no change to validate.

Including attachment states and/or secondary tenants in /re-attach responses is not essential for correctness here -- that's more of an optimization to enable tenants omitted in /re-attach to smoothly transition to secondary rather than detaching.