Timeline::shutdown can leave a dangling handle_walreceiver_connection tokio task

[problame]

Problem

diff --git a/pageserver/src/tenant/timeline/walreceiver.rs b/pageserver/src/tenant/timeline/walreceiver.rs
index 2fab6722b..2a2f9ab04 100644
--- a/pageserver/src/tenant/timeline/walreceiver.rs
+++ b/pageserver/src/tenant/timeline/walreceiver.rs
@@ -236,6 +236,10 @@ impl<E: Clone> TaskHandle<E> {
     async fn shutdown(self) {
         if let Some(jh) = self.join_handle {
             self.cancellation.cancel();
+            // we can stop being polled before the jh completes.
+            // it can happen because connection_manager_loop_step is in a tokio::select!() statement together with task_mgr cancellation:
+            // https://github.com/neondatabase/neon/blob/fb518aea0db046817987a463b1556ad950e97f09/pageserver/src/tenant/timeline/walreceiver.rs#L84-L104
+            // the consequence of that will be that we leave a tokio task that executes handle_walreceiver_connection running
             match jh.await {
                 Ok(Ok(())) => debug!("Shutdown success"),
                 Ok(Err(e)) => error!("Shutdown task error: {e:?}"),

Found together with @koivunej while investigating #7051

i.e., rapid /location_config that transition a tenant from Single to Multi and back with changing generation numbers, so, the code takes the slow path.

Consequence of this:

• the dangling task can continue to ingest WAL
• that means it can continue to try to
  • put / finish_write into the Timeline
    • which in turn can cause layers to be frozen & flushed
  • trigger on-demand downloads

We audited the codebase to ensure that, as of #7051 , none of the above will succeed after Timeline::shutdown returns.

Solution Proposal

Don't cancel-by-drop the connection_manager_loop_step.
Use shutdown_token() instead and make code responsive to it.

Impl

Tasks

Beta Give feedback

1. refactor(remote_timeline_client): infallible stop() and shutdown() #7234
   +0
2. fix(walreceiver): Timeline::shutdown can leave a dangling handle_walreceiver_connection tokio task #7235
   +0
3. refactor(walreceiver): eliminate task_mgr usage #7260
   +0
4. refactor(Timeline::shutdown): rely more on Timeline::cancel; use it from deletion code path #7233
   +0

[problame]

Update:

• Small preliminary refactoring & fix is merged
• Given I had the walreceiver code loaded into my head, it was opportune to refactor walreceiver such that it no longer uses task_mgr, because we want to do that for GA.
• Opened PR refactor(walreceiver): eliminate task_mgr usage #7260 for that, tracked in tasklist of this issue.
  • There's disagreement about the need/design around walreceiver early shutdown.
• Also opened further refactoring PR refactor(Timeline::shutdown): rely more on Timeline::cancel; use it from deletion code path #7233 which is stacked atop refactor(walreceiver): eliminate task_mgr usage #7260
  • It might fix pageserver: metrics lingering for deleted timelines #7221, which @jcsp has been working on and re-assigned to me since