Epic: pageserver: LSN "leases" to block GC temporarily

[jcsp]

What

Provide a pageserver API that enables temporarily preventing GC from proceeding past some arbitrary LSN.

Why

Two use cases:
A) Branch creation, where one might use get_lsn_by_timestamp to find an LSN, and then try to create a branch at that LSN: it's an awkward API experience if that LSN might have been GC'd between the two API calls
B) Where someone creates a temporary postgres instance that targets a particular LSN, but doesn't want to create a durable branch at that LSN (this is a good behavior that we should encourage, as branches have a cost).

Implementing this will enable safe use of ephemeral endpoints, so that when we need a read-only endpoint to a particular LSN, we don't have to take the overhead of creating a full-fat branch to do that.

How

Design doc at https://www.notion.so/neondatabase/LSN-Lease-Design-f8aa8333a9b7431d9905785ba7599745?pvs=4.

The API could look something like:

• In APIs like get_lsn_by_timestamp that return an LSN, implicitly grant the caller a lease for e.g. 60 seconds, and indicate that in the return value so that it's clear how long they have the lease for
• Also add an explicit get_lsn_lease API, for case B, where we might currently not get any HTTP API calls at all, the endpoint will just start and send page_service requests at its fixed LSN.

During GC, we would consult an in-memory map of leases, and set our cutoff lsn to min(cutoff_lsn, min(valid leases))

Leases are strictly advisory and are in-memory objects. To avoid issues across pageserver restarts, we may delay all GC by at least the default lease period (e.g. 60 seconds) at pageserver startup, so that we implicitly uphold any leases promised before restart.

computectl would consume this API as a kind of "heartbeat" when running as an ephemeral endpoint, to prevent GC of its LSN as long as it runs.

Synthetic size calculation should also account for any leased LSNs, to avoid letting users get free retention by running a small ephemeral endpoint for a long time.

Tasks

Beta Give feedback

1. Implement dummy lease API implementation #7808
   c/storage/pageserver t/feature +2
   
2. [DNM/PoC]: use lsn leases to temporarily block gc #7996
   +0
   
3. Implement real lease logic: some map of LSNs to times in Timeline, plus delaying GC by the lease period on startup. #8063
   c/storage/pageserver t/feature +2
   
4. Grant optional lease to get_lsn_by_timestamp request #8072
   c/storage/pageserver t/feature +2
   
5. Update synthetic size calculation to account for leases. #8071
   c/storage/pageserver t/feature +2
   
6. Use metrics to track lease usage #8120
   c/storage/pageserver +1
   
7. Make leases interoperate with live migration: we should wait for the lease duration after transitioning to AttachedSingle, before doing any GC

(Note: this ticket is not related to read replicas: they need a stronger feedback mechanism a la #7368)

[hlinnaka]

This would work for read replicas too AFAICS

[jcsp]

This would work for read replicas too AFAICS

Sort of: if some thing called into the API on their behalf regularly. Maybe the implementations can kind of converge: we can get the standby feedback via the path in #7368, but the place we actually check this during GC would be the same for leases and for the standby_horizon

[hlinnaka]

Yeah, we could have different APIs for creating and refreshing the leases, with the same concept and implementation internally.

The compute doesn't currently make any HTTP requests to the pageserver. The HTTP API ports are blocked from the compute, so they cannot. So we might need to add it to the libpq-based protocol.

[jcsp]

Plan:

1. Add a dummy lease API to the pageserver for integration purposes, doesn't have to do anything yet
   • Arpad did this in Implement dummy lease API implementation #7808
2. Add code in computectl to call into it periodically: as Heikki says, we need to figure out the network/auth story here. I think I'm okay with computes having HTTP access to pageserver as long as we only issue them tenant-scoped JWT tokens, but we could shoehorn this into the libpq protocol if we really had to.
   • @prepor will do this
3. Then implement real lease logic: an in-memory set of LSNs to retain on the Timeline object, and a delay after restart to not move the GC offset forward until the lease period has expired. The pageserver will control the lease period, and we anticipate setting this to something in the 10-60m range to make the lease "heartbeats" from computectl super rare.
   • @yliang412 is going to pick this up.

[kelvich]

Add code in computectl to call into it periodically: as Heikki says, we need to figure out the network/auth story here. I think I'm okay with computes having HTTP access to pageserver as long as we only issue them tenant-scoped JWT tokens, but we could shoehorn this into the libpq protocol if we really had to.

optionally compute_ctl can you postgres JWT that is used for getpage requests

[yliang412]

This week:

• Implement the LSN lease logic for real.
• Integrate leases into get_lsn_by_timestamp.
• Design how leases fit into the synthetic size calculation.

[Shridhad]

This week:

• Changes implemented. Found a small bug in the page server tests - Waiting for PR to be reviewed
• Might need a refactoring to fix the configuration-related issue. @prepor discussing the change with @jcsp