Leasing (S3)

[benbjohnson]

This pull request adds optional leasing to Litestream to allow it to run in ephemeral environments that don't have a persistent disk. This is implemented using conditional writes that are recently available in S3. Conditional writes have already been supported by some S3-compatible object stores such as Tigris Data.

How it works

Time-based leasing is a method of leader election that ensures that only one instance of Litestream is running at a time. On startup, a lease is acquired and continually renewed while the litesteam process is running. If Litestream fails to renew a lease, it will exit the process immediately since it cannot safely backup.

When another Litestream process starts up and sees an existing lease, it will continually retry the lease acquisition every second until it succeeds. This low retry interval allows for rolling restarts to come online quickly.

If Litestream exits normally then it will release the lease it had previously acquired and another process will be able to take over immediately. If Litestream has an unexpected exit then the lease will continue to exist until the timeout is reached. The lease timeout defaults to 30 seconds.

Usage

To enable leasing, configure it in your litestream.yml file:

lease:
  type: "s3"
  url: s3://mybkt/lease

The path of the URL (e.g. /lease) is the coordination point so any other Litestream processes will block if they are started with the same bucket & path and another process has already acquired the lease.

[rishi-kulkarni]

Out of curiosity, is there a way to avoid missing writes on the database B in a rolling restart scenario? Let's say my restart procedure is: 1. Restore db, 2. Wait for replication to begin, 3. Start my application. While leasing solves the multiple writers issue in step 2, the local copy of the db will miss any writes that went to the prior deployment between the first two steps, right?

[jamesgardnerfry]

Would love to see this enhancement completed and merged in.

It seems pretty important for deployments to platforms like Fly or LightSail where there is a risk that during a deployment a second container is temporarily spun up and the old one is not shut down until a health check passes. We don't want two containers writing to the same bucket at once.

Am I right in thinking that this risk can be minimised on Fly with:

fly deploy --ha=false --strategy immediate --update-only

Is there anything else that could make it safer?

Does anyone have an examples of other workaround such as using a DynamoDB or S3 conditional write as part of the wrapper script to litestream?