How do we manage HA with litestream?

[fnzv]

Hello there!
I read the documentation and litestream seems a great idea but how it's possible to manage failover in case the primary node fails?
I need to keep stand-by nodes constantly running the S3 restore procedure and balance the traffic in case of failure?

Thank you for your time

[benbjohnson]

There's not currently a failover mechanism but there's an issue for hot replicas (#1) that would maintain a live copy of the database. Between that and live read replicas (#8) you could have a standby although you'd need to manage the traffic redirection. Those features will probably be in the v0.4.0 release.

I would like to put together a guide for setting this up at some point. It depends on your exact infrastructure but I think it'd be useful to give folks a basic idea of how to do it generally.

[fnzv]

Thank you i will definitely subscribe to those issues, at the moment i just discovered this project but i am willing to test it out and see if this could be an alternative to replace mysql orchestrator (paradigm) with more simplified cluster mgmt and dbs

Looking forward for that release 👍

[simon-weber]

[I'm not sure what GH Discussions etiquette is for follow-up on Q&A's; feel free to move this to a new topic, @benbjohnson]

I've been thinking about this a bit in the context of k8s and litestream's current features. A non-HA web setup could look something like:

• limit the webapp to a specific node/host
• webapp pods access sqlite via a local path volume
• litestream replicates the db to s3
• on pod startup:
  • if there's no local db and one to replicate: restore it
  • if there's a local db and one to replicate: assume local is latest and don't restore*

Things get more complicated if you allow the pod to migrate if its host goes down, since you could end up with multiple active dbs**. What do folks think about something like this:

• if litestream ever makes a new generation during replication, shut down the app: if I understand Expected behavior with two sources and one replica? #61 and the generation docs correctly, this should only+always happen if there are multiple active dbs, so long as dbs are only initialized once and restored afterwards.
• if there are multiple generations during startup, don't allow the app to start: this should prevent new apps coming up on a "wrong" generation.
• the above cases alert so that I can manually move/delete/merge the wrong generation in s3. Then, I shut down the app so that it returns to its original host, then delete the db from the old host.

Did I miss anything big? Or - probably more likely - am I overthinking things?

* is there a way to detect whether the local db is even/ahead/behind the latest replicated db without restoring it first?
** I'm picturing something like a host getting cut off from the network, so that the app is migrated but the original host later returns with an outdated db. I'm not that confident in preventing writes via traffic redirection, and even if I were, something like a cron job could still write the outdated db.

[benbjohnson]

I've been thinking about this a bit in the context of k8s and litestream's current features

I think what you stated is a good approach. I was thinking of running an application with Litestream as a k8s StatefulSet so that it ensures there's only one at a time. Once live read replication is added it would also let you distinguish between a primary node (e.g. myapp-0) and replica nodes (e.g. myapp-1, myapp-2, etc).

Things get more complicated if you allow the pod to migrate if its host goes down, since you could end up with multiple active dbs

I've been debating different approaches for this. Ultimately what's needed is some kind of distributed lock or a distributed lease. It'd be nice if S3 provided a create-if-not-exists API but unfortunately they don't.

Regarding generations, it can be valid to have multiple generations. Snapshots can be stored for a period of time (e.g. a day) so if you were to recover a failed node and it started a new generation then you may want to keep the previous generation(s) for that time period. Litestream could implement a leasing strategy with S3 where it writes a "lock file" and it periodically updates it to maintain that lock. There are some race conditions with that approach but it would probably work well enough.

I've also considered adding a separate service folks could use as a distributed lock. It would essentially register a server as the current primary for a given replica URL (e.g. s3://mybkt/mypath) and it would prevent any other server from using that same path while the primary is registered. That service would also help with a serverless implementation so only one function can write at a time.

[simon-weber]

Thanks for the detailed response!

Ultimately what's needed is some kind of distributed lock or a distributed lease

Yeah, I guess so. I'm tempted to try doing it outside of litestream with dynamo, since there are plenty of locking libraries and I'm a sucker for squeezing things into free tiers 😁

Regarding generations, it can be valid to have multiple generations.

Gotcha. Can this happen during uninterrupted replication, maybe due to wonky retention settings? Or should it only happen if a WAL frame is missed for some reason (litestream restarted, node failure, etc)?

[benbjohnson]

It shouldn’t happen during uninterrupted replication. It would only happen if there was a broken lock implementation (e.g old versions of NFS).