Repository navigation
Live read replicas #8
Description
Activity
Would it be possible to sign frames with a private key? Think SQLite as a lightweight blockchain database. Read replicas could subscribe to a database by its public key identifier and then grab the latest frames over p2p transport of choice. There would be a ton of opportunities to use this for dapps.
Reacted by Luke Ramsden, Patrick Recher, Kamil Chmielewski, Valtteri Karesto, jdrydn and Jaanek OjaReacted by Adam Stevenson, boourns, dozn, Sibi Prabakaran, Burke Libbey, Oliver Severin Mulelid-Tynes, renthog, Sven Sauleau, Nathaniel Sabanski, IronSpecs and 1 moreReacted by Adam Quinn Snow and Noorain PanjwaniYou could sign WAL frames with a private key but I'm not sure I understand the use case. What's the benefit rather than streaming the WAL frames from a known source over HTTPS?
Signing the WAL before uploading could give a nice protection against the S3 bucket being compromised.
This should probably be a new issue though.
Reacted by Patrick Baxter@philips That's a good point. I was thinking about adding encryption too so it would mitigate an S3 compromise.
Reacted by Patrick Baxter, Mohamad Fadhil and Tuan Anh TranYou could sign WAL frames with a private key but I'm not sure I understand the use case. What's the benefit rather than streaming the WAL frames from a known source over HTTPS?
In the decentralized app world people are working on how do you write apps that are decentralized but still have good user experience without the severe limitations of blockchain apps. Now imagine if hackernews/reddit was open source golang with a sqlite database-- and every database change to hackernews is signed and broadcast. Any person could quickly stand-up a near-realtime clone at myhackernews.com. What's powerful is that now the entire community is forkable. If hackernews does something bad all it takes is myhackernews.com to alter the code/data and essentially fork the entire site. Or take it down another level. What if when you created a subreddit you were actually creating a new sqlite database with a keypair. Now it doesn't matter if reddit censors your subreddit. If users are represented by keypairs and (with this new litestream feature) all of their collective action is moderated by a keypair then it allows the person running the app to build out any features they want (without the constraints of traditional blockchains) but it keeps them honest because baked into the software anybody can "fork" the entire site and other users can decide to follow the new keypair if they want. Mastadon and other federated models fix some problems but you are still beholden to the operator of that particular domain. Git/github changed the game by making everything forkable. If entire website communities were forkable nobody would want to use anything else because it would keep operators honest and truly return ownership of shared data to the community.
Reacted by Jaanek OjaReacted by boourns, dozn, Mikkel Mansa Vilhelmsen, Xavier Petit, Conor Dockry, aliyome, Alexander Lyon, Adam Quinn Snow, Anik Das, Jannis and 11 moreI added a new issue for encryption and digital signatures for Litestream: #88
Thanks, very much awaited feature indeed. I'm a long time user of SQLite on ZFS, and proponent and contributor to ZFS filesystem project , offering streaming replications and snapshots, and would really love to see it being executed on database level, rather than on filesystem.
Reacted by Lambros Petrou and Justin (shocknet)Reacted by Ben JohnsonThis a network replication endpoint (probably
http) as well as figuring out how to apply WAL writes to a live database.Would it be possible to support read replicas just using the existing S3 bucket mechanism?
I'd love to be able to have Litestream push changes up to an S3 bucket (as it does now) and then have a separate read-only Litestream that polls that bucket for changes and uses them to continuously update a copy of that SQLite database file elsewhere.
I guess the concern with this would be cost - but would polling S3 for changes once a minute really be a significant cost over time? I've not done the napkin maths.
Reacted by Roger Hoover, Justin (shocknet), thestalnaker and Dave CahillReacted by Kurt Mackey, Roger Hoover and IronSpecs@simonw Yeah, that's definitely something I want to add with read replicas. There's about 108,000 minutes per month and each
LISTrequest costs $0.000005 so that comes out to $0.54/month/replica.There's a separate issue though in that you'd need to check for new generations as well in case replication was interrupted for some reason. That's much less likely so maybe that could check every hour instead to keep costs lower.
Reacted by Andrew Berezovskyi, Simon Eskildsen, Delaney and aliyomeAnother fun opportunity for this is to take advantage of it on Fly.io volumes.
I did a bunch of work recently to get Datasette working with those, but it currently only makes sense if you have a single instance - so you don't get to take advantage of Fly's ability to load balance your app between multiple regions: https://simonwillison.net/2022/Feb/15/fly-volumes/
What I'd really love to do is build a SQLite emulation of the way Fly handles PostgreSQL replication as described in https://fly.io/blog/globally-distributed-postgres/ - so one of my Fly instances would have the SQLite database that accepts writes, but every other instance would maintain a replicated read-only copy via Litestream.
Any time someone tried to write to the DB, I would ensure that their HTTP request went to the location with the write database.
Reacted by Matt Joiner, Koen Bok, Louis, Scott M., Ben Smith, Alejandro Baez, Ganesh Iyer, Fred Turkington, Dave Cahill, witt and 1 moreI think this may have been closed prematurely.
10 remaining items
I'm working on a different (but related) system for doing high availability with SQLite.
@warmchang Not quite. Those both use Raft to replicate. The system I'm working on is using a two-node synchronous replication setup similar to what you'd find in Postgres or MySQL synchronous replication. I'll have more details in the near future once I get a little further along with it.
Reacted by William Zhang, Can Rau, Paul Banks, Sam Isaacson, Andrej Fink, mike-wilson-hudson, Roman Heinrich, Tony Stenberg, Max Milton, Nathaniel Sabanski and 1 moreReacted by Nicolas Grilly, Olivier Mengué, Mohamad Fadhil, Nick Randall, Hugo Dias, Tuan Anh Tran, mayowa, Edgardo Alvarez, Can Rau, xuo and 6 moreIn case anyone else makes the same mistake I did: live read replicas are not supported in litestream 0.3.8 (the latest non-beta release as of 2022-06-26), but they are supported at 0.4.0-beta2 and at
main.Reacted by Justin (shocknet)hey folks, I wanted to give an update on live replication. The original implementation had some flaws so I reworked it, however, the new implementation still has some limitations & foot-guns that I'm not comfortable with. For example, you can only have one fixed primary. Also, you can corrupt your replicas by writing to them. Unfortunately, this is a consequence of Litestream running as a separate process so there's only so much it can do to manage those limitations.
As some of you may know, I've been working on a next generation tool for running SQLite on a distributed platform and, after some discussion, I've decided to keep Litestream as a pure disaster recovery tool and not include live replication. It's going to be difficult to maintain it long term—especially when when the new tooling will work better for this use case.
The bones of the new tooling are in place and we're moving it to internal testing this week. We're focusing on releasing a small but stable feature set at first and we'll grow it over time. I'll announce more here soon once we finish initial testing.
Thank you to everyone who has given feedback and tested out live replication in Litestream. I'm excited about the future of SQLite tooling and I hope y'all are too!
Update: We’ve open sourced the new replication tool which is called LiteFS. You can find it here: https://github.com/superfly/litefs
Reacted by Sam Wilson, Max Milton, Sebastian Szturo, Can Rau, laiso/Kazuya HIRASAWA, Mike Krieger, Ted Hahn, Coen Wessels, Alan Hollis, Gonçalo Mendes Cabrita and 32 moreReacted by Can Rau, laiso/Kazuya HIRASAWA, Virgiel, Ángel Sanz, swlkr, Trent, Aldira Putra Raharja, Kyle Conrad, Felipe Castro, Lukas Werner and 4 moreReacted by Fadhil, Ivan Kanakarakis, Can Rau, laiso/Kazuya HIRASAWA, Yann Coleu, Virgiel, Zdenko Vrabel, Santiago Fraire Willemoes, swlkr, Trent and 12 moreReacted by Can Rau, laiso/Kazuya HIRASAWA, Virgiel, Artyom Pervukhin, swlkr, Trent, Aldira Putra Raharja, Kyle Conrad, Felipe Castro, Tony Stenberg and 1 moreReacted by Can Rau, laiso/Kazuya HIRASAWA, Scott Block, Virgiel, GalaxySnail, Trent, Milan, Aldira Putra Raharja, Kyle Conrad, Felipe Castro and 3 moreThanks for the work Ben. Looking forward to the next-gen tooling. Keep us updated!
Reacted by Ben Johnson, Can Rau and Tony StenbergI'm working on a different (but related) system for doing high availability with SQLite.
@warmchang Not quite. Those both use Raft to replicate. The system I'm working on is using a two-node synchronous replication setup similar to what you'd find in Postgres or MySQL synchronous replication. I'll have more details in the near future once I get a little further along with it.
Would that mean write replicas or still read only replicas?
@CanRau It will still be a single primary node that can accept writes at a time but it is able to hand off that responsibility to another node without downtime during deploys.
Reacted by Ivan Kanakarakis and TrentReacted by Max Milton, Nicolas Grilly, Trent and Tony Stenberg@gedw99 Good question. I can’t believe I forgot to mention that part in my original comment. The new tooling is not fly.io specific. It is Apache 2 licensed. It will be able to run on any “stateful” server platforms like fly.io, Kubernetes, DigitalOcean, etc. We are also targeting pure serverless platforms like Vercel and Deno in the future too.
Reacted by Ivan Kanakarakis, Max Milton, Jannis, Virgiel, kohms, Trent, Jouni K. Seppänen, Ángel, kimihito, Kyle Conrad and 4 moreReacted by Nicolas Grilly, Trent, Ángel, Jacob Tallberg, Kyle Conrad, Tony Stenberg, Bryant Luk, David Oliver and IronSpecsI've decided to keep Litestream as a pure disaster recovery tool and not include live replication. It's going to be difficult to maintain it long term—especially when when the new tooling will work better for this use case.
Good call! 👍 to KISS & the UNIX philosophy. This theme continues in shipit.show/61 & shipit.show/62 (not out yet).
New link added to show notes:
Reacted by kohms, Kyle Conrad, Rio Kierkels, Tony Stenberg and leowmjwWhat is the reasoning behind not supporting leader election? CRDB is doing it really well. Is it due to latency or just overall complexity?
Hey there @benbjohnson , sorry to bother you, but I wanted to ask you something.
I understand you've trimmed the scope of this project into a backup system and moved on to work on LiteFS. My question is: Do you believe this approach of having an external process monitoring and sending changes to replicas is inherently flawed and can never work, or you switched strategies just looking for a lower hanging fruit?
I ask because all other replication systems seem to work by wrapping sqlite with some process (except of course for LiteFS). This repo was quite unique in that regard: it was a parallel process. It could have given us a way of creating large edge systems with custom topologies at a very low cognitive (and $) cost.
If anyone was inclined to continue your work, what words of advice do you have?
Reacted by Max MiltonReacted by Trent@fceruti I don't think that having an external process that manages replication is impossible but it makes things a lot harder. One of the benefits to the LiteFS approach is that it can detect every write so it's able to maintain a rolling checksum of the database to ensure consistency. When things go wrong in replication, the checksum is essential for determining when and where things go awry.
There's a lot of edge cases you need to worry about when the replication is an external process too. That external process may stop for some reason and then you'll need to have a way to detect changes since the last time it was up. You can do that quickly for small databases but it gets a lot slower for large ones.
We do have plans to add a VFS option for LiteFS. It's not quite an external process but it's useful for people that want to avoid FUSE.
Reacted by Max MiltonReacted by Francisco Ceruti
Currently, litestream replicas a database to cold storage (
fileors3). It would be nice to provide a way to replicate streams of WAL frames to other live nodes to serve as read replicas.This a network replication endpoint (probably
http) as well as figuring out how to apply WAL writes to a live database.