Repository navigation
Replies: 5 comments 3 replies
|
@burggraf I'm happy to discuss this interesting topic. For context: this was about a manual litestream setup for r/o "edge" replicas. We could certainly consider adding a --read-only flag. There are other related discussions: A few incoherent thoughts:
|
|
I'm doing some testing regarding running TB in "replica" mode, but so far, it doesn't look like there needs to be any changes for HA when using Litestream and Litestream continuous restore to keep replicas up-to-date. Ideally there's be a "read-only" flag for TB, but I'm not yet sure that it's even necessary. |
|
I've researched auto-failover both from the standpoint of a new, stand-alone application as well as using a hiqlite integration. They're both very complicated. At this point I think it makes more sense to do a simple manual failover system using litestream to maintain hot standby replicas. Even that gets a bit complicated, and while the initial release would be failover only, it would be really cool to offer read replica support since litestream is doing all the hard work to keep the databases in sync already. However, that WILL require a read-only mode (and a few other things.) But these things are considerably less complicated than a true raft-consesnsus auto failover system. Here's what's involved, more-or-less: Feature request: read-only TrailBase mode for externally replicated SQLite databasesWould you consider adding a supported read-only server mode to TrailBase, allowing it to serve existing Record API reads from SQLite databases maintained by an external replication process? Our immediate use case is Litestream continuous restore. We are not asking TrailBase to implement replication, leader election, failover, or write forwarding—only a safe read-only application-server mode. Use caseWe are building a small, operator-managed deployment with:
Initially, all application requests will be forwarded to the primary, with TrailBase stopped on replica nodes. If TrailBase gains a compatible read-only mode, we would enable a narrow allowlist of read requests against each local replica: Writes, authentication endpoints, administration, unknown/custom routes, and realtime subscriptions would continue going to the primary. Reads requiring read-after-write consistency would also use the primary. Replica reads would be explicitly eventually consistent, not linearizable. The controller would not be in the ordinary request path. Its downtime would prevent administrative transitions, not interrupt forwarding to the existing primary. Why proxy-level filtering is insufficientLitestream's restore documentation states that a database maintained with Our source inspection of TrailBase v0.33.11 found behavior that appears incompatible with this requirement:
For reference, the inspected paths include:
These are version-specific source findings, not a claim about every release. Please correct us if an existing supported mode already addresses this. Requested behavior1. Open replicated databases genuinely read-onlyApply read-only behavior to In this mode, TrailBase should:
The intended contract is no TrailBase-originated mutation of replicated database state. Normal SQLite read locking and necessary coordination files are a separate matter; we are not requesting that every filesystem operation be prohibited. 2. Suppress mutation independently of the proxyThe server should reject unsupported mutating routes itself, even if a request bypasses or is misclassified by the proxy. HTTP method alone is insufficient: for example, the inspected version includes a mutating GET logout endpoint. Mutating jobs and background work must also be disabled, including session cleanup, file-deletion processing, custom/WASM jobs, and runtime preference writes. Disabling custom routes and jobs entirely would be acceptable for an initial implementation. Replica startup and reads must not cause uploads, deletions, email delivery, or other application-side external mutations. 3. Preserve ordinary read API behaviorThe useful initial scope is existing Record API reads, with normal query handling, response formats, and access rules. If supported, authenticated reads should validate existing credentials using provisioned signing material without creating or refreshing sessions. Unsupported authentication behavior should fail explicitly rather than silently write local state or weaken access checks. We recognize that permissions and account state on an asynchronous replica can be stale. Our deployment would restrict eligible reads accordingly; this request does not assume immediate revocation consistency. Replica nodes would receive compatible configuration, keys, migrations and runtime assets through deployment tooling. Missing state should produce an error rather than bootstrap a new identity or database. 4. Keep optional writable state separateNode-local request logs are fine provided they do not mutate the replicated databases. An option to disable database-backed request logging would also be sufficient. Read-only mode should not persist administrative configuration changes or generate replacement secrets as a side effect of starting the replica. 5. Support a database that changes externallyThese are live, changing replicas, not immutable snapshots. The mode should use SQLite access compatible with the restore process's locking and update mechanism; We would like to qualify:
Automatic schema-cache refresh is not required for the first version. A documented procedure that takes the replica out of service and restarts it after a controlled schema rollout would be acceptable, provided restart itself remains read-only. We can help test this integration against pinned TrailBase/Litestream versions. We do not expect a read-only flag alone to guarantee compatibility with every external replication tool. Explicit non-goalsWe are not requesting:
For promotion, our tooling would fence the previous primary, stop the replica server and restore process, validate recovered state, then restart TrailBase in normal writable mode. Replication lag, routing eligibility, fencing and failover remain our responsibility. Suggested initial acceptance checksA minimal useful implementation would demonstrate that:
Would this fit TrailBase's direction? We would appreciate guidance on the smallest maintainable upstream implementation, any existing work we should follow, and whether you would welcome a focused PR or reproducible integration test. A CLI flag is one possible interface, but we have no preference about its name or implementation. |
|
I'm very open to adding a ready-only CLI flag. Note that using a CLI flag may require a restart when failing over to a replica.
Unquestionably there's an inherent heightened complexity to any distributed system. I was wondering if you could elaborate a bit more. I understand that a manual failover system is architecturally simpler than raft consensus, however I'm wondering if it may result in a system that is more complex to host and maintain. I'm ok with TB absorbing complexity if it ends up in something easier for users, something more robust or more versatile. What is your concern with hiqlite? |
|
This is awesome -- thank you.
This isn't true HA, of course, as you can lose writes in between the failure of the primary and the switch-over to the replica. The --read-only switch here just allows us to use the hot-standby replicas as read-only replicas to increase read speed, spread the load across multiple servers, and allow for region-aware applications to do reads from the closest server and thus decrease latency. To get true HA, we'd need two separate write transactions possible:
Why not make all writes "critical" writes? Because critical writes have a significant performance penalty -- there's always a tradeoff between durability and speed, and it can be quite significant for these writes. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
[DELETED]
All reactions