DSS Raft-Based Application-Level Consensus - August 2026 update

[barroco]

Raft-Based Application-Level Consensus passes the full DSS test suite (uss_qualifier and prober included) reproducing SQL-based behaviour end to end and outperforming the distributed SQL implementation in every performance scenario, meeting throughput, latency and error-rate targets.

Introduction

This update reports on the current status of the effort to move to application-level consensus for the DSS architecture.

Background

Following Cockroach Labs' late-2024 license change, the Technical Steering Committee (TSC) investigated mitigation strategies.

At the time, the options evaluated were:

• accepting CockroachDB's commercial terms.
• migrating to a different database (rqlite, YugabyteDB, Vitess, TiDB).
• adopting a distributed key-value store.
• moving consensus into the DSS application itself.

On the application-level consensus track, CAS Paxos was rejected due to a dormant ecosystem, no reusable library and implementation-sensitive correctness. This left Raft via the etcd/raft library the only realistic algorithm. A full redesign document (InterUSS DSS 2024 redesign) was attempted but did not find common ground, notably on the storage backing the consensus engine. The proof-of-concept attempts did not succeed either, in part because the etcd/raft library was not provided as a standalone project at the time.

As a result, the TSC prioritized the switch to YugabyteDB (delivered from v0.19). However, it was agreed to reconsider this decision if a contributor could provide a working DSS prototype using etcd/raft library.

Goals

The goals remain unchanged from the investigation - the Must-Haves remain the review criteria for any consensus design, especially:

• Respond to a USS only after the transaction is recorded in the DAR (DSS Airspace Representation) as required by F3548-21 - DSS0215 and F3411-22a - DSS0070.
• Maintainable by available InterUSS contributors.
• Room to grow to roughly country-scale traffic.
• A pool of three or more instances survives the failure of one without substantial latency impact.
• operators control pool membership.
• No cloud-provider lock-in.
• Every DSS instance is an equal peer (temporary Raft leadership is acceptable).
• Explainable to semi-technical regulators.
• USSs can meet their own performance requirements.

Desirable in addition: no restricted-license dependencies, minority-failure tolerance, low deployment and maintenance effort, resistance to accidental DAR corruption, and the ability for a participant to improve its own performance unilaterally.

Load testing scenarios based on production experience have added concrete targets since 2024 for country-scale traffic: SCD same-cell throughput is expected at ~2 QPS (#1541), multi-cells around ~45 QPS, with limited latency (< 5s) and error rates (< 1%), in a 9 node environment with various latencies (see this comment). Since then, the assumption on the number of nodes has been updated to meet users feedback (>24).

What did the thesis prototypes unlock?

Mariem Baccari's EPFL master thesis (Distributed transactions ensuring deconfliction of long range drone flights, Feb 2026, in the investigation folder) met the TSC's revisit condition on a working toy DSS:

• A working single-Raft DSS prototype on the intended etcd raft library, with its WAL (Write-Ahead Log) and snapshotting: writes applied only after a majority accepts, ~300-400 QPS sustained on three nodes, leader crash absorbed with a ~20% throughput dip and ~25-second recovery, and stable, linear latency under injected network delay.
• A working multi-Raft prototype answering the scalability question that the 2024 conversation left open: S2-cell-bounded geographic shards, each its own Raft group, with cross-shard transactions via two-phase commit through a transaction-coordination Raft group. Three isolated shards delivered the expected three time single-Raft throughput. However, the overhead from the two-phase commit and other multi-Raft induced coordination efforts affect performance scalability during cross-shard transactions based workloads. Therefore, the multi-Raft extension still requires more effort into the design in order to achieve the performance goals.
• Resolution of the old blocker. The library packaging problem that sank the 2024 toy DSS is gone: the implementation now underway imports etcd/raft and reuses etcd's rafthttp transport, WAL, and snapshotter directly, so InterUSS only implements a simple wrapper (~1,100 lines at the current snapshot).

The design relies on etcd/raft as the underlying consensus engine. The library implements the Raft consensus algorithm and is actively maintained and used by systems such as CockroachDB and etcd itself.

Execution Plan

The thesis findings are moving to the repository: the incremental raftstore implementation is landing on master as a distinct datastore. The design decisions are captured in the package’s README and discussed in issues / pull requests. The architecture and progress is tracked in the issue tracker (#1463) and the measurements in #1594.

As part of the move and build up on the existing code base, an approach to the storage question raised in the redesign document is proposed and implemented (#1526): in-memory per-service state, made durable via the Raft log and snapshots.

The work is organised into four stages, each gated on a measured result. At the time of writing this post, gates 0 (functional) and 1 (performance) passed test criteria and code is under review.

• Gate 0: Functional practicability: does application-level Raft comply with the DSS test suites?
• Gate 1: Performance fitness: does it meet country-scale traffic requirements?
• Gate 2: Production readiness: can users run it in production ?

Gate 0: Proof of Functional Practicability

Goal: the Raft implementation passes the full DSS test suite, including uss_qualifier and prober.
Status: Tests passed on code now under review

Gate 0 reproduces the behaviour of the SQL implementation, including its legacy cases, on top of application-level Raft. It passes the full DSS test suite.

This stage addressed multiple aspects:

• Consensus layer: wraps etcd/raft's consensus implementation, including cluster configuration with the various base options. The DSS consensus layer only makes high-level calls: proposing values, stepping the clock, and consuming Raft updates. The architecture overview is documented in #1463.
• Transport: etcd/raft is transport agnostic. Our implementation relies on rafthttp, etcd's own peer-to-peer transport implementation which is already optimized for raft. The alternative was to implement a grpc transport from scratch. With the thesis prototype naive implementation, the overall performance was similar so the more practical choice of rafthttp was preferred for the moment and can be revisited later if needed (#1464).
• In-memory data store as projected view: etcd/raft leaves storage to the application. The options for holding state in memory were evaluated in #1526; following that evaluation, in-memory data stores were implemented for every DSS component (RID, SCD, AUX).
• Consistency guarantees: Both reads and writes go through the same path as a Raft request (#1474) and are replicated via a consensus round, ensuring linearizability. In the future, using ReadIndex for read-only requests will maintain the same guarantees without the need for a consensus round.
• Store update: #1525 applies committed proposals to the in-memory store, completing the write path: a write is confirmed to the USS only once the proposal has been committed and applied locally, as DSS0215 requires.

The existing test baseline passes against the initial implementation, as shown by a77b08d in #1538 (uss_qualifier and prober results).
Code is currently reviewed and progress is tracked in #1463.

Gate 1: Performance Fitness

Goal: the Raft implementation meets the throughput, latency and error-rate targets recorded in #1541.
Status: Tests passed on code now under review

Results are recorded in #1594. The Raft implementation outperforms the distributed SQL implementation in every test scenario: SCD same-cell throughput is at ~95 QPS (45x requirement), multi-cells around ~80QPS (2x requirement), with limited latency (<300ms below <5s requirement) and error rates (0% below 1% requirement), in a 9-node environment with various latencies.

Payload size was identified as an immediate limiting factor: when simulating network latency, bandwidth becomes the main constraint. #1587 addresses the low-hanging fruits with:

• better encoding of proposals
• compression of proposals
• OVN format optimisations
• removal of unused keys

Analysis of the remaining bottlenecks continues and is expected to yield further improvements.

Gate 2: Production Readiness

Goal: a DSS user can deploy, upgrade, observe and change the membership of a Raft-backed pool in production.
Status: Work in progress

This stage will cover what a pool needs over its lifetime and optimizations required for a large number of DSS instances (>24), especially:

• Fault injection: partitions, clock skew and node restart during apply need to be tested.
• Membership changes: cluster membership is fixed at Gate 1. Adding and removing nodes needs tooling, and membership changes must remain safe while the pool is serving traffic.
• Updates: the SQL migration system relies on a shared datastore and does not carry over. Its replacement must handle snapshots already on disk, proposals in multiple version formats, and enabling new behaviour coherently across a mixed-version pool.
• Non-voting nodes: as the number of participants grows, not all have to be voters. Voting membership must be decoupled from pool membership, and the voting set balanced, automatically or by configuration.
• Pool authentication: TLS is the core of authentication of a DSS pool today, and tooling, procedures and related code need to be adjusted for the raft pooling before being production-ready.
• Monitoring/Diagnostics: #1545 adds a first set of metrics to the OpenTelemetry interface. The set will be extended to support operational diagnostics of the pool.
• User documentation: deployment, membership changes, upgrades, monitoring and TLS all need documented procedures.

Future-Level Scale

If future demand requires it, the single-leader bottleneck may be removed and the single raft group may be split into multiple independent Raft groups. The thesis prototype partitions the airspace into static S2-cell-bounded shards, one Raft group each, with cross-shard transactions coordinated by two-phase commit through a dedicated group, the protocol CockroachDB and Google Spanner use for the same purpose.

What will change for users?

The Raft datastore will ship as a selectable datastore alongside the SQL ones. Current deployments are unaffected. When enabled, the DSS node becomes a single container: consensus moves inside the DSS process, and no external datastore needs to be deployed.

Next steps

Following the successful demonstration of functional practicability and performance fitness, design and implementation efforts will now move toward production readiness (Gate 2). Future work may focus on scaling to sustain very-high-usage traffic, if required.