A faithful, production-quality reproduction of 1Pipe: Scalable Total Order Communication in Data Center Networks (SIGCOMM '21, ACM DOI 10.1145/3452296.3472909).
This repository is an independent, clean-room implementation. (The published paper is © ACM and is not redistributed here — see the ACM Digital Library.) It provides:
- a Rust workspace whose state machines are the single source of truth for every component (host library, software switch, RDMA host, simulator);
- two deployment targets — pure Linux + UDP (commodity, runs on a laptop) and the original RDMA + Tofino testbed (P4 source);
- a discrete-event simulator and a figure harness reproducing the
sim-reproducible paper figures (hardware-only ones are modelled, clearly
labelled — see
eval/REPORT.md); - the four paper applications: transactional KVS, TPC-C, replicated remote hash table, and a Ceph 1-RTT replication patch;
- a unified
1pipecommand-line tool, and an animated web visualizer that shows total-order barriers spreading as timestamp "ripples" (viz/).
Every 1Pipe mechanism (§4–§7) is implemented and tested, and the central
property — that all receivers observe one global total order — is verified
directly (docs/CLAIMS.md maps each mechanism and claim to its test). Reliable
1Pipe — acks, retransmission, and controller-coordinated failure recovery
(detect a crashed peer from silence, excise it, recall it from outstanding
scatterings, and converge on an agreed recovery cut so survivor messages in
flight are never dropped) — runs not only in the simulator but as a deployable
stack over real UDP sockets (crates/1pipe-net), as independent processes
(1pipe node, one per machine) with no coordinator. Runtime clock
synchronisation (§4.2) runs there too: an NTP-style four-timestamp probe
measures each peer's offset with the path delay cancelled and raises the host's
logical clock to the network-leading one, correcting residual NTP/PTP skew —
robust to jitter (no runaway), proven to converge over real UDP (injected skew up
to 600 µs corrected to well under half the raw spread). It is exercised by an
in-process fault-injection test, a
clock-sync convergence test, and a multi-process test that spawns the real
1pipe node binary and SIGKILLs one process mid-run, asserting the survivors
recover and stay in total order. Run it yourself: 1pipe net --kill (one-process
demo) or launch 1pipe node on each host.
The reusable orchestration — ReliableEndpoint, FailureDetector, and
ClockSync — is canonical in 1pipe-core and run verbatim by the deployable UDP
stack; the lower-level primitives (barrier, beacon, minimax, reliable 2PC commit)
are additionally shared by the simulator and mirrored by the Tofino data plane,
while the simulator implements the reliable spec in event-driven form on those
same primitives (see docs/CLAIMS.md for the exact split).
125 tests pass.
The honest scope line: results are simulated, not measured on the paper's
RDMA/Tofino testbed, and a few testbed-only figures are explicit models — see
docs/CLAIMS.md and eval/REPORT.md.
| Crate | What it provides |
|---|---|
crates/1pipe-core |
Timestamp, BarrierAggregator, BeaconScheduler, MinimaxSync, the §5 reliable 2PC (ScatteringCommit, CommitWatermark), and the sans-I/O ReliableEndpoint (§5) + FailureDetector + ClockSync (§4.2/§4.3) that the deployable stack runs on. Pure algorithms, zero I/O. |
crates/1pipe-wire |
24-byte on-the-wire packet header + codec (§5.1). |
crates/1pipe-sim |
Discrete-event simulator: fat-tree topology, hosts/switches running the core state machines, best-effort and reliable delivery, metrics. |
crates/1pipe-eval |
Figure-by-figure reproduction harness emitting CSV (see Evaluation). |
crates/1pipe-apps |
The four §7 applications: transactional KVS, TPC-C, replicated remote hash table, Ceph 1-RTT replication. |
crates/1pipe-net |
Commodity Linux + UDP deployment: real UdpSocket endpoints, a best-effort ordered receiver, and ReliableHost — the full reliable stack (acks, retransmission, beacon-driven barriers, runtime NTP-style clock sync, and crash recovery with an agreed cut). Runs on a laptop. |
crates/1pipe-cli |
The unified 1pipe CLI (sim / eval / trace / sweep / net / node). |
viz/ |
Web visualizer: animated timestamp ripples (no build step). |
p4/onepipe.p4 |
Tofino (TNA) data-plane barrier aggregation. Requires the Intel Tofino SDE; see p4/README.md. |
The paper's measurements come from a 32-server / 10-switch RDMA + Tofino
testbed, which is not available here. Every figure in eval/ is produced by
the simulator, which runs the real core algorithms over a modelled
network. It reproduces the paper's mechanisms and trends; absolute numbers
track the testbed only to the fidelity of the model constants. See
eval/README.md. The Tofino P4 is provided as source but
cannot be built or run without the licensed SDE.
# Install Rust (one-time):
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
. "$HOME/.cargo/env"
# Build everything:
cargo build --release --workspace
# Run the test suite (includes proptest oracles for the §4 invariants and
# appendix correctness proofs):
cargo test --workspace
# Lint + format checks (CI gates):
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warningsOne binary drives everything (cargo run -p onepipe-cli --release -- <cmd>, or
build once and call ./target/release/1pipe):
1pipe sim --hosts 32 --msgs 200 [--reliable] [--loss P] [--skew-ns N] # run one config
1pipe eval all --out eval/figures # regenerate figures
1pipe eval --list # list figure names
1pipe trace --out viz/traces --hosts 8 # emit visualizer traces
1pipe sweep --hosts 8 --overhead-budget 0.005 # tune the beacon interval
1pipe net --hosts 4 --msgs 20 [--kill 3 --kill-after-ms 4] # REAL reliable cluster over UDP
1pipe node --id 0 --bind 10.0.0.1:9000 --peer 1=10.0.0.2:9000 ... # ONE real host (one per machine)1pipe sweep is the parameter trainer: it sweeps the beacon interval and
recommends the value that minimises reordering delay while keeping beacon
bandwidth overhead under the budget (the paper's §6.2 trade-off).
1pipe net is the only subcommand that runs the real stack rather than the
simulator: it stands up N reliable hosts on loopback over real UDP sockets,
broadcasts among them, and prints per-host deliveries plus whether total order
held. Pass --kill ID to crash a host mid-run and watch the survivors detect
the silence, recover, and stay in total order. Every subcommand has --help.
$ 1pipe net --hosts 4 --msgs 40 --kill 3 --kill-after-ms 4
host 0: delivered 120 [h1:40 h2:40 h3:40]
host 1: delivered 120 [h0:40 h2:40 h3:40]
host 2: delivered 120 [h0:40 h1:40 h3:40]
host 3 (crashed): delivered 1 [h0:1]
[net] total order across receivers: OK
[net] survivors recovered: OK
1pipe node goes further — it runs one host as this process, so you deploy
it for real by launching one per machine (they need only NTP/PTP-synced clocks,
no coordinator):
# on host A (10.0.0.1):
1pipe node --id 0 --bind 10.0.0.1:9000 --peer 1=10.0.0.2:9000 --peer 2=10.0.0.3:9000 --msgs 100
# on host B (10.0.0.2):
1pipe node --id 1 --bind 10.0.0.2:9000 --peer 0=10.0.0.1:9000 --peer 2=10.0.0.3:9000 --msgs 100
# on host C (10.0.0.3): ... --id 2 ...Kill any node (Ctrl-C, kill -9, or pull its cable) and the survivors detect
the silence, recover, and keep delivering in total order. The
processes_recover_and_keep_total_order_when_one_is_killed test does exactly
this — spawns the real binary four times and SIGKILLs one.
Regenerate all figures (deterministic CSV under eval/figures/, diffable in
CI; one gnuplot script each in eval/plot/):
1pipe eval all --out eval/figuresFourteen figures span throughput/latency scaling, reordering-vs-depth (the
central claim), beacon overhead, loss sensitivity, failure recovery, minimax
convergence, the deployable clock-sync convergence and recovery cut, and
the application speedup. Figures whose paper counterparts were
hardware measurements are produced as explicitly labelled models. The
figure→paper-section→caveat map and the simulated-not-testbed discussion are in
eval/REPORT.md.
An animated, in-browser view of how total-order barriers propagate through the
fabric like waves: nodes are coloured by their current barrier timestamp,
packets are dots coloured by the timestamp they carry, and switches forwarding
min(inputs) visibly merge the waves. Scenarios for normal operation, packet
drops, link failure, and host failure (reliable recovery: a crashed host is
excised and survivors recover up to the agreed cut).
1pipe trace --out viz/traces --hosts 8 # (pre-generated traces are committed)
python3 -m http.server --directory viz 8080 # open http://localhost:8080See viz/README.md and docs/visualizer.md.
docs/architecture.md— crates, data flow, the single-source-of-truth design.docs/barrier-mechanism.md— how the split-horizon, valley-free barrier works and the deadlock it avoids.docs/reliable-and-recovery.md— the deployable stack: reliable delivery, runtime clock sync (the NTP estimator and why a naive one ratchets), and failure recovery (the agreed cut, gossip agreement, rejoin) — with the honest limits.docs/visualizer.md— the trace format and the animation.docs/CLAIMS.md— the honest audit: every paper mechanism and claim mapped to its code and test, marked validated / modeled / not-implemented.eval/REPORT.md— every figure, its paper section, and its caveat.
crates/
1pipe-core/ Algorithmic core: §4 barrier, §4.1.2 beacons,
§4.2 minimax, §5 reliable 2PC, §4.3 excision.
1pipe-wire/ 24-byte packet format from §5.1.
1pipe-sim/ Discrete-event simulator + scenarios + tracer.
1pipe-eval/ Figure reproduction harness + parameter sweep.
1pipe-apps/ The four §7 applications.
1pipe-net/ Commodity Linux + UDP deployment.
1pipe-cli/ The unified `1pipe` command-line tool.
docs/ Architecture, mechanism, and visualizer guides.
eval/ Generated figure CSVs, gnuplot scripts, REPORT.md.
viz/ Web visualizer (HTML/CSS/JS) + committed traces.
p4/ Tofino (TNA) data plane (needs the Intel SDE).
Dual-licensed under MIT or Apache-2.0, your choice. See LICENSE-MIT and
LICENSE-APACHE.