Benchmark Methodology EN

English · 简体中文

Benchmark methodology

The measured subject is the rust-raknet implementation. The October 12 results contain 623 validated cross-protocol measurements, built from public revision 1199ac7. The dedicated plain-send sparse RTT retest is documented separately on the results page.

Define equivalent delivery semantics

All throughput comparisons use reliable ordered messages: rust-raknet ReliableOrdered, C KCP message mode, one bidirectional stream per QUIC connection and length-prefixed TCP records. The peer verifies the full payload and message order before echoing. Raw unreliable UDP send rates are not comparable to verified reliable delivery.

Implementation	Version or revision	Configuration
rust-raknet	1.0.0 + send-policy changes, local 8c06cf4	Ordinary/batch APIs, default/configured policy separately; Tokio 1.53.1
TCP	Linux 7.0.11-76070011-generic; repository driver 0.1.0	TCP_NODELAY; 4 B little-endian length prefix; whole-record writes
Official C KCP	b1a7a2101dcbb96017681a500d6b82bbe5a88766	Unmodified ikcp.c/h, not Rust KCP
quic-go	v0.63.0; Go 1.27.2	TLS 1.3, congestion control and flow control enabled

C KCP uses ikcp_nodelay(kcp, 1, 10, 2, 1) and immediate flush after writes/input. Windows are 64/64 for single connections and 64/128 for concurrency; no FEC or encryption. QUIC retains security and control mechanisms, so its feature costs differ from unencrypted rust-raknet and C KCP.

Measure four workloads separately

Workload	Question	Method
Single-connection throughput	How much can one connection deliver continuously?	Application window 64; 64 / 800 / 4,096 B; loss/delay profiles
Concurrent throughput	How do many active connections use CPU?	64 / 256 / 1,024 / 2,048 connections; window 16 per connection
Sustained-load delivery latency	How long do messages wait under load?	Separate --loaded-rtt runs; timestamp before sending to verified echo
Sparse-message latency	What is the round-trip cost without sustained load?	Sequential request/echo, separate CPU affinity, sufficient samples

Concurrent throughput measures 1,048,576 messages per run. Each connection completes 20 warmup echoes; a barrier starts the burst, and connections close only after all bursts finish. Sustained-load latency measures 262,144 messages per run and clears warmup samples. It is neither one-way network delay nor server-only processing time.

Metrics and units

• Throughput: verified application payload bytes in one direction / seconds / 1,048,576, in MiB/s. Headers and ACKs are excluded; directions are not added together.
• RTT: from sending a message to fully verifying its echo, including both endpoints, queues and network.
• Median latency: half the messages take no longer than this value. 95% / 99% latency describes the corresponding fraction and helps inspect slower messages.
• Memory: Linux wait4 ru_maxrss, converted from KiB to MiB by dividing by 1,024. C/S mean client/server whole-process peak RSS.

RSS covers startup, warmup, measurement and teardown, including runtime, allocator and application buffers. It excludes kernel socket buffers, is not per-connection memory and does not make the sum of endpoint peaks a simultaneous peak. Much TCP state is in the kernel, so RSS is not total system networking memory.

Timestamping, storing RTT samples and sorting change CPU and memory usage. Run loaded latency separately from plain throughput and report that run's own RSS.

Loss injection

netem probabilities apply to network packets, including data and ACKs, not application messages. Losing a coalesced datagram can affect several messages; a TCP segment may also contain multiple records. Random loss traces differ across runs.

“0% injected loss” does not guarantee zero actual loss: UDP receive buffers can overflow. Record namespace-local UdpRcvbufErrors, other UDP errors and qdisc counters each run. Loss feedback disables coalescing, including loss during warmup.

Inject loss only inside a private network namespace. Never change host interfaces or the host loopback qdisc.

Fairness and repeatability

The latest environment was an i7-9700F with eight logical CPUs; Rust 1.98.1 / GCC 13.3.0 / Go 1.27.2, optimized builds. Loopback MTU was 1,500 B, GSO/GRO limited to one packet, netem queue 100,000, and process niceness 10.

Single-connection Rust/Go used four workers each; C used one event loop. These unpinned runs do not compare limits under equal CPU budgets. Concurrent runs used four workers and four server sockets with disjoint four-CPU sets for server and client. Sparse latency pinned endpoints to separate CPUs; affinity does not reserve exclusive cores.

Single-connection rust-raknet nominal MTU was 1,428 B, aligned with C KCP / QUIC UDP 1,400 B. Concurrent rust-raknet nominal MTU was 1,400 B, QUIC UDP 1,372 B and C KCP UDP 1,400 B. C KCP had a 28 B larger IPv4 packet budget; do not extrapolate to equal fragmentation conditions.

Repeat each throughput/load-latency profile three times and sparse latency five times. Rotate implementation order and create new servers/connections per run. Report medians and min–max ranges. Reported latency percentiles are medians of per-run percentiles, not percentiles over pooled samples.

Drawing conclusions

Compare medians, ranges, slow-message latency and RSS together. Plain send and explicit batching are different application workloads; policy settings are local choices. A faster median in one profile does not establish a universal winner. The current result page records source revisions, toolchains and the separate 73.1 µs retest.

Reliable ordered loopback echoes do not establish mixed-mode fairness, adaptive congestion control, WAN contention, burst-loss/jitter behavior or NAT traversal. The public seven-column runner now includes both policy variants; retain identical library sources and resolved dependencies across default/policy builds.