Benchmark Reproduction EN

English · 简体中文

Reproducing benchmarks

The 1.4.0 public runner has seven columns, not the old five-column matrix. It requires prebuilt default/policy Rust drivers, official C KCP and quic-go drivers. Full runs total 623 measurements: 168 single throughput, 252 concurrent throughput, 168 loaded latency and 35 sparse latency.

1. Isolated environment

Use Linux with at least eight available CPUs, Python 3, C/Rust/Go compilers and ip, tc, taskset, nice, lscpu. Prepare missing tools only in a disposable container/VM/task sandbox. No host toolchain/configuration/network changes.

TASK_ROOT/
  work/                   source revision 1199ac7 (or document your revision)
  policy/                 separate copy with per-socket SendOptions setters
  home/ cargo/ rustup/     task-specific configuration/cache/toolchain
  target/ target-policy/  default/configured release builds
  go/                     Go toolchain, including bin/go
  gomod/ gocache/ gopath/  private Go caches
  kcp/                    upstream ikcp.c, ikcp.h, LICENSE, revision.txt
  bin/                    measure-process, kcp-single, kcp-concurrent,
                          quic-single, quic-concurrent
  logs/ results/ tmp/

export HOME="$TASK_ROOT/home"
export CARGO_HOME="$TASK_ROOT/cargo"
export RUSTUP_HOME="$TASK_ROOT/rustup"
export CARGO_TARGET_DIR="$TASK_ROOT/target"
export GOMODCACHE="$TASK_ROOT/gomod"
export GOCACHE="$TASK_ROOT/gocache"
export GOPATH="$TASK_ROOT/gopath"
export GOTOOLCHAIN=local
export TMPDIR="$TASK_ROOT/tmp"

Create these directories and export TASK_ROOT explicitly. Ensure toolchain binaries and OS utilities are available through the task process PATH; do not edit shell startup files. Pin dependencies and download them before timed runs. Rust 1.98.1, Tokio 1.53.2, Go 1.27.2 and quic-go v0.63.0 were used on October 12; a fresh lockfile may resolve different releases, so archive the measured lockfiles and binary hashes.

2. Default build and C/Go peers

Put unmodified upstream C KCP source and its license at TASK_ROOT/kcp. Pin commit b1a7a2101dcbb96017681a500d6b82bbe5a88766; write that full SHA in revision.txt. The Go module pins quic-go v0.63.0; preserve go.mod/go.sum.

# From TASK_ROOT/work, in the task environment:
cargo build --release --manifest-path examples/test_benchmark/Cargo.toml
cargo build --release --example concurrency_benchmark --example sharded_echo
cc -O2 -std=c11 examples/test_benchmark/measure_process.c -o "$TASK_ROOT/bin/measure-process"
cc -O3 -DNDEBUG -std=c11 -I "$TASK_ROOT/kcp"   examples/test_benchmark/kcp/benchmark.c "$TASK_ROOT/kcp/ikcp.c"   -o "$TASK_ROOT/bin/kcp-single"
cc -O3 -DNDEBUG -std=c11 -pthread -I "$TASK_ROOT/kcp"   examples/test_benchmark/kcp/concurrency.c "$TASK_ROOT/kcp/ikcp.c"   -o "$TASK_ROOT/bin/kcp-concurrent"
# From TASK_ROOT/work/examples/test_benchmark/quic:
"$TASK_ROOT/go/bin/go" build -o "$TASK_ROOT/bin/quic-single" ./single
"$TASK_ROOT/go/bin/go" build -o "$TASK_ROOT/bin/quic-concurrent" ./concurrent

QUIC test certificates and disabled certificate verification are benchmark fixtures, not a production security configuration.

3. Configured policy build

Copy the same source into TASK_ROOT/policy. Add socket.set_send_options(rust_raknet::SendOptions::default()).await? after each RakNet client connects and each server accepts in examples/test_benchmark/src/main.rs, examples/concurrency_benchmark.rs and examples/sharded_echo.rs. Preserve TCP branches and identical library code, lockfiles and all other workload settings. Do not apply the setter only to the client.

CARGO_TARGET_DIR="$TASK_ROOT/target-policy" cargo build --release   --manifest-path "$TASK_ROOT/policy/Cargo.toml"   --features send-policy --example concurrency_benchmark --example sharded_echo
CARGO_TARGET_DIR="$TASK_ROOT/target-policy" cargo build --release   --manifest-path "$TASK_ROOT/policy/examples/test_benchmark/Cargo.toml"   --features rust-raknet/send-policy

Merely enabling the feature does not reproduce configured policy columns. The runner already selects target-policy for these variants; do not modify its variant list to add them again.

4. Private network and complete matrix

Enter a private user/network/PID sandbox with one-user UID mapping, namespace UID 0 and namespace-local CAP_NET_ADMIN. Hide host home, make the host view read-only and allow writes only in this task. Both endpoints share this network. The runner asserts namespace separation, working directory and task HOME/CARGO_HOME; it does not fall back to host network shaping.

# Capture this in the parent before entering the private network namespace:
export TASK_HOST_NETNS="$(readlink /proc/self/ns/net)"
# Inside the namespace, after environment/directory checks:
ip link set lo up
cd "$TASK_ROOT/work"
SMOKE=1 python3 examples/test_benchmark/run_comparison.py single > "$TASK_ROOT/results/smoke.jsonl"
python3 examples/test_benchmark/run_comparison.py single > "$TASK_ROOT/results/single.jsonl"
python3 examples/test_benchmark/run_comparison.py concurrent > "$TASK_ROOT/results/concurrent.jsonl"
LOADED_RTT=1 python3 examples/test_benchmark/run_comparison.py concurrent > "$TASK_ROOT/results/loaded.jsonl"
python3 examples/test_benchmark/run_comparison.py latency > "$TASK_ROOT/results/latency.jsonl"

Smoke measurements are not formal results. Each case creates fresh connections, rotates column order and checks echoes/sample counts. Keep error records; failure stops the suite. The runner pins sparse latency to separate cores and concurrent/loaded endpoints to separate four-core groups. It configures only its private loopback qdisc, MTU and packet aggregation.

5. Interrupted execution and aggregation

If execution stops between completed cases, keep the partial logs and interruption record. In a fresh private namespace with the same mode, binaries and settings, set BENCH_RESUME_FILE to the existing validated JSONL and append new output to it. Completed exact cases are skipped; new metadata is recorded. Input containing a failed measurement is refused. Diagnose failed cases instead of discarding them or repeating until the median improves.

measure-process uses Linux wait4 for whole-process client/server peak RSS, including warmup and sample storage but excluding kernel socket buffers. Group rows by profile, variant and loaded flag; report medians/min–max with units. RTT percentiles are calculated per run, then aggregated. Preserve JSONL, logs, source/binary hashes, lockfiles, qdisc and UDP counters outside source commits.

The separate async/blocking guide uses run_api_comparison.py and different profiles; do not mix its rows into the cross-protocol table.

Runner source · Driver instructions · Methodology