#15 - Add C++ bench infrastructure for DGX Spark performance report

[RamyaGuru]

Summary

Lays the C++ groundwork for the DGX Spark performance report (#15). Code and doc-skeleton only; per-cell numbers fill in via a follow-on commit on this same PR before it's marked ready-for-review.

• Adds TokenBucketPacer (--target-gbps software pacer) and seconds= field on bench output across DPDK / RoCE / Socket benches. Zero src/ changes.
• Adds examples/bench_capture_environment.sh (slow-moving system state snapshot) and examples/run_spark_bench.sh (sweep wrapper that runs the bench under mpstat + nvidia-smi dmon and emits one CSV row per
  cell).
• Adds docs/performance-dgx-spark.md skeleton with full final structure: native-shape and matched 8 KB headline tables (so cross-backend Gbps comparisons are honest), per-backend sweep dimensions (DPDK packets
  / RoCE messages / TCP stream), and explicit "HDS deferred on GB10" note.
• Wires the doc into mkdocs.yml, docs/index.html News card, README.md, and AGENTS.md drift-hotspot list.

Drop accounting routes through the wrapper (no manager changes): DPDK via DAQIRI_LOG_INFO parsing, RDMA via "CQ error" lines, Socket UDP via /proc/net/udp diff, Socket TCP via nstat.

Test plan

• DPDK smoke (daqiri_bench_raw_gpudirect): TX/RX complete: lines have the new seconds= format; ~84 Gbps cross-chip on Spark, 0 drops.
• Pacer accuracy: --target-gbps 1 holds 1.055 Gbps (within the +/-5% software pacer accuracy documented in the report).
• Socket UDP smoke (daqiri_bench_socket): new sent_packets/recv_packets/sent_bytes/recv_bytes/seconds= format; iteration cap behavior preserved.
• bench_capture_environment.sh captures kernel cmdline, CPU, NUMA, hugepages, mlx5 PCIe, OFED, NVIDIA GPU state, isolcpus, governor, IRQ.
• run_spark_bench.sh dpdk smoke produces a valid CSV row with gbps~=96, drops=0, drops_kind tagged correctly.
• scripts/check_doc_refs.py clean (every YAML covered by decision tree; every bench cross-referenced).
• scripts/check_html_links.py clean after mkdocs build --strict.
• RDMA loopback smoke is deferred — kernel routes local-to-local IPs through lo, requires a network namespace for single-host testing. Lands with the data-fill commit on this PR.
• Data-fill commit (this PR, second commit): run run_spark_bench.sh {dpdk,rdma,socket-udp,socket-tcp} {sweep,drop-curve} and populate the C++ loopback cells in docs/performance-dgx-spark.md.

Follow-up PRs

• [FEA] Benchmark DAQIRI on DGX Spark (C++) #15 workloads: introduces examples/bench_post_process.{h,cu} (cuFFT + cuBLAS post-process layer, examples-only — no library dependency change), refactors rx_count_worker to defer burst free until the
  post-process CUDA event signals, and fills FFT/GEMM cells.
• [FEA] Benchmark DAQIRI on DGX Spark (Python) #16 Python loopback: daqiri_bench_rdma.py + daqiri_bench_socket.py mirroring this PR's CLI/output. Uses the existing pybind11 surface — no new bindings.
• [FEA] Benchmark DAQIRI on DGX Spark (Python) #16 Python workloads: pybind11 wrappers for the [FEA] Benchmark DAQIRI on DGX Spark (C++) #15-workloads post-process layer; completes Spark v1.

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

Adds the C++ benchmarking infrastructure for the DGX Spark performance report: a TokenBucketPacer class shared across all three bench executables (DPDK, RoCE, Socket), seconds= timing on all completion lines, and two new shell scripts (run_spark_bench.sh sweep wrapper, bench_capture_environment.sh environment snapshot). The doc skeleton for docs/performance-dgx-spark.md is wired into mkdocs.yml, the index page, and README.md; DPDK GPUDirect data is fully populated.

• TokenBucketPacer and seconds= output are added to raw_bench_common.{h,cpp} and consumed uniformly by all three bench binaries; raw_bench_common.cpp is linked into the rdma and socket targets via CMakeLists.txt.
• run_spark_bench.sh drives the sweep and drop-curve matrices, captures per-run CPU/GPU counters, and emits one CSV row per cell; two parsing bugs affect socket backends (hex mis-decode of a decimal /proc/net/udp column and RDMA field names used in the socket fallback).
• Socket and RoCE results are explicitly deferred with documented follow-up issues; the DPDK GPUDirect section is complete with drop-curve, payload×target, and payload×batch matrices."

Confidence Score: 3/5

Safe to merge for the DPDK path; two shell-script parsing bugs will silently produce all-zero CSV rows for socket backends once those bench bugs are fixed.

The C++ changes are clean and zero src/ changes are made. The DPDK data in the performance doc is fully populated and the sweep/drop-curve infrastructure works correctly for DPDK. The two issues are both in run_spark_bench.sh's output-parsing logic for socket backends: the /proc/net/udp drops column is a decimal integer but is fed through strtonum("0x" …), inflating any non-zero drop count; and the socket bench's completion field names (sent_packets, sent_bytes) don't match the RDMA-named fallback the script uses (send_completions, send_bytes), so every socket CSV row would show 0 Gbps. Neither bug affects the DPDK results already in the doc, but both would corrupt socket data when the deferred socket fixes land on this same PR.

examples/run_spark_bench.sh — the snapshot_proc_net_udp hex-decode and the socket/RDMA field-name mismatch in the third fallback block

Important Files Changed

Filename	Overview
examples/run_spark_bench.sh	New sweep wrapper; two parsing bugs: /proc/net/udp drops column decoded as hex instead of decimal, and socket bench field names (sent_packets/sent_bytes) don't match the RDMA-named fallback (send_completions/send_bytes), causing all-zero socket CSV rows.
examples/raw_bench_common.cpp	Adds TokenBucketPacer implementation and seconds= timing to rx_count_worker. Sliced-sleep approach correctly handles stop-flag interruptibility; ostringstream trick for atomic stdout write is sound.
examples/raw_bench_common.h	Adds TokenBucketPacer class declaration with correct in-class initializers; default constructor is safe because wait_for_bytes() short-circuits when target_bps_==0.0.
examples/raw_gpudirect_bench.cpp	Wires TokenBucketPacer into tx_worker, adds per-worker timing, and emits structured TX complete line; changes are minimal and correct.
examples/rdma_bench.cpp	Adds pacer, byte counters, and structured completion lines. Uses wall-clock secs from main() (post-join) consistently for both server and client.
examples/socket_bench.cpp	Adds pacer, byte tracking, and structured Client/Server completion lines. Field names (sent_packets/sent_bytes) are inconsistent with the run_spark_bench.sh parsing fallback, but the C++ output itself is correct.
examples/bench_capture_environment.sh	New environment snapshot script; run_section handles missing commands gracefully; cat_section uses deliberate unquoted glob expansion for multi-path sections.
scripts/spark_data_fill.sh	Driver loop with pre-flight, hugepage cleanup, and PIPESTATUS-aware exit capture. RDMA explicitly rejected; socket backends will inherit the run_spark_bench.sh parsing bugs.
docs/performance-dgx-spark.md	Performance report skeleton with full data for DPDK GPUDirect; deferred sections clearly documented with footnotes and known-limitations section.
examples/CMakeLists.txt	Links raw_bench_common.cpp and CUDA::cudart into rdma and socket bench targets to support the new TokenBucketPacer and chrono usage.

Sequence Diagram

sequenceDiagram
    participant Fill as spark_data_fill.sh
    participant Wrapper as run_spark_bench.sh
    participant Env as bench_capture_environment.sh
    participant Bench as daqiri_bench_*
    participant CSV as runs.csv

    Fill->>Fill: preflight checks (hugepages, binary, carrier)
    Fill->>Wrapper: run_backend_mode(backend, sweep)
    Wrapper->>Env: capture environment.txt once per result set
    loop each (payload x batch x target_gbps) cell
        Wrapper->>Wrapper: generate_yaml(payload, batch)
        Wrapper->>Wrapper: snapshot_cpu_stat before
        Wrapper->>Bench: bench_bin yaml --seconds 30 [--target-gbps G] [--mode both]
        Note over Bench: TokenBucketPacer paces TX worker
        Bench-->>Wrapper: "stdout (complete lines) + stderr (DAQIRI_LOG_*)"
        Wrapper->>Wrapper: snapshot_cpu_stat after
        Wrapper->>Wrapper: extract pkts/bytes/secs from stdout
        Wrapper->>Wrapper: parse drops per backend
        Wrapper->>CSV: append CSV row
    end
    Fill->>Fill: clean_orphan_hugepages between modes

Loading

_{Reviews (1): Last reviewed commit: "#15 - Add payload x target_gbps matrix a..." | Re-trigger Greptile}

[greptile-apps]

The drops column at field 13 in /proc/net/udp is printed by the kernel as a plain decimal integer (%d/%u in udp4_format_sock). Wrapping it in strtonum("0x" …) interprets it as hexadecimal, so any drop count ≥ 10 is inflated — e.g., 10 drops becomes 16, 100 drops becomes 256. The before/after delta subtraction preserves the error, so the socket-udp CSV rows will carry wrong drop counts the moment UDP drops start occurring during the drop-curve sweep.

Suggested change

	snapshot_proc_net_udp() {
	awk 'NR>1 { sum += strtonum("0x" $13) } END { print sum+0 }' /proc/net/udp 2>/dev/null || echo 0
	}
	snapshot_proc_net_udp() {
	awk 'NR>1 { sum += $13+0 } END { print sum+0 }' /proc/net/udp 2>/dev/null || echo 0
	}

[greptile-apps]

Socket field-name mismatch in third fallback

socket_bench.cpp emits sent_packets and sent_bytes, but this block queries the RDMA field names send_completions and send_bytes. Neither will match the socket bench's stdout, so pkts and bytes both fall through to the :-0 default. Every socket-udp and socket-tcp CSV row will report packets=0, bytes=0, pps=0, gbps=0 regardless of how the bench actually performed — this would silently produce a zero-filled data-fill for socket backends once those bugs are resolved.