Benchmarks Methodology

Benchmark Methodology & Statistical Rigor

To prevent experimental bias, thermal throttling artifacts, and WAN noise, benchmarks follow strict scientific protocols.

---

Experimental Protocol

1. Paired Difference Testing ($N=50$): Each trial pairs LangGraph and AgentMesh on identical inputs.
2. Interleaved ABBA Order: Trial execution order alternates (A-B-B-A) to distribute thermal and background OS variance equally.
3. Trace Replay Mode: Replays pre-recorded real LLM responses under calibrated micro-sleeps to isolate control-plane latency from cloud API jitter.
4. Hypothesis Testing: Evaluated with two-tailed paired $t$-tests ($lpha = 0.001$) and 95% confidence intervals.\n