Repository navigation
Benchmarking Guide
Measure Precisely · Compare Fairly · Report Transparently
A standardized engineering methodology for evaluating the computational performance, numerical behavior, and resource efficiency of EveryO.
Home · Performance Optimization · Testing and Quality Assurance · Best Practices
Repository: github.com/krishanth7/EveryO
Maintainer: @krishanth7
Document Classification: Engineering methodology
Project Status: Experimental / Alpha
Important
Benchmark Integrity
EveryO is designed primarily for education, experimentation, and numerical verification. It is not represented as a production-scale training runtime.
All benchmark results must identify their hardware, software versions, input shapes, numerical precision, measurement methodology, and execution conditions.
The numerical examples in this guide are illustrative calculations, not measured EveryO benchmark results.
- Purpose and Scope
- Benchmarking Principles
- Benchmark Categories
- Benchmark Environment
- Reproducibility Requirements
- Benchmark Workflow
- Timing Methodology
- Statistical Analysis
- Tensor Operation Benchmarks
- Matrix Multiplication Benchmarks
- Automatic Differentiation Benchmarks
- Neural Network Benchmarks
- Training Throughput
- Optimizer Benchmarks
- Data Pipeline Benchmarks
- Memory Profiling
- CPU Performance
- CUDA Performance
- Mixed Precision Benchmarks
- Quantization Benchmarks
- Distributed Training Benchmarks
- ONNX Runtime Benchmarks
- Numerical Accuracy
- Performance Profiling
- Performance Regression Testing
- Benchmark Reporting
- Common Benchmarking Mistakes
- Acceptance Criteria
- Benchmark Submission Guidelines
- Official Resources
Benchmarking provides objective evidence about how software behaves under defined workloads.
For EveryO, benchmarking should answer:
- How long does an operation take?
- How does runtime change with tensor size?
- How much memory does an operation consume?
- How does backward execution compare with forward execution?
- How does performance vary across devices?
- How does numerical precision affect results?
- Does a code change introduce a performance regression?
EveryO benchmarking should emphasize:
- Reproducibility.
- Correctness.
- Transparency.
- Fair comparison.
- Resource efficiency.
- Regression detection.
This guide covers:
- CPU tensor execution.
- Automatic differentiation.
- Neural network layers.
- Model training.
- Optional CUDA acceleration.
- Mixed precision.
- Quantization.
- Distributed execution.
- ONNX interoperability.
A faster operation is not an improvement if it produces incorrect results.
All implementations must receive equivalent inputs and perform equivalent computations.
Exclude setup overhead from kernel-only benchmarks, but include it when measuring end-to-end latency.
Do not report a single favorable measurement as representative performance.
Benchmark reports should contain sufficient information for another developer to repeat the experiment.
A result on one device does not establish universal performance superiority.
| Category | Primary Measurement |
|---|---|
| Tensor operations | Latency |
| Matrix multiplication | Latency and throughput |
| Automatic differentiation | Forward/backward latency |
| Neural networks | Inference latency |
| Training | Samples per second |
| Optimizers | Update latency |
| Data pipeline | Batches per second |
| Memory | Peak allocation |
| CUDA | Kernel and end-to-end latency |
| Mixed precision | Speed, memory, numerical error |
| Quantization | Model size and numerical fidelity |
| Distributed training | Scaling efficiency |
| ONNX | Runtime latency and output agreement |
Every benchmark should record its environment.
benchmark:
framework: EveryO
framework_version: "0.2.0"
environment:
python_version: "record actual version"
numpy_version: "record actual version"
operating_system: "record actual OS"
hardware:
cpu: "record CPU model"
gpu: "record GPU model or none"
ram_gb: "record available RAM"
execution:
device: cpu
dtype: float32
warmup_iterations: 10
measured_iterations: 100This is a suggested reporting schema, not a required EveryO configuration file.
python --version
python -m pip show everyo
python -m pip show numpygit rev-parse HEADRecord:
- CPU model.
- Core count.
- Thread configuration.
- BLAS implementation, when relevant.
- Power and thermal conditions.
Record:
- GPU model.
- CUDA Toolkit version.
- Driver version.
- Device memory.
- Whether measurements include host-device transfers.
Use consistent shapes, dtypes, and input distributions.
When random inputs are used, record the seed.
Avoid benchmarking during unrelated CPU- or GPU-intensive workloads.
Use multiple iterations and, when appropriate, independent benchmark runs.
python -m pip freeze > benchmark-requirements.txtKeep individual timing samples where practical, not just the final summary.
DEFINE WORKLOAD
|
v
VERIFY OUTPUT
|
v
RECORD ENVIRONMENT
|
v
WARM UP
|
v
RUN MEASUREMENTS
|
v
COLLECT RAW RESULTS
|
v
COMPUTE STATISTICS
|
v
VALIDATE COMPARISON
|
v
PUBLISH REPORT
Define:
- Operation.
- Input shape.
- Dtype.
- Device.
- Reference implementation.
- Correctness tolerance.
Run warm-up iterations before collecting timed samples.
Check numerical results before reporting performance.
Publish both the measured results and their limitations.
Python's time.perf_counter_ns() is appropriate for measuring elapsed wall-clock time.
import statistics
import time
def benchmark(operation, warmup=10, repeat=100):
if warmup < 0 or repeat < 1:
raise ValueError("Invalid benchmark iteration count")
for _ in range(warmup):
operation()
samples_ms = []
for _ in range(repeat):
start = time.perf_counter_ns()
operation()
end = time.perf_counter_ns()
samples_ms.append((end - start) / 1_000_000)
return {
"median_ms": statistics.median(samples_ms),
"mean_ms": statistics.mean(samples_ms),
"min_ms": min(samples_ms),
"max_ms": max(samples_ms),
"samples_ms": samples_ms,
}This helper measures synchronous CPU execution.
For asynchronous GPU operations, synchronization or device-specific timing is required.
Without synchronization, the measured interval may capture only kernel submission overhead.
For very short operations, Python call and timer overhead can materially affect results.
Measure larger batches of repeated operations or use a suitable dedicated microbenchmark framework when necessary.
[ \bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i ]
The median is often useful for latency reporting because it is less sensitive to extreme observations.
[ s=\sqrt{\frac{\sum_{i=1}^{n}(x_i-\bar{x})^2}{n-1}} ]
Consider reporting:
- P50.
- P90.
- P95.
- P99.
Percentiles are especially useful when evaluating latency variability.
| Metric | Meaning |
|---|---|
| Mean | Average latency |
| Median | Typical latency |
| P95 | Tail latency |
| Standard deviation | Variability |
| Minimum | Fastest observation |
| Maximum | Slowest observation |
- Addition.
- Multiplication.
- Reduction.
- Reshape.
- Broadcasting.
- Matrix multiplication.
| Workload | Shape |
|---|---|
| Small vector | (128,) |
| Medium vector | (4096,) |
| Large vector | (1048576,) |
| Small matrix | (128, 128) |
| Medium matrix | (512, 512) |
| Large matrix | (2048, 2048) |
These are suggested workloads, not official performance thresholds.
import everyo as eo
a = eo.tensor([1.0, 2.0, 3.0])
b = eo.tensor([4.0, 5.0, 6.0])
result = a + b
print(result)For serious timing measurements, use appropriately sized inputs and exclude tensor construction when measuring only the addition operation.
Matrix multiplication is an important computational workload.
[ C=AB ]
For:
[ A\in\mathbb{R}^{M\times K} ]
and:
[ B\in\mathbb{R}^{K\times N} ]
the result is:
[ C\in\mathbb{R}^{M\times N} ]
[ FLOPs\approx2MKN ]
This counts approximately one multiplication and one addition per inner-product term.
[ GFLOPS= \frac{2MKN}{t\times10^9} ]
where (t) is elapsed time in seconds.
For a 1024 × 1024 matrix multiplication:
[ FLOPs\approx2(1024)^3 ]
[ FLOPs\approx2.147\times10^9 ]
If an implementation hypothetically takes 0.05 seconds:
[ GFLOPS\approx42.95 ]
This is a mathematical illustration, not an EveryO measurement.
The repository documents:
python benchmarks/benchmark_matmul.py --sizes 128 512 1024 2048Check the script's current command-line options before modifying the invocation.
Measure the time required to compute an operation's output.
Measure the time required to compute gradients.
[ T_{\text{step}}= T_{\text{forward}}+ T_{\text{loss}}+ T_{\text{backward}}+ T_{\text{optimizer}} ]
- Measure forward and backward separately.
- Use identical computation graphs.
- Control gradient accumulation.
- Exclude one-time graph construction only when the workload definition permits it.
- Validate gradients against a reference.
A forward-only benchmark does not measure training performance.
- Multilayer perceptrons.
- Convolutional networks.
- Recurrent networks.
- Attention-based models.
- Transformer encoders.
Record:
- Architecture.
- Parameter count.
- Batch size.
- Input dimensions.
- Output dimensions.
- Dtype.
- Device.
[ T_{\text{inference}}= T_{\text{forward}} ]
[ Throughput= \frac{N_{\text{samples}}}{T_{\text{elapsed}}} ]
Ensure competing implementations use equivalent mathematical operations and execution conditions.
[ Throughput= \frac{\text{Processed Samples}}{\text{Elapsed Seconds}} ]
[ StepsPerSecond= \frac{\text{Completed Steps}}{\text{Elapsed Seconds}} ]
[ T_{\text{epoch}}= \sum_{i=1}^{B}T_i ]
where (B) is the number of batches.
Suppose:
- 10,000 samples are processed.
- Total elapsed time is 25 seconds.
Then:
[ Throughput=\frac{10000}{25}=400 ]
Result:
400 samples per second.
This is an illustrative calculation.
Clearly state whether the measurement includes data loading and preprocessing.
EveryO includes:
- SGD.
- Momentum.
- Nesterov.
- Adam.
- AMSGrad.
- Optimizer step latency.
- Memory overhead.
- Convergence behavior.
- Numerical stability.
Optimizer speed should be evaluated using the same parameter count and gradient shapes.
A faster optimizer step does not necessarily imply faster convergence.
Report both computational cost and optimization outcomes.
- Dataset access latency.
- Batch creation time.
- Samples per second.
- Preprocessing overhead.
Measure:
- Raw data access.
- Preprocessing.
- Batching.
- Training computation.
Performance improvements must not change sample-label alignment or silently omit records.
Use identical datasets and batch sizes across comparisons.
- Tensor storage.
- Gradient storage.
- Model parameters.
- Optimizer state.
- Intermediate activations.
- Data batches.
For a dense tensor:
[ MemoryBytes= N_{\text{elements}}\times BytesPerElement ]
For a float32 tensor of shape (1024, 1024):
[ Memory= 1024\times1024\times4 ]
[ Memory=4,194,304\text{ bytes} ]
Approximately:
4 MiB of raw tensor data.
This excludes object overhead, gradients, temporary arrays, and allocator behavior.
Use appropriate system or Python memory profiling tools.
Clearly distinguish:
- Process resident memory.
- Python-managed allocations.
- NumPy native allocations.
- GPU device memory.
Record:
- CPU architecture.
- Physical cores.
- Logical threads.
- BLAS threading.
- Input sizes.
- Dtype.
BLAS libraries may use multiple CPU threads.
Different thread settings can significantly affect matrix multiplication performance.
For controlled experiments, configure thread-related environment variables before importing NumPy.
Example:
OMP_NUM_THREADS=1 python benchmarks/benchmark_matmul.py --sizes 128 512A single-thread benchmark and a multithread benchmark answer different performance questions.
Do not combine them without explanation.
EveryO documents a small native CUDA kernel implementation.
GPU benchmarking should focus on operations actually supported by that implementation.
Kernel latency: Device execution time.
Transfer latency: Host-to-device and device-to-host movement.
End-to-end latency: Complete operation including relevant transfers.
For asynchronous execution, synchronize appropriately before recording elapsed time.
[ Speedup= \frac{T_{\text{CPU}}}{T_{\text{GPU}}} ]
If a CPU operation hypothetically takes 100 ms and a GPU operation takes 25 ms:
[ Speedup=4 ]
This is an illustrative 4× speedup calculation, not a measured EveryO result.
- Confirm CUDA availability.
- Verify output correctness.
- Record GPU model.
- Report transfer inclusion.
- Test on actual hardware.
Where supported, compare:
- Float32.
- Mixed float16/float32 execution.
- Runtime.
- Memory.
- Numerical error.
- Gradient stability.
Use an appropriate float32 reference.
Mixed precision is not guaranteed to improve performance on CPU.
NumPy may perform float16 arithmetic through higher-precision internal operations.
Do not report speed improvements without also assessing numerical behavior.
EveryO provides dynamic quantization utilities.
Its documented implementation uses int8 weights without specialized faster int8 execution kernels.
- Model size.
- Numerical fidelity.
- Output differences.
- Inference latency.
[ Reduction= \left( 1-\frac{Size_{\text{quantized}}}{Size_{\text{original}}} \right)\times100% ]
Smaller weights do not automatically produce faster inference.
Avoid claiming quantization speedups unless they are directly measured.
Measure the performance impact of multiple workers.
[ S(p)=\frac{T(1)}{T(p)} ]
where (p) is the number of workers.
[ E(p)=\frac{S(p)}{p}\times100% ]
Suppose:
- Single-worker time = 100 seconds.
- Four-worker time = 40 seconds.
Then:
[ S(4)=2.5 ]
[ E(4)=62.5% ]
These are illustrative values.
Record:
- Worker count.
- Process topology.
- Machine count.
- Network configuration.
- Communication overhead.
- Dataset partitioning.
Same-host process testing does not establish cross-machine reliability.
Compare EveryO inference with the exported ONNX model under equivalent conditions.
python -m pip install -e ".[onnx]"import everyo as eo
model = eo.Sequential(
eo.Linear(4, 8),
eo.ReLU(),
eo.Linear(8, 2),
)
model.eval()
eo.export_onnx(
model,
"benchmark_model.onnx",
input_shape=(1, 4),
)Measure:
- EveryO inference latency.
- ONNX Runtime latency.
- Output differences.
- Model load time separately.
Do not include model export time in one implementation's inference latency while excluding equivalent setup from the other.
Traced ONNX graphs may encode fixed execution paths or shape assumptions.
Benchmark only supported input configurations.
Performance and correctness must be evaluated together.
[ E_{\text{abs}}=|y_{\text{test}}-y_{\text{reference}}| ]
[ E_{\text{rel}}= \frac{|y_{\text{test}}-y_{\text{reference}}|} {\max(\epsilon,|y_{\text{reference}}|)} ]
NumPy provides:
import numpy as np
np.testing.assert_allclose(
actual,
expected,
rtol=1e-5,
atol=1e-7,
)Here, actual and expected are arrays supplied by the benchmark.
The tolerances are illustrative; select values appropriate to the operation, dtype, and numerical reference.
Every performance result involving alternative implementations should include numerical validation.
EveryO provides profiling functionality.
Identify:
- Expensive operations.
- Unexpected execution frequency.
- Memory-intensive stages.
- Training bottlenecks.
import everyo as eo
with eo.profile() as profiler:
output = model(input_tensor)
print(profiler.summary())The model and input tensor must be initialized beforehand.
EveryO's profiler supports Chrome trace export.
Consult the current API reference for exact export usage and compatibility.
Profiling can alter execution timing.
Do not assume profiled execution has identical latency to uninstrumented execution.
Detect unexpected performance degradation after code changes.
Record performance from a known commit.
Run the same workload against the proposed change.
[ Regression%= \frac{T_{\text{candidate}}-T_{\text{baseline}}} {T_{\text{baseline}}}\times100% ]
If baseline latency is 20 ms and candidate latency is 24 ms:
[ Regression=20% ]
This is an illustrative calculation.
Investigate regressions using repeated runs and equivalent environments.
A small difference within normal measurement noise should not automatically be treated as a confirmed regression.
# EveryO Benchmark Report
## 1. Summary
Describe the benchmark objective.
## 2. Environment
- EveryO version:
- Git commit:
- Python version:
- NumPy version:
- Operating system:
- CPU:
- GPU:
- RAM:
## 3. Workload
- Operation:
- Input shape:
- Dtype:
- Device:
- Batch size:
## 4. Methodology
- Warm-up iterations:
- Measured iterations:
- Synchronization method:
- Timing method:
## 5. Results
| Metric | Value |
|---|---|
| Median latency | |
| Mean latency | |
| P95 latency | |
| Throughput | |
| Peak memory | |
## 6. Numerical Validation
- Reference:
- Absolute error:
- Relative error:
- Tolerance:
- Result:
## 7. Limitations
Describe limitations.
## 8. Reproduction
Provide exact commands.Benchmark reports should include reproducible commands and environment details.
Provide raw timing samples when practical.
Clearly separate observations from explanations and hypotheses.
| Mistake | Problem | Better Practice |
|---|---|---|
| One timing sample | Unreliable result | Repeat measurements |
| No warm-up | Startup effects | Use warm-up iterations |
| Different tensor shapes | Invalid comparison | Match workloads |
| Different precision | Misleading speed comparison | Report dtype |
| Missing GPU synchronization | Incorrect timing | Synchronize |
| Ignoring transfer time | Incomplete performance picture | Report timing scope |
| No correctness check | Fast but incorrect result | Validate outputs |
| Unreported hardware | Poor reproducibility | Record specifications |
| Cherry-picked results | Biased conclusions | Report distributions |
| Missing software versions | Hard to reproduce | Record dependencies |
| Confusing FLOPS and FLOPs | Unit ambiguity | Label operations and rates |
| Comparing unequal models | Invalid conclusions | Match architecture and workload |
A benchmark should be considered suitable for publication when:
- The workload is clearly defined.
- Hardware is documented.
- Software versions are recorded.
- Inputs are reproducible.
- Warm-up is performed where appropriate.
- Multiple measurements are collected.
- Results include statistical summaries.
- Numerical correctness is verified.
- Timing scope is explicit.
- Raw results are retained where practical.
- Limitations are disclosed.
- Reproduction commands are provided.
No universal latency or throughput threshold is imposed by this guide.
Community benchmark contributions are welcome.
- Confirm the benchmark runs.
- Verify correctness.
- Record environment metadata.
- Avoid unsupported performance claims.
- Include reproducible instructions.
A benchmark pull request should explain:
- Why the workload matters.
- What is being measured.
- How correctness is checked.
- What dependencies are required.
- Whether hardware-specific tests can be skipped.
Maintainers may evaluate:
- Reproducibility.
- Correctness.
- Maintainability.
- Benchmark stability.
- Relevance to EveryO's objectives.
A merged benchmark does not imply that EveryO outperforms other frameworks.
| Resource | Link |
|---|---|
| EveryO Repository | GitHub |
| README | README.md |
| Benchmark Scripts | benchmarks/ |
| Performance Documentation | docs/performance.md |
| CUDA Documentation | docs/cuda.md |
| Scaling Documentation | docs/scaling.md |
| API Reference | docs/api-reference.md |
| Testing | GitHub Actions |
| Contributing | CONTRIBUTING.md |
| NumPy | numpy.org |
| Python | python.org |
| ONNX Runtime | onnxruntime.ai |
Next Recommended Page: Numerical Accuracy and Validation →
Home · Performance Optimization · Best Practices
EveryO — Open Machine Learning. Built for Developers.
Project Maintainer: @krishanth7
Open Source · MIT Licensed · Community Driven
© 2026 EveryO Contributors