AIPerfKit

An x86 CPU AI benchmark suite that measures how fast — and explains why.

Run AI workloads across core → CCD → NUMA → socket scaling, capture deep telemetry (IPC, memory bandwidth, power, cache, frequency), and produce normalized JSON, charts, and interactive HTML reports.

---

Table of contents

• Why AIPerfKit
• Features
• Installation
• Quick start
• One-stop DUT runner
• Benchmarks
• Telemetry & derived metrics
• Platform auto-enable
• Run profiles
• CLI reference
• Results layout
• Licensed benchmarks
• Architecture
• Development
• Troubleshooting
• Documentation
• Contributing
• License

---

Why AIPerfKit

Most CPU benchmarks give you a single number. AIPerfKit gives you the number and the reason behind it. For every scaling point it records throughput alongside IPC, memory bandwidth, cache hit rates, package power, and frequency — so you can tell a compute-bound workload (scales with cores, stable IPC) from a memory-bound one (throughput saturates, IPC collapses) at a glance.

It is designed for repeatable, hands-off runs: point it at a machine (local or over SSH), and it discovers the topology, installs prerequisites, sweeps the core/CCD/NUMA/socket space, collects telemetry, and emits reports — with checkpoint/resume, retries, and a thermal watchdog for long sweeps.

Features

• 22 benchmark plugins across 6 categories — primitives, classical ML, inference runtimes, LLM serving, memory, and quick-score suites.
• Scaling sweeps from a single core through CCD / NUMA / socket to the full platform, with SMT-on/off and NUMA-local/interleave variants.
• Deep telemetry via 10 collectors (perf, turbostat, RAPL, mpstat, Intel PCM, AMD uProf, and more) plus derived metrics (perf/watt, joules/op, CCD efficiency, socket-scaling efficiency, NUMA penalty, SMT gain).
• Platform auto-enable — resolves AMD vs Intel collectors and artifact-gated benchmarks automatically from discovered topology.
• Automatic provisioning — detects the distro (apt / dnf / yum) and installs mandatory OS and pip prerequisites in bulk before a run.
• One-stop DUT runner — describe a machine once in a YAML file and run the full suite locally or over SSH with a single command.
• Reliability built in — preflight checks, lockfile, checkpoint/resume, retries, thermal watchdog, and SHA-256 integrity manifests.
• Data platform — ingest runs into SQLite/Parquet, snapshot baselines, and gate regressions in CI.
• Reproducible & CI-friendly — a --mock mode runs the entire pipeline with synthetic data (no hardware or licensed tools required).

Installation

Requires Python 3.10+ on x86_64 Linux (AMD EPYC or Intel Xeon). Windows is supported as a control host that drives remote Linux DUTs.

git clone <your-repo-url> AIPerfKit
cd AIPerfKit
python -m pip install -e ".[charts,ssh,dev]"

Extra	Adds	Use for
charts	kaleido, matplotlib	Static PNG charts in reports
ssh	paramiko	Remote deploy / run over SSH
pdf	weasyprint	PDF report export
dev	pytest, ruff, mypy	Development & testing
all	everything above + pyarrow	Full functionality

Quick start

# 1. Mock run — full pipeline, no hardware or tools needed
aiperfkit run --profile configs/run_profiles/smoke.yaml --mock --report

# 2. Inspect the machine (topology + available plugins)
aiperfkit discover

# 3. Real smoke run on Linux (~10 min); auto-installs prerequisites
aiperfkit run --profile configs/run_profiles/smoke.yaml --auto-install --report

# 4. Full scaling sweep (hours); licensed tools stay disabled by default
aiperfkit run --profile configs/run_profiles/full_scaling.yaml --auto-install --report

Optionally fetch open (Apache-2.0) models first:

./artifacts/scripts/fetch_open_artifacts.sh      # or .ps1 on Windows

One-stop DUT runner

Describe a device under test (DUT) once, then run the full suite against it with a single command — locally or over SSH. DUT definitions live in configs/duts/.

Create your DUT file from the template, then run it:

# 1. Copy the public template and edit host/user/credentials
cp configs/duts/example.yaml configs/duts/mydut.yaml

# 2. List configured DUTs
python scripts/run_dut_suite.py --list-duts

# 3. Full suite over SSH (reads configs/duts/mydut.yaml)
.\scripts\run_dut_suite.ps1 -Dut mydut

# Fast smoke validation instead of the full sweep
.\scripts\run_dut_suite.ps1 -Dut mydut -Smoke

# Or run on the current machine (committed, credential-free example)
./scripts/run_dut_suite.sh --dut local

The runner reads connection details, credentials, and the target profile from the DUT YAML, then deploys, provisions, runs, verifies, and fetches results. See Adding a DUT for new machines.

Credentials. Real DUT files (configs/duts/*.yaml, except the templates and local.yaml) are gitignored so hosts and passwords never get committed. Supply credentials via AIPERFKIT_SSH_PASSWORD / AIPERFKIT_SUDO_PASSWORD env vars, a gitignored configs/duts/secrets/<name>.yaml, or (least preferred) inline fields — see the security note.

Benchmarks

22 plugins are registered; run aiperfkit discover to see which are available on your machine. Licensed suites ship as fully-coded plugins that stay disabled until you supply the binary/dataset.

Category	Benchmarks	Notes
Primitive	libxsmm_gemm, onednn	libxsmm_gemm builds libxsmm + a bundled GEMM microbenchmark from source
Classical ML	sklearn_bench, xgboost_bench, lightgbm_bench	Honor CPU affinity / OMP_NUM_THREADS per scaling point
Inference	onnxruntime, openvino, pytorch, deepsparse, tensorflow_serving, spec†, mlperf†	Open models fetched via artifacts/scripts
LLM serving	llamacpp, vllm, tgi, ollama, xfastertransformer	Endpoint benchmarks disabled when the server is unreachable
Memory	stream, intel_mlc†	STREAM builds from source into ~/.aiperfkit/tools/
Quick score	phoronix, geekbench6†, geekbench_ai†	Single-shot scores

† Licensed / commercial — disabled in default profiles. See Licensed benchmarks.

Telemetry & derived metrics

Collectors are selected automatically for the platform; missing metrics are recorded as NOT_COLLECTED rather than guessed.

Collector	Metrics	Requirement
perf	IPC, branch-miss, L1/L2/L3 cache	linux-tools-$(uname -r), perf_event_paranoid <= 0
turbostat	package power, busy frequency, throttle	root + msr module
rapl	package energy, average power	readable /sys/class/powercap/*
mpstat / sar	CPU utilization	sysstat
time_v	peak memory, min/avg/max frequency	none (no root)
intel_pcm	memory bandwidth (Intel)	pcm-memory; auto-selected on Intel
amd_uprof	memory bandwidth (AMD)	free AMD uProf download; optional; pinned to CPU 0
pidstat / numastat	per-process + NUMA stats	sysstat / numactl

Derived metrics are computed post-run: performance_per_watt, joules_per_op, ccd_efficiency, socket_scaling_efficiency, numa_penalty, smt_gain, and LLM tokens_per_watt. See RUNBOOK §3 for the full metric catalog and privilege model.

Platform auto-enable

With auto_enable: true (the default), the resolver merges your profile with discovered topology and artifacts before benchmarks run:

Input	Behavior
topology.vendor	AMD → amd_uprof; Intel → intel_pcm
artifacts/manifest.yaml	Sets model_path / binary_path when files exist
Missing licensed artifact	Benchmark stays disabled (reason in manifest)
Endpoint unreachable	vllm / tgi / ollama / tensorflow_serving disabled
enabled: false in YAML	Never overridden

Every decision is recorded in run_manifest.json → platform_resolution. Point at downloaded models with AIPERFKIT_ARTIFACTS_DIR (default ./artifacts).

Run profiles

Profile	Purpose	Typical duration
smoke.yaml	Fast validation, 3–4 scaling points	~10 min
validation.yaml	CI mock: all plugins once at full platform	~5 min (mock)
full_scaling.yaml	Full core → CCD → NUMA → socket sweep	hours

OS provisioning. When auto_install: true (default in run profiles), the mandatory OS and pip packages for the selected benchmarks are installed automatically via distro detection (apt on Debian/Ubuntu, dnf/yum on RHEL/Fedora). Manual bulk scripts: scripts/install_tools_ubuntu.sh and scripts/install_tools_rhel.sh.

CLI reference

aiperfkit discover     Show topology + available benchmarks/collectors
aiperfkit doctor       Check prerequisites for a profile (no install)
aiperfkit provision    Install missing prerequisites
aiperfkit run          Run a profile with scaling + telemetry
aiperfkit report       Build HTML/PDF reports from a results dir
aiperfkit compare      Diff two PlatformProfile.json files
aiperfkit deploy       Deploy + run on a remote host over SSH
aiperfkit db-ingest    Ingest a run into the SQLite/Parquet results DB
aiperfkit baseline     Snapshot a run as a named performance baseline
aiperfkit regress      Gate a run against a baseline (exit 1 on regression)
aiperfkit trends       Build the historical trend dashboard from the DB
aiperfkit verify       Verify SHA-256 checksums for a run directory

Useful run flags: --mock, --auto-install / --no-install, --report, --dry-run, --resume <RUN_DIR>, --force, --check-only.

Results layout

Each run writes to a single self-describing folder, results/<system>_<timestamp>/ (e.g. results/example-epyc_20260721_022452/). The <system> prefix is the run's system_name — resolved in this order:

1. --system-name on the CLI (or run.system_name in a DUT YAML), then
2. system_name: in the run profile, else
3. the machine hostname where the run executes (fallback).

Set system_name per DUT so results are labeled by device (e.g. example-epyc) rather than by whatever host the run lands on. For remote runs, the DUT runner forwards it so the remote sweep is labeled the same way.

results/<system>_<timestamp>/
├── raw/                  # tool logs (perf, turbostat, benchmark stdout)
├── normalized/           # per-point + combined JSON with telemetry
├── charts/               # PNG charts
├── reports/              # interactive HTML dashboards
├── logs/                 # run.log: full timestamped console log for the run
├── run_manifest.json     # topology, platform_resolution, preflight
├── preflight.json        # disk/thermal/perf readiness checks
├── checkpoint.jsonl      # completed (benchmark, point) records for --resume
└── checksums.json        # SHA-256 of every artifact (aiperfkit verify)

Console output is timestamped and streamed live (with a periodic "still running" heartbeat during long single benchmarks) and mirrored verbatim to logs/run.log, so a run is fully captured and never looks stuck.

Licensed benchmarks

Commercial benchmark binaries are never downloaded or bundled. The plugins are fully implemented but disabled; enable them after you obtain the tool and accept its license.

Benchmark / tool	License	Enable after you obtain it
spec	Commercial (SPEC)	set binary_path, enabled: true
geekbench6 / geekbench_ai	Commercial (Primate Labs)	set binary_path
mlperf	MLPerf terms + datasets	set harness + dataset paths
intel_mlc	Intel download agreement	place mlc under artifacts/licensed/intel_mlc/

Free optional telemetry: amd_uprof (AMDuProfPcm) — install from AMD uProf; runs are not blocked if it is missing. Open (Apache-2.0) models are fetched by artifacts/scripts/fetch_open_artifacts.*. See docs/SCOPE.md and artifacts/licensed/*/README.md for details.

Architecture

aiperfkit/
├── cli.py            # argparse entry point (lazy-imported handlers)
├── topology/         # CPU discovery: vendor, cores, CCD/NUMA/socket layout
├── platform/         # resolver: auto-enable collectors + artifact-gated benches
├── config/           # run-profile, platform, and DUT YAML loaders
├── provision/        # prerequisite requirements + bulk OS/pip provisioning
├── benchmarks/       # 22 benchmark plugins (+ subprocess workers)
├── telemetry/        # 10 collectors + manager
├── core/             # runner, registry, scaling sweep, checkpoint/resume
├── report/           # HTML/PDF report + platform comparison
├── data/             # SQLite/Parquet DB, baselines, regression, trends, integrity
└── deploy/           # SSH deploy + remote run

Benchmarks and collectors are plugins registered in a central registry, so adding a workload means dropping in one class — no changes to the runner.

Development

python -m pip install -e ".[dev,charts]"

pytest -q                        # 127 tests, hardware-free (mock fixtures)
ruff check aiperfkit tests       # lint
mypy aiperfkit                   # type check

CI runs lint + type-check + tests on Python 3.10 / 3.11 / 3.12, plus mock validation and smoke runs, on every push and PR (see .github/workflows/ci.yml).

Troubleshooting

Symptom	Likely cause / fix
IPC / cache all empty	linux-tools-$(uname -r) missing or perf_event_paranoid > 0
Power / energy empty	RAPL powercap unreadable; run with a sudo path or as root
another AIPerfKit run holds …	a sweep is running, or a stale lock — remove results/.aiperfkit.lock if stale
Metrics only at low core counts	workload too short at high core counts; increase iterations
A remote run appears stuck	inspect / recover with the diagnostic helpers below

Remote diagnostics (SSH DUTs):

python scripts/_query_run_progress.py <dut>   # progress + ETA of the active run
python scripts/_remote_diag.py <dut>          # processes, locks, checkpoint state
python scripts/_remote_tail_log.py <dut>      # tail the remote run log
python scripts/_remote_cleanup.py <dut>       # kill orphan workers + clear stale locks

More in RUNBOOK §8.

Documentation

Doc	Contents
docs/X86_RUN_GUIDE.md	Step-by-step local & remote runs
docs/RUNBOOK.md	Operations, telemetry, regression gating, troubleshooting
docs/SCOPE.md	What is intentionally out of scope and why
configs/duts/README.md	DUT catalog & adding machines
CONTRIBUTING.md	Dev setup, adding plugins, PR checklist
CHANGELOG.md	Release history
THIRD_PARTY_LICENSES.md	Dependency/tool/model licenses & attribution

A note on credentials

Real DUT configs (configs/duts/*.yaml) are gitignored — only example.yaml (template), local.yaml (credential-free), and secrets/example.yaml are committed. Keep hosts and passwords out of git by using AIPERFKIT_SSH_PASSWORD / AIPERFKIT_SUDO_PASSWORD env vars, SSH keys, or a gitignored configs/duts/secrets/<name>.yaml. If you ever committed real credentials in the past, rotate them. See configs/duts/README.md.

Contributing

Contributions are welcome! Please read CONTRIBUTING.md for dev setup, how to add a benchmark or collector plugin, and the PR checklist (ruff, mypy, pytest must pass).

License

AIPerfKit is released under the MIT License. Third-party benchmarks and tools invoked by AIPerfKit remain under their own licenses; commercial binaries are never bundled or downloaded. All dependencies are permissive (MIT/BSD/Apache); the only copyleft dependency, paramiko (LGPL-2.1), is optional, unmodified, and not bundled. See THIRD_PARTY_LICENSES.md for the full inventory and attribution — including STREAM result-labeling notes.