Reproducibility and Benchmarks

Reproducibility and Benchmarks

Record an experiment precisely, rerun it with correctness checks, preserve raw benchmark measurements, and interpret performance claims within their measured scope.

Scope: FlagQuantum 0.3.1. The four complete Python scripts and the maintained CPU benchmark commands below were verified on October 20, 2026 with Python 3.12.14, PyTorch 2.13.0, and macOS ARM64. All execution here is local CPU work. No GPU, distributed system, external-framework benchmark, cloud platform, or QPU performance was tested for this page. Timings shown are observations from one documentation run, not performance targets.

Read Installation and Compatibility, Training Parameterized Circuits, and Choosing a Simulation Method first as needed.

Formula/code review — October 20, 2026: The equations are paired with code excerpts, and the local CPU examples were re-run with Python 3.12.14 and PyTorch 2.13.0. See the review record for executed commands, distributed checks, and untested hardware/provider paths. Earlier environment and output records are retained.

1. Decide What Must Be Reproduced

Goal	Evidence needed
Reproduce an exact simulator result	Circuit and inputs, execution method, dtype, outputs, versions, and numerical tolerance.
Reproduce a stochastic experiment	All relevant RNG seeds/state, sampling policy, shots/trajectories, versions, and uncertainty. Exact samples may not transfer across runtimes.
Reproduce training	Initial parameters, data/splits, loss, differentiation method, optimizer configuration, update count, and evaluation point.
Reproduce latency or throughput	Identical workload and output semantics, timing scope, hardware/runtime settings, warmup, raw samples, repetition policy, and stability checks.
Reproduce a hardware observation	The provider/task/receipt, physical mapping, calibration time, shots, compiler/executed-program evidence, and a statistical comparison rather than an assumed fixed histogram.

A deterministic numerical result does not imply deterministic timing. A successful correctness test does not establish a speedup. A saved package version does not capture the workload or the machine by itself.

2. Record the Environment and Workload

At minimum, retain:

Area	Record
Software identity	FlagQuantum version, exact wheel/source revision, installed dependency versions, Python, optional backends, and build/toolchain details for custom builds.
Hardware/runtime	OS, CPU model and topology, GPU/accelerator model/count, memory, driver/runtime, thread counts, affinity/NUMA placement, power settings, and relevant feature switches.
Workload	Source files and hashes, serialized IR or circuit construction, input data/initial state, qubit count/order, batch, depth/layers, and parameter values.
Execution	Mode, backend, device, dtype, requested outputs, gradient method, approximation/truncation policy, seeds, shots, trajectories, and fallback behavior.
Distributed execution	World/local sizes, ranks and placement, nodes/interconnect, collective backend, sharding/replication semantics, and all-rank correctness/logs.
Measurement	Exact command, warmup, sample count, calls per sample, synchronization, measured phase, raw times, memory API, and aggregation rule.

The collector below inventories the active interpreter, records selected settings and named source hashes, and optionally records the revision/status of an explicitly supplied FlagQuantum source checkout. It does not dump the complete process environment.

Example 1: Save an Environment Manifest

Save as record_environment.py:

import argparse
import hashlib
import importlib.metadata as metadata
import json
import os
import platform
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path
import torch

parser = argparse.ArgumentParser(description='Record the active environment and explicitly named source files.')
parser.add_argument('--output', type=Path, default=Path('repro-artifacts/environment.json'))
parser.add_argument('--script', type=Path, action='append', default=[])
parser.add_argument('--source-checkout', type=Path, help='Optional FlagQuantum source checkout used for this experiment')
args = parser.parse_args()
packages = {}
for distribution in metadata.distributions():
    name = distribution.metadata.get('Name')
    if name:
        packages[name] = metadata.version(name)
source = None
if args.source_checkout is not None:
    def git(*arguments):
        return subprocess.check_output(['git', '-C', str(args.source_checkout), *arguments], text=True).strip()
    source = {'checkout': str(args.source_checkout.resolve()), 'commit': git('rev-parse', 'HEAD'),
              'status_porcelain': git('status', '--porcelain')}
settings = ['OMP_NUM_THREADS', 'MKL_NUM_THREADS', 'OPENBLAS_NUM_THREADS', 'CUDA_VISIBLE_DEVICES',
            'CUBLAS_WORKSPACE_CONFIG', 'FQ_NATIVE_CPU_CX_GATHER', 'FQ_STATEVECTOR_CPU_DIRECT_LOCAL']
manifest = {
    'schema': 'wiki.experiment_environment.v1',
    'captured_at_utc': datetime.now(timezone.utc).isoformat(),
    'python': sys.version, 'executable': sys.executable,
    'platform': platform.platform(), 'machine': platform.machine(), 'processor': platform.processor(),
    'logical_cpu_count': os.cpu_count(),
    'packages': dict(sorted(packages.items(), key=lambda item: item[0].lower())),
    'torch': {'version': torch.__version__, 'cuda_build': torch.version.cuda,
              'cuda_available': torch.cuda.is_available(), 'num_threads': torch.get_num_threads(),
              'num_interop_threads': torch.get_num_interop_threads(),
              'deterministic_algorithms': torch.are_deterministic_algorithms_enabled(),
              'build_config': torch.__config__.show()},
    'selected_environment': {name: os.environ[name] for name in settings if name in os.environ},
    'source_checkout': source,
    'script_sha256': {str(path): hashlib.sha256(path.read_bytes()).hexdigest() for path in args.script},
    'invocation': sys.argv,
}
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(json.dumps(manifest, indent=2, sort_keys=True) + '\n')
print('FlagQuantum:', metadata.version('flagquantum'))
print('PyTorch:', torch.__version__)
print('Python:', platform.python_version())
print('CUDA available:', torch.cuda.is_available())
print('Recorded script files:', len(args.script))
print('Environment manifest written:', args.output)

Save all four scripts on this page before running the following commands in their common directory:

python record_environment.py --script reproduce_experiment.py --script run_packaged_benchmark.py --script benchmark_public_run.py

Observed output in the verification environment:

FlagQuantum: 0.3.1
PyTorch: 2.13.0
Python: 3.12.14
CUDA available: False
Recorded script files: 3
Environment manifest written: repro-artifacts/environment.json

Your versions, CUDA availability, interpreter paths, and machine fields may differ. With a source installation, add --source-checkout /absolute/path/to/FlagQuantum only for the checkout actually used. A clean commit plus recorded changes is more informative than a revision from an unrelated working directory. Preserve the dirty diff and required untracked inputs separately; porcelain status only lists their presence.

The manifest records the collector process. An experiment can change thread counts or deterministic settings internally, and a child benchmark can receive explicit environment overrides; record those in the corresponding experiment/launch artifact as the scripts below do. The allowlist is not an exhaustive inventory of every possible runtime switch. Supplement CPU/GPU/driver/topology details that the portable collector cannot establish.

A package inventory is not a portable lockfile. For ordinary pip environments, also save python -m pip freeze --all output and keep the exact package artifacts/index constraints or a tested lockfile appropriate to the target platform. Custom accelerator builds, editable installs, local patches, and binary build options need additional provenance. Version strings alone do not identify every installed byte.

3. Reproduce Training, Sampling, and a Saved Circuit

Example 2: Run Twice and Replay Serialized IR

Save as reproduce_experiment.py:

import argparse
import json
from pathlib import Path
import torch
import flagquantum as fq

parser = argparse.ArgumentParser()
parser.add_argument('--output', type=Path, default=Path('repro-artifacts/experiment.json'))
args = parser.parse_args()
torch.set_num_threads(1)
torch.set_num_interop_threads(1)
torch.use_deterministic_algorithms(True)
seed, steps, learning_rate, target_value = 7, 80, 0.1, 0.25
options = fq.ExecutionOptions(mode='statevector', backend='pytorch', device='cpu', precision='complex64', seed=seed)

def experiment():
    torch.manual_seed(seed)
    theta = torch.nn.Parameter(torch.tensor(0.6))
    optimizer = torch.optim.SGD([theta], lr=learning_rate)
    losses = []
    for _ in range(steps):
        optimizer.zero_grad()
        value = fq.run(fq.Circuit(1).ry(0, theta), options=options, outputs=fq.expectation(fq.Z(0))).expectation()
        loss = (value - target_value).square().mean()
        loss.backward()
        losses.append(loss.item())
        optimizer.step()
    final_circuit = fq.Circuit(1).ry(0, theta.detach().clone())
    final = fq.run(final_circuit, options=options, outputs=fq.expectation(fq.Z(0))).expectation().item()
    assert abs(final - target_value) < 1e-5
    bell = fq.Circuit(2).h(0).cx(0, 1)
    counts = fq.run(bell, options=options, outputs=fq.counts(), shots=128).counts[0]
    assert sum(counts.values()) == 128 and set(counts) <= {'00', '11'}
    return {'losses': losses, 'theta': theta.item(), 'final_expectation': final, 'counts': counts}, final_circuit.to_ir()

first, ir = experiment()
second, _ = experiment()
assert first == second  # Same process, versions, CPU settings, and seeded request.
encoded = ir.to_json(indent=2)
restored = fq.CircuitIR.from_json(encoded)
assert restored.content_hash == ir.content_hash
replayed = fq.run(restored, options=options, outputs=fq.expectation(fq.Z(0))).expectation().item()
assert abs(replayed - first['final_expectation']) < 1e-7
args.output.parent.mkdir(parents=True, exist_ok=True)
ir_path = args.output.with_name(args.output.stem + '-circuit.json')
ir_path.write_text(encoded + '\n')
args.output.write_text(json.dumps({
    'schema': 'wiki.reproducible_experiment.v1', 'seed': seed, 'steps': steps,
    'learning_rate': learning_rate, 'target': target_value, 'initial_theta': 0.6,
    'mode': 'statevector', 'backend': 'pytorch', 'device': 'cpu', 'precision': 'complex64',
    'threads': torch.get_num_threads(), 'interop_threads': torch.get_num_interop_threads(),
    'flagquantum': fq.__version__, 'torch': torch.__version__, 'deterministic_algorithms': True,
    'shots': 128, 'result': first, 'final_ir_identity': ir.content_hash,
}, indent=2, sort_keys=True) + '\n')
print('Repeated training and sampling match:', first == second)
print(f'Initial loss: {first["losses"][0]:.6f}')
print(f'Final parameter: {first["theta"]:.6f}')
print(f'Post-update expectation: {first["final_expectation"]:.6f}')
print('Seeded Bell counts:', dict(sorted(first['counts'].items())))
print('Saved-IR replay matches: True')
print('Saved experiment:', args.output)
print('Saved final circuit:', ir_path)

Run:

python reproduce_experiment.py

Observed output in the verified environment:

Repeated training and sampling match: True
Initial loss: 0.331011
Final parameter: 1.318116
Post-update expectation: 0.250000
Seeded Bell counts: {'00': 58, '11': 70}
Saved-IR replay matches: True
Saved experiment: repro-artifacts/experiment.json
Saved final circuit: repro-artifacts/experiment-circuit.json

This example fixes the initial angle, CPU thread settings, deterministic-algorithm request, seed, exact-statevector mode, precision, loss, optimizer, and step count. It runs two fresh training experiments and compares their complete result records. The fresh final evaluation occurs after the last optimizer update. The saved IR is then reloaded and independently executed again.

The training target is the expectation cos(theta)=0.25; the numerical assertion checks that target. Bell-count assertions check total shots and allowed outcomes rather than promising a particular random split. {'00': 58, '11': 70} is this environment's seeded output, not a cross-version requirement.

The JSON experiment artifact retains the configuration, losses, final parameter, counts, and IR identity. The final-circuit JSON freezes a forward program, not an optimizer checkpoint or the entire training computation graph. To resume training exactly, save/restore the model and optimizer state, step, relevant RNG states, data-loader/sampler progress, and any scheduler/scaler state used by the actual workload.

PyTorch explicitly limits reproducibility guarantees across releases, platforms, and CPU/GPU implementations. Deterministic algorithms can also reject unsupported operations or change performance. Seed Python, NumPy, other backends, data sampling, and rank-local generators when your experiment uses them; setting only torch.manual_seed does not seed every source of randomness. See PyTorch reproducibility guidance.

For noisy simulation, record noise-model identity, trajectory IDs/count, truncation, and shots separately. For hardware, compare statistical distributions and calibration context; an identical seed cannot force identical QPU outcomes. See Noise and Open-System Simulation and QPU Digital Twins.

4. Discover and Run Maintained Benchmarks

The installed CLI is flagquantum-benchmark. Using the module form selects the active Python interpreter explicitly:

python -m flagquantum.benchmarking list
python -m flagquantum.benchmarking info statevector_local
python -m flagquantum.benchmarking run environment_probe --json-output benchmark-artifacts/environment-probe.json

list prints available runners and info describes the selected runner. environment_probe writes the flagquantum.benchmark.environment.v1 payload. It is a lightweight host/process manifest; its device string reflects visibility configuration and is not proof that a GPU was detected or exercised. Its schema is not a complete package/driver inventory, so retain the richer manifest as well.

For runner-specific help in this release, call the runner module directly:

python -m flagquantum.benchmarking.statevector_local --help

The dispatcher can consume --help itself, so dispatcher help is not necessarily the selected runner's option list.

Example 3: Execute the Maintained Local Statevector Runner

Save as run_packaged_benchmark.py:

import argparse
import json
import os
import subprocess
import sys
from pathlib import Path

parser = argparse.ArgumentParser(description='Run the maintained CPU statevector benchmark and retain its raw artifact.')
parser.add_argument('--output-dir', type=Path, default=Path('benchmark-artifacts'))
parser.add_argument('--iterations', type=int, default=7)
args = parser.parse_args()
args.output_dir.mkdir(parents=True, exist_ok=True)
raw_path = args.output_dir / 'statevector-local.json'
command = [sys.executable, '-m', 'flagquantum.benchmarking', 'run', 'statevector_local',
           '--device', 'cpu', '--n-wires', '8', '--batch-size', '2', '--layers', '2',
           '--warmup', '2', '--iterations', str(args.iterations), '--json-output', str(raw_path)]
environment = os.environ.copy()
environment['OMP_NUM_THREADS'] = '1'
environment['MKL_NUM_THREADS'] = '1'
completed = subprocess.run(command, env=environment, text=True, capture_output=True, check=False)
(args.output_dir / 'runner-stdout.txt').write_text(completed.stdout)
(args.output_dir / 'runner-stderr.txt').write_text(completed.stderr)
(args.output_dir / 'launch.json').write_text(json.dumps({
    'command': command,
    'explicit_environment': {'OMP_NUM_THREADS': '1', 'MKL_NUM_THREADS': '1'},
}, indent=2) + '\n')
completed.check_returncode()
payload = json.loads(raw_path.read_text())
assert payload['correctness']['passed']
assert payload['workload']['dtype'] == 'complex64'
assert payload['world_size'] == 1
assert len(payload['optimized']['samples_seconds']) == args.iterations
print('Runner schema:', payload['schema_version'])
print('Correctness passed:', payload['correctness']['passed'])
print('Raw samples per measured path:', args.iterations)
print(f'Optimized median ms: {1000*payload["optimized"]["median_seconds"]:.6f}')
print(f'Sequential reference median ms: {1000*payload["sequential_reference"]["median_seconds"]:.6f}')
print(f'Reference / optimized ratio: {payload["speedup"]:.6f}')
print('Stability gate passed:', payload['stability_gate_passed'])
print('Scalability claim allowed:', payload['scalability_claim_allowed'])
print('Release gate allowed:', payload['release_gate_allowed'])
print('Raw benchmark written:', raw_path)

Run:

python run_packaged_benchmark.py

Output observed during one verification run; timing, ratio, and stability values will vary:

Runner schema: flagquantum.statevector.local_performance.v1
Correctness passed: True
Raw samples per measured path: 7
Optimized median ms: 1.018750
Sequential reference median ms: 1.639000
Reference / optimized ratio: 1.608834
Stability gate passed: True
Scalability claim allowed: False
Release gate allowed: False
Raw benchmark written: benchmark-artifacts/statevector-local.json

The wrapper runs an 8-qubit, batch-2, two-layer CPU workload with explicit single-thread launch settings, two warmup calls, and seven samples per path. It retains the raw runner JSON, stdout/stderr, and command/environment overrides. Reuse a new --output-dir for each measurement you want to preserve; these example writers can replace files at the same path.

The maintained runner compares an optimized refreshed Circuit.state path with a deliberately unfused same-IR sequential dense-matrix reference. This is an internal implementation baseline. Its ratio does not compare FlagQuantum with Qiskit, Cirq, PennyLane, a cloud service, or hardware. The reference shares FlagQuantum semantics/gate definitions, so it is also not a wholly independent framework correctness proof.

The runner checks statevector agreement and preserves raw times, error/tolerance, sample counts, and coefficient of variation. Its successful process exit establishes the runner's correctness verdict; inspect the separate stability fields yourself. A successful exit is not a release/performance gate pass.

The payload explicitly reports world_size=1, scalability_claim_allowed=False, and release_gate_allowed=False. Preserve these limits. A fast CPU result cannot be relabeled as release-grade, GPU, multi-card, distributed, or QPU evidence.

The runner's CPU peak_memory_allocated_bytes=0 is an unmeasured/sentinel field in this path, not evidence that the simulation consumes zero memory. GPU allocator figures, process RSS, peak resident memory, and planned state bytes measure different quantities and must be labeled separately.

5. Measure the Public Application Path Separately

Internal state-evolution latency and a complete application API call answer different questions. If your application calls fq.run and requests probabilities, benchmark that same path and output.

Example 4: Time fq.run with an Explicit Scope

Match Timing Equations to the Benchmark Implementation

Quantity	Corresponding code in benchmark_public_run.py	Interpretation
$t_i=(t_{\mathrm{end}}-t_{\mathrm{start}})/C$	(time.perf_counter() - started) / calls_per_sample	One sample is mean seconds per batch call over C calls.
$t_{\mathrm{reported}}=\mathrm{median}_i(t_i)$	median = statistics.median(timings)	Median across recorded samples; warmup is excluded.
$\mathrm{CV}=\sigma_{\mathrm{population}}(t_i)/\mathrm{mean}_i(t_i)$	cv = statistics.pstdev(timings) / statistics.fmean(timings)	Population standard deviation matches the actual implementation.
$t_{\mathrm{ms}}=1000t_{\mathrm{s}}$	1000 * median	The printed milliseconds are per batch call, not per circuit.
$\sum_k p_{b,k}=1$	torch.allclose(probabilities.sum(dim=1), torch.ones(2), atol=1e-5)	Validate output semantics before interpreting timing.

The script measures CPU fq.run on an already constructed circuit. Accelerator timing requires explicit device synchronization; this CPU benchmark does not validate accelerator latency. Strong-scaling speedup $S_P=T_1/T_P$ and efficiency $S_P/P$ require additional matched measurements and are not produced by this script.

Save as benchmark_public_run.py:

import argparse
import json
import platform
import statistics
import time
from pathlib import Path
import torch
import flagquantum as fq

parser = argparse.ArgumentParser(description='Measure CPU fq.run planning, execution, and requested-output construction.')
parser.add_argument('--output', type=Path, default=Path('benchmark-artifacts/public-run.json'))
args = parser.parse_args()
torch.set_num_threads(1)
torch.set_num_interop_threads(1)
options = fq.ExecutionOptions(mode='statevector', backend='pytorch', device='cpu', precision='complex64')
circuit = fq.Circuit(8, bsz=2)
for layer in range(2):
    for qubit in range(8):
        circuit.ry(qubit, 0.2 + 0.01*qubit)
    for qubit in range(7):
        circuit.cx(qubit, qubit+1)

def evaluate():
    return fq.run(circuit, outputs=fq.probabilities(), options=options).probabilities

warmup, samples, calls_per_sample = 2, 9, 5
with torch.no_grad():
    for _ in range(warmup):
        evaluate()
    timings = []
    for _ in range(samples):
        started = time.perf_counter()
        for _ in range(calls_per_sample):
            probabilities = evaluate()
        timings.append((time.perf_counter() - started) / calls_per_sample)
assert tuple(probabilities.shape) == (2, 256)
assert torch.allclose(probabilities.sum(dim=1), torch.ones(2), atol=1e-5)
assert (probabilities >= -1e-7).all()
median = statistics.median(timings)
cv = statistics.pstdev(timings) / statistics.fmean(timings)
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(json.dumps({
    'schema': 'wiki.public_run_timing.v1', 'flagquantum': fq.__version__, 'torch': torch.__version__,
    'platform': platform.platform(), 'device': 'cpu', 'precision': 'complex64',
    'threads': torch.get_num_threads(), 'interop_threads': torch.get_num_interop_threads(), 'n_qubits': 8, 'batch_size': 2, 'layers': 2,
    'circuit_ir': circuit.to_ir().to_dict(), 'ir_identity': circuit.to_ir().content_hash,
    'output': 'full-register probabilities', 'gradients': False,
    'scope': 'fq.run on an already constructed circuit; excludes circuit construction and disk I/O',
    'warmup_calls': warmup, 'calls_per_sample': calls_per_sample,
    'samples_seconds_per_call': timings, 'median_seconds': median, 'coefficient_of_variation': cv,
    'probability_checks_passed': True,
}, indent=2, sort_keys=True) + '\n')
print('Probability checks passed: True')
print('Timing scope: fq.run on an already constructed circuit')
print('Samples / calls per sample:', samples, '/', calls_per_sample)
print(f'Median milliseconds per batch call: {1000*median:.6f}')
print(f'Coefficient of variation: {cv:.6f}')
print('Saved public-run benchmark:', args.output)

Run:

python benchmark_public_run.py

Output observed during one verification run; the numerical timing and variation are not fixed expected outputs:

Probability checks passed: True
Timing scope: fq.run on an already constructed circuit
Samples / calls per sample: 9 / 5
Median milliseconds per batch call: 3.115617
Coefficient of variation: 0.051812
Saved public-run benchmark: benchmark-artifacts/public-run.json

This measures a warmed fq.run call on an already constructed circuit, including the work that call performs to prepare/execute the request and construct the full probability output. Circuit construction, initial imports, startup, and JSON writing are outside the timed region. It is inference under torch.no_grad(), not a backward or optimizer-step benchmark.

The probability checks validate shape, normalization, and nonnegativity; they do not compare the full output to an independent simulator. The official runner's statevector comparison above provides a different correctness check for its own workload. Do not substitute either for application-specific correctness requirements.

Each recorded value averages five calls. The median is therefore milliseconds per batch call, with batch size 2; it is not a measured per-item latency, nor is the distribution of block averages a per-request tail-latency distribution. The raw artifact records that distinction.

An earlier run of the same timed loop on this documentation host measured 4.334683 ms with CV 0.226195, while the displayed run measured 3.115617 ms with CV 0.051812. Both observations were retained; the difference illustrates why one timing is insufficient for a performance conclusion.

Keep every valid measured sample. If variation is high, report it, inspect background load/thermal behavior, and repeat a predeclared protocol. Do not select only the fastest run or keep changing the workload/settings until a desired ratio appears. Small smoke workloads can be dominated by Python, dispatch, synchronization, and machine scheduling.

6. Interpret Performance Results Correctly

Metric	Meaning and caveat
Median wall time	Typical observed duration for the stated unit of work; state warmup, batching, and timing boundaries.
Mean, range, raw samples, CV	Show variability. CV is standard deviation divided by mean; few samples and averaging can hide important behavior.
Throughput	Completed work per unit time under a stated concurrency and saturation policy; batch_size / call_time is not automatically independent-job throughput.
Speed ratio	baseline_time / candidate_time for semantically matched workloads. Name the baseline and preserve the absolute times.
Peak memory	Report the API and scope: allocator allocation/reservation, process RSS, planned state, per-rank versus aggregate, and result buffers.
Strong scaling	Fixed total workload as devices/nodes increase; report speedup and efficiency against an appropriate baseline.
Weak scaling	A declared per-rank/work-growth policy as ranks increase; not the same experiment as fixed-work strong scaling.
Time to accuracy	Runtime to a stated error/gradient/training criterion, including iterations and approximation policy; single-step latency is insufficient.

Match Semantics Before Comparing Engines

Compare the same circuit/initial state, parameter values, batch, dtype, output representation, and numerical tolerance. Normalize qubit conventions first. Match exact versus approximate methods, noise, shots/trajectories, and gradient families. Requesting a full statevector is different from one scalar expectation; analytical expectations differ from finite-shot estimates.

For training, separate forward, full parameter gradient, optimizer update, and end-to-end convergence. A parameter-shift/finite-difference gradient with many circuit executions is not a like-for-like substitute for one native reverse-mode gradient without explicit accounting. A changed approximation or accuracy target changes the workload.

The maintained comparison runners include simulator_compare, simulator_compare_cirq, simulator_compare_pennylane, simulator_workload_corpus, and differentiable_simulator_corpus, with their own extras and support boundaries. Inspect each runner before installing dependencies or choosing a comparison. The documented differentiable CPU corpus matches differentiation families; it does not invent a native gradient contract for a bridge lacking one. These external comparisons were not executed for this page.

Synchronization, Warmup, and Caching

CPU perf_counter timing in these examples measures elapsed wall time. Accelerators are often asynchronous: use the runtime's supported synchronization before/after timed work or correctly recorded device events according to the intended metric. Timing host enqueue alone is not accelerator execution latency. Include host/device copies or result materialization when they are part of the application's work.

Separate first-call/compile/startup latency from warmed steady-state performance. For JIT, graph capture, gate fusion, or memoized observables, record what is cached and which phase is measured. Verify that repeated calls actually evaluate the requested workload rather than reading a previously materialized result. The official local runner uses an explicit refreshed state computation for this reason.

Thread count, affinity, NUMA placement, frequency/thermal state, concurrent jobs, and power policy can materially change timing. Match those settings in A/B comparisons. Preserve both successful and failed correctness/stability results; do not silently discard failed cases from an aggregate ratio.

Memory and Distributed Claims

A dense statevector alone stores B*2**n complex amplitudes; a density matrix stores B*4**n elements. Those formulas exclude temporaries, gradients, cached data, communication buffers, retained trajectories, and output materialization.

For distributed work, state sharding, replicated data parallelism, and independent-job throughput are different semantics. Report per-rank and aggregate memory, communication evidence, node topology, collective backend, and any final gather. A multi-process launch or a row labeled with several GPUs does not establish that the state or gradient was actually distributed. Preserve claim blockers returned by the runner/report. See Distributed Simulation and Training.

7. Reproduction Checklist for a Shareable Result

A useful artifact directory contains:

• An environment manifest, exact commands, source revision/dirty changes, dependency/build identity, and hashed scripts/data.
• The workload/circuit or serialized IR, initial parameters, seeds and execution/optimizer configuration.
• Correctness reference, tolerance, actual error, and relevant statistical uncertainty.
• Raw timing and memory samples, output logs, warmup/call grouping, hardware/runtime settings, and run timestamps.
• A summary that names the measured path, baseline, absolute results, variability, and evidence limits.

The example manifests can contain local paths and the official probe includes a hostname. Review those details before sharing; they contain context rather than a required public identity. Store application credentials separately from experiment artifacts.

For repository research scripts, use a checkout matching the artifact's recorded commit and install the exact documented extras/toolchain. Do not assume that running inside a source checkout imports the same binary package as a separately installed wheel. The installed maintained runners above are the shortest verified path on this page.

A historical benchmark report is evidence for its recorded machine, workload, software, and methodology. Re-running a tiny example does not reproduce a published full-scale report, and a new aggregation over old raw measurements does not observe new hardware. New source changes or different settings require a fresh, clearly identified result rather than overwriting the meaning of the old artifact.

8. Common Problems

Problem	Check
Benchmark CLI is missing	Use the same environment's python -m flagquantum.benchmarking; verify FlagQuantum 0.3.1 is installed in that interpreter.
Help shows only dispatcher arguments	Use info for discovery and the selected runner module's --help for options.
Seeded results differ across environments	Compare exact dependency/build/device settings and supported deterministic operations; use appropriate numerical/statistical tolerances.
Manifest settings differ from experiment settings	The collector is a separate process; inspect explicit in-script and child-process overrides.
A correctness assertion fails	Resolve output order, dtype, approximation, input, and algorithm differences before interpreting speed.
A timing ratio changes substantially	Inspect raw samples, thermal/load state, thread/affinity policy, measured phase, and baseline identity.
Reported CPU peak memory is zero	Check whether the field is a sentinel/unmeasured value; collect memory with a suitable documented API.
JSON timing reruns differ	They should: measurement noise changes raw samples. Compare workload identity and methodology before comparing statistics.
A GPU/distributed workflow is unavailable	Use its dedicated setup/runbook and actual hardware evidence; a CPU smoke run cannot validate it.

Sources and Related Guides

FlagQuantum sources below are pinned to v0.3.1:

• Benchmark overview and reproduction commands
• Maintained benchmarking interfaces and evidence limits
• Runner discovery and dispatcher · Environment probe
• Local statevector runner: timing, reference, correctness, and claim fields
• Benchmark payload contracts
• PyTorch reproducibility guidance · PyTorch benchmark utilities
• Training Parameterized Circuits · Choosing a Simulation Method · Troubleshooting and FAQ