Release v2.0.0 · alubbock/microbench

Microbench v2 is a significant upgrade with many new features versus v1.1.0.
Be sure to review the breaking changes before upgrading.

New features

Command-line interface (microbench -- COMMAND): wrap any external
command and record host metadata alongside timing without writing Python
code. Useful for SLURM jobs, shell scripts, and compiled executables.
- Records command, returncode (list, one per timed iteration),
  alongside the standard timing fields. Mixins are specified by short
  kebab-case names without the MB prefix (e.g. host-info);
  original MB-prefixed names are also accepted.
- Use
  --mixin MIXIN [MIXIN ...] to select metadata to capture (defaults to
  host-info, slurm-info, loaded-modules, python-info
  and working-dir)
- Use --show-mixins to
  list all available mixins with descriptions; use --field KEY=VALUE to
  attach extra labels
- Use --iterations N and --warmup N for repeat
  timing
- Use --stdout[=suppress] and --stderr[=suppress] to capture
  subprocess output into the record (output is re-printed to the terminal
  unless =suppress is given)
- Use --monitor-interval SECONDS to sample
  child process CPU and memory over time.
- Some mixins expose
  their own configuration flags (shown in --show-mixins and --help)
- Capture failures are non-fatal by default
  (capture_optional = True), making the CLI safe across heterogeneous
  cluster nodes.
- The process exits with the first non-zero returncode seen
  across all timed iterations if present, or zero (success) otherwise.
summary(results) / bench.summary(): prints min / mean / median /
max / stdev of call.durations across all results. No dependencies required
beyond the Python standard library. bench.summary() is a one-liner
convenience that calls bench.get_results() internally. The module-level
summary(results) accepts any list of dicts and can be composed with other
results-processing steps.
```
from microbench import MicroBench, summary

bench = MicroBench()

@bench
def my_function():
    ...

for _ in range(10):
    my_function()

bench.summary()
# n=10  min=0.000031  mean=0.000038  median=0.000036  max=0.000059  stdev=0.000008

# or with explicit results list:
summary(bench.get_results())
```
bench.time(name) sub-timing: [Python API] label phases inside a single benchmark
record with named timing sections. Sub-timings accumulate in call.timings as
[{"name": ..., "duration": ...}, ...] in call order. Compatible with
bench.record(), bench.arecord(), @bench (sync and async), and
bench.record_on_exit(). Calling outside an active benchmark is a silent
no-op; call.timings is absent when bench.time() is never called.
```
with bench.record('pipeline'):
    with bench.time('parse'):
        data = parse(raw)
    with bench.time('transform'):
        result = transform(data)
```
Async support: [Python API] the @bench decorator now detects async def functions
and returns an async def wrapper that must be awaited. A new
bench.arecord(name) method provides the async counterpart of
bench.record() for use with async with. All mixins, static fields,
output sinks, iterations, and warmup work identically to the sync path.
MBLineProfiler raises NotImplementedError at decoration time when used
with an async function (line profiling of coroutines is not supported).
```
@bench
async def fetch():
    await asyncio.sleep(0.01)

asyncio.run(fetch())

async with bench.arecord('load'):
    await load_data()
```
Note: elapsed time includes event-loop interleaving from other concurrent
tasks; run in an otherwise-idle event loop for repeatable results.
bench.record_on_exit(name, handle_sigterm=True): [Python API]
registers a
process-exit handler that writes one benchmark record when the script
terminates. Captures wall-clock duration from the call site to exit plus
all mixin fields. Designed for SLURM jobs and batch scripts where
restructuring code around a decorator is impractical. Key behaviours:
- By default installs a SIGTERM handler (main thread only) that writes
  the record, chains to any existing SIGTERM handler, then re-delivers
  SIGTERM so the process exits with the conventional code 143 (128 + 15).
- Wraps sys.excepthook to capture unhandled exceptions into an
  exception field before the process exits.
- Adds an exit_signal field when the exit was triggered by SIGTERM.
- Falls back to writing the record to sys.stderr if the primary output
  sink raises (e.g. filesystem unmounted at exit time).
- Calling a second time on the same instance replaces the first
  registration and resets the start time.
bench.record(name) context manager: [Python API] times an arbitrary code block
and writes one record, without requiring the code to be in a named
function. All mixins, static fields, and output sinks behave identically
to the decorator form.
Exception capture: [Python API] when a benchmarked block raises — via
bench.record() or a @bench-decorated function — the record is
written before the exception propagates. An exception field is added
containing {"type": ..., "message": ...}. The exception is always
re-raised. With --iterations N, timing stops at the first exception.
MBPythonInfo mixin replaces MBPythonVersion: records a python dict
with version, prefix (sys.prefix), and executable (sys.executable),
giving a complete picture of the running interpreter in one field. MBPythonVersion
has been removed (see breaking changes above). MBPythonInfo is included
in MicroBench by default (Python API) and in the CLI default mixin set;
--no-mixin suppresses it on the CLI as usual.
MBLoadedModules mixin: captures the loaded Lmod / Environment
Modules software stack into a loaded_modules dict mapping module name
to version string (e.g. {"gcc": "12.2.0", "openmpi": "4.1.5"}). Reads
the standard LOADEDMODULES environment variable — no subprocess, no
extra dependencies. Empty dict when no modules are loaded. Included in
the CLI defaults alongside MBHostInfo, MBSlurmInfo, and MBWorkingDir.
MBWorkingDir mixin: captures the absolute path of the working
directory at benchmark time into call.working_dir. No dependencies.
Included in the CLI defaults — useful for reproducibility when comparing
results across nodes or directories.
MBGitInfo mixin: captures the repository root path, current commit
hash, branch name, and dirty flag (uncommitted changes present) via
git ≥ 2.11 on PATH. Stored in git. Set git_repo to inspect
a specific repository directory.
MBPeakMemory mixin: captures peak Python memory allocation during the
benchmarked function as call.peak_memory_bytes (bytes), using
tracemalloc from the standard library. No extra dependencies required.
MBSlurmInfo mixin: captures all SLURM_* environment variables into
a slurm dict (keys lowercased, SLURM_ prefix stripped). Empty dict
when running outside a SLURM job. Supersedes the manual
env_vars = ('SLURM_JOB_ID', ...) pattern.
MBFileHash mixin: records a cryptographic checksum of specified files
in the file_hashes field (a dict mapping path to hex digest). Defaults to
hashing sys.argv[0] — the running script. Set hash_files to an iterable
of paths to hash specific files instead. Set hash_algorithm to any
algorithm accepted by hashlib.new (default: 'sha256').
MBCgroupLimits mixin: captures the CPU quota and memory limit enforced by
the Linux cgroup filesystem. Works for SLURM jobs and Kubernetes pods (cgroup
v1 and v2). Fields in cgroups: cpu_cores_limit (float — quota ÷ period,
or null if unlimited), memory_bytes_limit (int or null if unlimited),
version (1 or 2). Returns {} on non-Linux systems or when the
cgroup filesystem is unavailable.
```
class Bench(MicroBench, MBSlurmInfo, MBCgroupLimits):
    pass
```
```
{
  "cgroups": {
    "cpu_cores_limit": 4.0,
    "memory_bytes_limit": 17179869184,
    "version": 2
  }
}
```
MBCondaPackages improvements:
- Queries the environment identified by CONDA_PREFIX (the shell's active conda
  environment) rather than sys.prefix. Falls back to sys.prefix when
  CONDA_PREFIX is not set.
- Falls back to CONDA_EXE if conda is not on PATH (common in non-interactive
  SLURM batch scripts where conda is activated but its bin/ is not on PATH).
- Replaces the separate conda_versions field with a unified conda dict
  containing name (CONDA_DEFAULT_ENV), path (CONDA_PREFIX), and
  packages (the version dict). Either of name/path may be None if
  the corresponding variable is unset. With get_results(flat=True) these
  expand to conda.name, conda.path, conda.packages.<pkg> etc.
MicroBenchBase: the core benchmarking machinery is now exposed as
MicroBenchBase (no default mixins). MicroBench inherits from both
MicroBenchBase and MBPythonInfo. Subclass MicroBenchBase directly when
you need a completely bare benchmark class with no automatic captures.
warmup parameter: pass warmup=N to run the function N times
before timing begins, priming caches or JIT compilation without affecting
results. Warmup calls are unrecorded and do not interact with the monitor
thread or capture triggers.
Multi-sink output architecture (#52): Results can now be written to
multiple destinations simultaneously by passing an outputs list to
MicroBench. Three classes make up the new output API:
- Output — abstract base class; subclass this to implement custom sinks.
- FileOutput — writes JSONL to a file path or file-like object (wraps the
  previous default behaviour).
- RedisOutput — writes to a Redis list.
- HttpOutput - New for v2 - POST each benchmark result to an HTTP/HTTPS endpoint.
The existing outfile parameter and class-level outfile attribute continue
to work as shorthand for a single FileOutput. Passing both outfile and
outputs raises ValueError.

Example — write to a file and Redis simultaneously:
```
from microbench import MicroBench, FileOutput, RedisOutput

bench = MicroBench(outputs=[
    FileOutput('/home/user/results.jsonl'),
    RedisOutput('microbench:mykey', host='redis-host', port=6379),
])
```
get_results() delegates to the first sink that supports reading back
results (FileOutput and RedisOutput both do).
get_results(format=..., flat=...): get_results now accepts two
keyword arguments.
- format='dict' (default) — returns a list of dicts; no pandas required.
- format='df' — returns a pandas DataFrame (previous default behaviour).
- flat=True — flattens nested dict fields (e.g. slurm, cgroups,
  git) into dot-notation keys (slurm.job_id, call.name). Works for both
  formats without requiring pandas.
capture_optional class attribute: set capture_optional = True on
a benchmark class to catch exceptions from capture_ and capturepost_
methods instead of aborting the benchmark call. Failures are recorded in
call.capture_errors (a list of {"method": ..., "error": ...} dicts);
the field is absent when all captures succeed. Designed for production
jobs on heterogeneous cluster nodes where optional dependencies may not
be present on every node.
python-dateutil dependency removed from LiveStream: timestamp
parsing now uses datetime.fromisoformat() from the standard library.
Remove python-dateutil from your environment if it was
only installed for microbench.

Breaking changes (vs v1.1.0)

Removed deprecated mixins: MBPythonVersion, MBHostCpuCores, and
MBHostRamTotal have been removed.
- Replace MBPythonVersion with MBPythonInfo (captures python.version,
  python.prefix, python.executable).
- Replace MBHostCpuCores and/or MBHostRamTotal with MBHostInfo, which
  now captures host.cpu_cores_logical, host.cpu_cores_physical, and
  host.ram_total automatically when psutil is installed.

Namespace restructuring of record fields: All benchmark record fields are
now grouped into top-level namespace dicts, making records self-documenting
and easier to query. The complete rename table is below.

Core fields move into mb (static config) and call (per-call data):

Old key	New key
`mb_run_id`	`mb.run_id`
`mb_version`	`mb.version`
`timestamp_tz`	`mb.timezone`
`duration_counter`	`mb.duration_counter`
`start_time`	`call.start_time`
`finish_time`	`call.finish_time`
`run_durations`	`call.durations`
`function_name`	`call.name`
`telemetry`	`call.monitor`
`env_*` (flat keys)	`env.*` (dict)
`package_versions` (from `capture_versions`)	`python.loaded_packages`

Mixin fields move to typed namespaces:

Old key	New key
`hostname`	`host.hostname`
`operating_system`	`host.os`
`cpu_cores_physical`	`host.cpu_cores_physical`
`ram_total`	`host.ram_total`
`args`	`call.args`
`kwargs`	`call.kwargs`
`return_value`	`call.return_value`
`line_profiler`	`call.line_profiler`
`package_versions` (MBGlobalPackages)	`python.loaded_packages`
`package_versions` (MBInstalledPackages)	`python.installed_packages`
`package_paths`	`python.installed_package_paths`
`nvidia_<attr>` (multiple flat dicts)	`nvidia` (list of per-GPU dicts with `uuid` key)

Unchanged namespaces: conda.

Migration: Use get_results(flat=True) to access fields via dot-notation
keys (call.name, mb.run_id, host.hostname, etc.) in pandas or scripts
without rewriting nested dict access. Alternatively, update field access to
the new nested structure: result['call']['name'], result['mb']['run_id'],
result['host']['hostname'], etc.

get_results() now returns a list of dicts by default: The default
format='dict' returns a list of plain Python dicts and requires no
dependencies. Pass format='df' to get a pandas DataFrame (previous
behaviour). Update existing callers: bench.get_results() →
bench.get_results(format='df').
telemetry renamed to monitor (#51): The background sampling thread
has been renamed throughout the API to better reflect its intent (continuous
monitoring, not data transmission).
- TelemetryThread → MonitorThread
- Class variable telemetry_interval → monitor_interval
- Class variable telemetry_timeout → monitor_timeout
- Result field bm_data['telemetry'] → bm_data['monitor']
- Internal attribute self._telemetry_thread → self._monitor_thread

MicroBenchRedis removed (#52): Use
MicroBench(outputs=[RedisOutput(...)]) instead.

Before:

from microbench import MicroBenchRedis

class RedisBench(MicroBenchRedis):
    redis_connection = {'host': 'localhost', 'port': 6379}
    redis_key = 'microbench:mykey'

bench = RedisBench()

After:

from microbench import MicroBench, RedisOutput

bench = MicroBench(outputs=[RedisOutput('microbench:mykey',
                                         host='localhost', port=6379)])

LiveStream updated for v2 record schema: field references updated from
the v1 flat schema (function_name, hostname, start_time,
finish_time) to the v2 nested schema (call.name, host.hostname,
call.start_time, call.finish_time). Records produced by microbench v1
are no longer parsed correctly by LiveStream; this is expected given the
v2 schema migration documented in the breaking changes section above.

Full list of changes

feat!: rename telemetry → monitor by @alubbock in #51
feat!: multi-sink output architecture (Output, FileOutput, RedisOutput) by @alubbock in #52
docs: add MkDocs Material documentation site by @alubbock in #54
chore(deps): update dependency python to 3.14 by @renovate[bot] in #55
feat: add warmup=N parameter to MicroBench by @alubbock in #56
feat: add MBSlurmInfo mixin by @alubbock in #57
feat: add capture_optional class attribute by @alubbock in #58
docs: restructure user guide and document LiveStream/envdiff by @alubbock in #59
docs: add citation section to README and docs by @alubbock in #60
feat: add MBPeakMemory mixin by @alubbock in #61
docs: improve clarity and reduce jargon by @alubbock in #62
feat: add MBFileHash mixin by @alubbock in #64
feat: add MBGitInfo mixin by @alubbock in #63
feat: add python -m microbench CLI by @alubbock in #65
feat: add MBLoadedModules mixin for Lmod/Environment Modules capture by @alubbock in #66
feat: stream --stdout/--stderr to terminal in real time by @alubbock in #67
feat: add bench.record() context manager and transparent exception capture by @alubbock in #68
feat: add bench.record_on_exit() for process-lifetime benchmarking by @alubbock in #69
refactor: split init.py and test_base.py into focused submodules by @alubbock in #70
feat: add async support for @bench decorator and bench.arecord() by @alubbock in #71
fix: start monitor thread at record_on_exit() call time, not exit time by @alubbock in #73
feat: add bench.time(name) sub-timing API by @alubbock in #72
feat: add --monitor-interval for CLI subprocess CPU/RAM monitoring by @alubbock in #74
feat: add MBCgroupLimits mixin by @alubbock in #75
feat!: results API — get_results(format, flat) and summary() by @alubbock in #76
feat: --mixin accepts multiple space-separated values by @alubbock in #77
feat: kebab-case mixin names and --show-mixins by @alubbock in #78
feat: autodiscover mixin CLI args (CLIArg); add --git-repo and --hash-file by @alubbock in #79
feat: add MBWorkingDir mixin and include in CLI defaults by @alubbock in #80
fix: make --all and --no-mixin mutually exclusive by @alubbock in #81
feat: microbench entry point, MBPythonInfo, MicroBenchBase, conda improvements by @alubbock in #82
feat!: restructure record fields into consistent namespaces (2.0.0) by @alubbock in #83
feat: merge MBHostCpuCores and MBHostRamTotal into MBHostInfo by @alubbock in #84
feat: add --version flag to CLI by @alubbock in #85
feat: add --timeout and --timeout-grace-period to CLI by @alubbock in #86
feat!: remove deprecated MBPythonVersion, MBHostCpuCores, MBHostRamTotal by @alubbock in #87
feat: add --nvidia-attributes and --nvidia-gpus CLI flags by @alubbock in #88
docs: document MBNvidiaSmi Python API attributes; add command-level tests by @alubbock in #89
feat: add --dry-run flag to CLI by @alubbock in #90
refactor: rename mixins.py to _mixins.py; export CLIArg from top-level package by @alubbock in #91
feat: add HttpOutput and --http-output CLI flag by @alubbock in #92
feat: add --redis-output CLI flag; group --help by category by @alubbock in #93
refactor: split god-module into core, mixins, outputs, and cli subpackages by @alubbock in #94
refactor: move tests to top-level tests/ package by @alubbock in #95
fix: address all v2.0 review findings — code bugs, docs, config, README by @alubbock in #97
docs: update README and docs by @alubbock in #98
fix: cli return code is now first non-zero value by @alubbock in #99
docs: further docs rewrites by @alubbock in #100
docs: more CLI documentation, other tweaks by @alubbock in #101
docs: fix minor issues by @alubbock in #102
docs: rework long changelog item into list by @alubbock in #103
docs: note about shell script usage with CLI by @alubbock in #104
docs: changelog fixes by @alubbock #105 in #105

Full Changelog: v1.1.0...v2.0.0

Navigation Menu

v2.0.0

Choose a tag to compare

Sorry, something went wrong.

Sorry, something went wrong.

Uh oh!

No results found

New features

Breaking changes (vs v1.1.0)

Full list of changes

Contributors

Uh oh!