NVBench Python Library (v0.3.0)

[oleksandr-pavlyk]

These are the release notes for the cuda-bench Python package version 0.3.0, dated August 5th, 2026.

cuda-bench is in "experimental" status, meaning that its API and feature set can change quite rapidly.

New Features

• Support for using decorators to register Python benchmarks, set axes and options
• Support for using timers to control timed bracket with cuda.bench.State.exec(launchable, timer=True)
• New cuda.bench.results.BenchmarkResult class for parsing JSON output
• New features from NVBench C++ library:
  • Improved performance of stdrel and entropy stopping criteria
  • New sample-count stopping criterion with criterion parameter --target-samples <count>
  • Control for number of warm-up runs with --cold-warmup-runs <count> and --cold-max-warmup-walltime <seconds>
  • Control for batch size in batched-measurements loop with --batch-target-time <seconds>
• Updates to scripts:
  • New nvbench-compare-robust script for robust benchmark data comparison that takes bulk data into account to classify comparison as FAST, SLOW, SAME, AMBG (undecided) with diagnostics and ability to generate matched data loader for further processing or visualization
  • New nvbench-json-summary script to recreate Markdown table generated by NVBench-instrumented script from saved JSON output
  • All scripts load third-party dependencies lazily and report actionable error message upon failure to import
• cuda-bench packages NVBench library installation prefix, including headers, libraries and CMake scripts. Location of the prefix can be obtained via executable module: python -m cuda.bench.nvbench --prefix

Notable Bug Fixes

• Issue with interruptibility of Python benchmark scripts was fixed
• Issue with editable install of the package was fixed

---

What's Changed

• Update python examples by @oleksandr-pavlyk in Update python examples #331
• Correct description/hint entries for summaries with name "Noise" by @oleksandr-pavlyk in Correct description/hint entries for summaries with name "Noise" #335
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci[bot] in [pre-commit.ci] pre-commit autoupdate #333
• Modularize color handling by @oleksandr-pavlyk in Modularize color handling #336
• Port changes to the package from Add scaffolding to build C++/Python docs #323 by @oleksandr-pavlyk in Port changes to the package from #323 #337
• Correct Python API signature of State.get_axis_values_as_strings by @oleksandr-pavlyk in Correct Python API signature of State.get_axis_values_as_strings #346
• Correct cuda cccl examples per change in api by @oleksandr-pavlyk in Correct cuda cccl examples per change in api #353
• Create directories for output files by @oleksandr-pavlyk in Create directories for output files #360
• Add .coderabbit.yaml following in footsteps of CCCL by @oleksandr-pavlyk in Add .coderabbit.yaml following in footsteps of CCCL #359
• Implement CLI option to control warmups for cold measurements by @oleksandr-pavlyk in Implement CLI option to control warmups for cold measurements #339
• Make nvbench::main exported cmake target map to static library by @oleksandr-pavlyk in Make nvbench::main exported cmake target map to static library #350
• Enable building of NVBench on Windows by @oleksandr-pavlyk in Enable building of NVBench on Windows #362
• Provide BenchmarkResult class for parsing JSON output of NVBench-instrumented benchmarks by @oleksandr-pavlyk in Provide BenchmarkResult class for parsing JSON output of NVBench-instrumented benchmarks #356
• Implement decorators to register benchmarks add axis and options by @oleksandr-pavlyk in Implement decorators to register benchmarks add axis and options #347
• Implement Timer, and support State.exec(fn, timer=True) by @oleksandr-pavlyk in Implement Timer, and support State.exec(fn, timer=True) #364
• Fix exception safety by @oleksandr-pavlyk in Fix exception safety #358
• Add stopping criterion sample count by @oleksandr-pavlyk in Add stopping criterion sample count #341
• Add python api for cold warmup parameters by @oleksandr-pavlyk in Add python api for cold warmup parameters #363
• Fix docutil error when building docs by @oleksandr-pavlyk in Fix docutil error when building docs #365
• Add Windows support by @mfranzrebsal in Add Windows support #354
• nvbench::cpu_timer changed to use steady_clock by @oleksandr-pavlyk in nvbench::cpu_timer changed to use steady_clock #371
• include cleanup across nvbench/ by @oleksandr-pavlyk in include cleanup across nvbench/ #377
• docs: separate measurement options from stopping criteria by @omribz156 in docs: separate measurement options from stopping criteria #373
• Reduce stdrel criterion complexity and ensure termination by @oleksandr-pavlyk in Reduce stdrel criterion complexity and ensure termination #374
• Updated devcontainer imageset to 26.08 by @oleksandr-pavlyk in Updated devcontainer imageset to 26.08 #381
• Remove unused python_wheel section in ci/matrix by @oleksandr-pavlyk in Remove unused python_wheel section in ci/matrix #382
• Introduce robust metrics by @oleksandr-pavlyk in Introduce robust metrics #379
• Build and test cuda-bench wheels for Python 3.10-3.14 by @oleksandr-pavlyk in Build and test cuda-bench wheels for Python 3.10-3.14 #380
• Rename IR to IQR in */ir/absolute and */ir/relative tags by @oleksandr-pavlyk in Rename IR to IQR in */ir/absolute and */ir/relative tags #385
• Use UV to build Python wheels by @oleksandr-pavlyk in Use UV to build Python wheels #400
• Avoid std::back_insert_iterator of fmt::memory_buffer in fmt::format by @oleksandr-pavlyk in Avoid std::back_insert_iterator of fmt::memory_buffer in fmt::format #397
• Update NVbench to use fmt 12.2 by @oleksandr-pavlyk in Update NVbench to use fmt 12.2 #398
• Corrections for python/README.md by @oleksandr-pavlyk in Corrections for python/README.md #401
• Implemented the NVBench PR review workflow. by @oleksandr-pavlyk in Implemented the NVBench PR review workflow. #403
• Harden quartile statistics and measurement timeout handling by @oleksandr-pavlyk in Harden quartile statistics and measurement timeout handling #392
• Store jsonbin sidecar paths relative to JSON output file location by @oleksandr-pavlyk in Store jsonbin sidecar paths relative to JSON output file location #405
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci[bot] in [pre-commit.ci] pre-commit autoupdate #409
• Add Python 3.14 to classifiers of project section in TOML file by @oleksandr-pavlyk in Add Python 3.14 to classifiers of project section in TOML file #413
• Fix cuda_timer move ownership by @fallintoplace in Fix cuda_timer move ownership #418
• Readability improvement by using device_clock alias by @oleksandr-pavlyk in Readability improvement by using device_clock alias #419
• Bump CTK 13.2->13.3 by @oleksandr-pavlyk in Bump CTK 13.2->13.3 #420
• Reject --jsonbin stdout or --jsonbin stderr by @oleksandr-pavlyk in Reject --jsonbin stdout or --jsonbin stderr #423
• Avoid redefining existing preprocessor variable in version.cuh by @oleksandr-pavlyk in Avoid redefining existing preprocessor variable in version.cuh #428
• Made some classes immovable, in addition to already being not-copyable by @oleksandr-pavlyk in Made some classes immovable, in addition to already being not-copyable #422
• Cuda bench package headers cmake and main by @oleksandr-pavlyk in Cuda bench package headers cmake and main #417
• Fix editable install of cuda-bench by @oleksandr-pavlyk in Fix editable install of cuda-bench #430
• Evolution of nvbench compare that uses bulk data by @oleksandr-pavlyk in Evolution of nvbench compare that uses bulk data #386
• Fix GCC 14 maybe-uninitialized warning by @PointKernel in Fix GCC 14 maybe-uninitialized warning #434
• Fix test failure for nvbench tooling deps in editable install by @oleksandr-pavlyk in Fix test failure for nvbench tooling deps in editable install #435
• Destroy CUDA stream after timing is finished by @upsj-nv in Destroy CUDA stream after timing is finished #438
• Changed plot title in nvbench_compare.py by @oleksandr-pavlyk in Changed plot title in nvbench_compare.py #432
• Implement cuda.bench.State.set_stream(provider) by @oleksandr-pavlyk in Implement cuda.bench.State.set_stream(provider) #443
• Workflow for executing examples by @oleksandr-pavlyk in Workflow for executing examples #445
• Add nvbench-compare-robust --plot-output/--plot-along-output options by @oleksandr-pavlyk in Add nvbench-compare-robust --plot-output/--plot-along-output options #433
• Fix duplicate fmt ALIAS targets when fmt is already provided by @igorpeshansky in Fix duplicate fmt ALIAS targets when fmt is already provided #447
• Add CLI option to set target-time parameter for batch measurement loop by @oleksandr-pavlyk in Add CLI option to set target-time parameter for batch measurement loop #439
• Bump torch from 2.11.0 to 2.13.0 in /python/examples/requirements by @dependabot[bot] in Bump torch from 2.11.0 to 2.13.0 in /python/examples/requirements #446
• print_constraints_candidate return status of 1 should not fail job by @oleksandr-pavlyk in print_constraints_candidate return status of 1 should not fail job #450

New Contributors

• @mfranzrebsal made their first contribution in Add Windows support #354
• @omribz156 made their first contribution in docs: separate measurement options from stopping criteria #373
• @fallintoplace made their first contribution in Fix cuda_timer move ownership #418
• @upsj-nv made their first contribution in Destroy CUDA stream after timing is finished #438
• @igorpeshansky made their first contribution in Fix duplicate fmt ALIAS targets when fmt is already provided #447
• @dependabot[bot] made their first contribution in Bump torch from 2.11.0 to 2.13.0 in /python/examples/requirements #446

Full Changelog: python-0.2.1...python-0.3.0

---

This discussion was created from the release NVBench Python Library (v0.3.0).