Skip to content

Min time becomes measurement option - #424

Draft
oleksandr-pavlyk wants to merge 13 commits into
NVIDIA:mainfrom
oleksandr-pavlyk:min-time-is-measurement-option
Draft

Min time becomes measurement option#424
oleksandr-pavlyk wants to merge 13 commits into
NVIDIA:mainfrom
oleksandr-pavlyk:min-time-is-measurement-option

Conversation

@oleksandr-pavlyk

@oleksandr-pavlyk oleksandr-pavlyk commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

--min-time is now a part of nvbench::benchmark_base, nvbench::state.

min-time used to be stdrel-stopping-criterion parameter, but
was being used by hot measuring loop (see #408).

This PR add m_min_time data member to benchmark_base and state
classes as measurement option, same as --min-samples. Setter/getter
methods are added.

Option parser is modified to recognize --min-time <duration> as option associated
with benchmark and calls .set_min_time(duration) setter.

Measuring loops have access to min-time parameter, but delegate
what to do with measurement options (--min-time and --min-samples)
to individual stopping criteria via new stopping criteria method
bool is_elibigible_to_stop(stopping_context context).

The stopping_context argument contains the number-of-samples collected,
min-samples, total-sample-time, min-time. The method allows stopping
criteria to customize treatment of measurement options:

  • stdrel: honors both --min-time and --min-samples (same as before)
  • entropy: honors --min-samples, ignores --min-time (same as before)
  • target-samples: ignores both (intentional change)

Closes #408.

CLI option --min-time is now always recognized, irrespective of which
stopping-criterion is current during option parsing.

Python API modified to match C++.

docs/cli_help.md modified:

  • --min-time moved to measurement options section
  • each documented stopping criterion states how it takes measurement options into account

min-time used to be stdrel-stopping-criterion parameter, but
was being used by hot measuring loop (see NVIDIA#408).

This PR add m_min_time data member to benchmark_base and state
classes as measurement option, same as `--min-samples`. Setter/getter
methods are added.

Measuring loops have access to this parameter, but delegate
what to use with measurement options (--min-time and --min-samples)
to individual stopping criteria.

Stopping criteria acquired `bool is_elibigible_to_stop(stopping_context)`
method with stopping_context populated with number-of-samples collected,
min-samples, total-sample-time, min-time. The method allows stopping
criteria to customize treatment of measurement options:

  * stdrel: honors both --min-time and --min-samples
  * entropy: honors --min-samples, ignores --min-time
  * target-samples: ignores both

Closes NVIDIA#408.

CLI option `--min-time` is now always recognized, irrespective of which
stopping-criterion is current during option parsing.
@oleksandr-pavlyk

This comment was marked as outdated.

@coderabbitai

This comment was marked as outdated.

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4c3bc65b-7a82-470f-9e45-3e4fbedca044

📥 Commits

Reviewing files that changed from the base of the PR and between 5061a4f and 0e4748d.

📒 Files selected for processing (34)
  • docs/cli_help.md
  • examples/CMakeLists.txt
  • nvbench/benchmark_base.cuh
  • nvbench/benchmark_base.cxx
  • nvbench/detail/measure_cold.cu
  • nvbench/detail/measure_cold.cuh
  • nvbench/detail/measure_cpu_only.cuh
  • nvbench/detail/measure_cpu_only.cxx
  • nvbench/detail/measure_hot.cu
  • nvbench/detail/measure_hot.cuh
  • nvbench/detail/measure_timeout_warnings.cuh
  • nvbench/detail/sample_count_criterion.cuh
  • nvbench/detail/sample_count_criterion.cxx
  • nvbench/detail/stdrel_criterion.cuh
  • nvbench/detail/stdrel_criterion.cxx
  • nvbench/detail/validate_min_time.cuh
  • nvbench/json_printer.cu
  • nvbench/option_parser.cu
  • nvbench/state.cuh
  • nvbench/state.cxx
  • nvbench/stopping_criterion.cuh
  • python/cuda/bench/__init__.pyi
  • python/cuda/bench/_decorators.py
  • python/src/py_nvbench.cpp
  • python/test/test_cuda_bench.py
  • testing/benchmark.cu
  • testing/entropy_criterion.cu
  • testing/exception_safety.cu
  • testing/measure_timeout_warnings.cu
  • testing/option_parser.cu
  • testing/sample_count_criterion.cu
  • testing/state.cu
  • testing/state_generator.cu
  • testing/stdrel_criterion.cu

📝 Walkthrough

Summary by CodeRabbit

  • New Features
    • Added minimum accumulated measurement time controls (--min-time, plus C++/Python min_time/set_min_time APIs) and included min_time in JSON output.
    • Stopping criteria now use a shared stopping context to coordinate minimum samples and minimum time.
  • Documentation
    • Refreshed CLI help to clarify --min-time interaction with --stopping-criterion.
  • Bug Fixes
    • Updated timeout-warning logic and stopping eligibility so warnings reflect true minimum-time blocking.
  • Tests
    • Expanded coverage for option parsing/validation, stopping-context behavior, decorator/API wiring, and JSON emission.

Walkthrough

The change models --min-time as benchmark measurement configuration, propagates it through benchmark state and stopping contexts, updates criterion eligibility and timeout warnings, exposes Python APIs, updates documentation and examples, and adds coverage for parsing and stopping behavior.

Changes

Minimum measurement time

Layer / File(s) Summary
Configuration and stopping contract
nvbench/benchmark_base.*, nvbench/state.*, nvbench/stopping_criterion.cuh, nvbench/option_parser.cu, nvbench/json_printer.cu
Minimum-time configuration is validated, stored on benchmarks and states, parsed from --min-time, copied through generated state, included in stopping contexts, and emitted in JSON results.
Criterion and measurement execution
nvbench/detail/measure_*, nvbench/detail/*criterion.*, nvbench/detail/measure_timeout_warnings.cuh
Cold and CPU-only measurements pass sample/time context to stopping criteria; hot measurements use a separate batch target time; stdrel and sample-count define eligibility behavior; timeout warnings use criterion-aware minimum-time checks.
Public interfaces and usage
python/cuda/bench/*, python/src/py_nvbench.cpp, docs/cli_help.md, examples/CMakeLists.txt
Python benchmark/state APIs and decorators expose minimum-time configuration, CLI help describes shared semantics, and all examples receive the option.
Behavioral validation
testing/*, python/test/test_cuda_bench.py
Tests cover validation, criterion context rules, CLI parsing and propagation, timeout warnings, state generation, exception safety, Python decorators, and updated stdrel behavior.

Assessment against linked issues

Objective Addressed Explanation
Move --min-time out of stdrel criterion parameters and make it usable with other criteria [#408]

Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]

This comment was marked as resolved.

@oleksandr-pavlyk

This comment was marked as outdated.

@coderabbitai

This comment was marked as outdated.

coderabbitai[bot]

This comment was marked as resolved.

@oleksandr-pavlyk

This comment was marked as outdated.

@coderabbitai

This comment was marked as outdated.

coderabbitai[bot]

This comment was marked as resolved.

@oleksandr-pavlyk

This comment was marked as outdated.

@coderabbitai

This comment was marked as outdated.

@coderabbitai

This comment was marked as outdated.

@oleksandr-pavlyk

This comment was marked as outdated.

@coderabbitai

This comment was marked as outdated.

coderabbitai[bot]

This comment was marked as resolved.

@oleksandr-pavlyk

This comment was marked as outdated.

@coderabbitai

This comment was marked as outdated.

coderabbitai[bot]

This comment was marked as resolved.

@oleksandr-pavlyk

This comment was marked as outdated.

@coderabbitai

This comment was marked as outdated.

coderabbitai[bot]

This comment was marked as resolved.

Add a separate hot batch target time with a 100 us floor
so --min-time 0, or very small positive min-time values,
do not produce degenerate one-launch hot batches.

Keep the user-provided min-time unchanged for stopping and timeout
diagnostics; only the internal batch-size estimate uses the clamped
target.
@oleksandr-pavlyk
oleksandr-pavlyk force-pushed the min-time-is-measurement-option branch from a753dd8 to 0e4748d Compare July 22, 2026 17:25
@oleksandr-pavlyk

This comment was marked as outdated.

@coderabbitai

This comment was marked as outdated.

@oleksandr-pavlyk
oleksandr-pavlyk marked this pull request as ready for review July 22, 2026 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batched min-time is modeled as a stdrel criterion parameter, causing unintuitive CLI behavior with other stopping criteria

1 participant