Add OVPhysX sensor micro-benchmarks by AntoineRichard · Pull Request #6567 · isaac-sim/IsaacLab

AntoineRichard · 2026-07-16T16:18:42Z

Description

Adds kitless OVPhysX micro-benchmarks for contact, frame-transformer, IMU, PVA, joint-wrench, and ray-caster sensor operations.

The live workloads keep physics stepping outside timed regions and use the shared latency runner to report:

synchronized completion latency;
host submission latency;
a matched synchronized observer floor;
blocking native-read latency where the backend exposes it;
structured summary and JSON output with mean, sample standard deviation, sample count, p50, and p95.

Validation rejects missing contacts, non-finite values, invalid ray hits, or entirely zero joint wrenches. The benchmarks require the optional ovphysx runtime wheel.

Dependencies

Direct dependency: Add OVPhysX asset micro-benchmarks #6566.
Transitive dependencies: Improve benchmark sampling and synchronized timing diagnostics #6474, Promote benchmarks to library API #6564, and Fix articulation data micro-benchmarks #6565.
Satisfied prerequisite: Refactor train and play into rl subpackage #6553 is already merged.

Type of change

New feature
Documentation update

Validation

Small live smoke runs for all five scripts and both IMU/PVA modes.
Three workstation runs per sensor at 4,096 environments, 50 warm-ups, and 500 measured operations for the original workload validation.
Shared benchmark unit suite and structured-output checks after the runner refactor.
./isaaclab.sh -f
./isaaclab.sh -d

The earlier workstation measurements validate workload execution, but are not retained here as reference performance numbers because the timing/output protocol was subsequently standardized.

Checklist

I have read and understood the contribution guidelines
I have run the pre-commit checks
I have made corresponding documentation changes
My changes generate no new warnings
I have validated every workload path on OVPhysX
I have added the required changelog fragment
My name already exists in CONTRIBUTORS.md

Exclude configurable warmup frames from runtime metrics and compute throughput from total measured work over total wall time. Use longer defaults so the quick benchmark is less sensitive to startup and short sampling windows.

Add an opt-in runtime benchmark probe around the environment's SimulationContext step calls. Report host time inside and outside that boundary while preserving the original method after measurement.

Measure environment and synchronized simulation stepping across runtime, training, and play benchmarks. Store the resulting Isaac Lab overhead in benchmark schema 1.1 while leaving regular training and play entry points uninstrumented.

Require an explicit benchmark flag before wrapping environment and simulation steps. Disabled benchmarks now avoid method replacement and device synchronization while schema 1.1 continues to represent measured runs.

Delegate synchronization and elapsed-time measurement to the existing Isaac Lab Timer. Keep the interception layer focused on recording per-step samples so environment, simulation, and derived overhead metrics include standard deviation and peak values.

Record the env.step boundary in every benchmark workflow without adding explicit synchronization. Keep the nested synchronized simulation probe optional, and expose environment-step FPS in schema 1.2 so runtime, play, and training results can be compared on the same scope.

Name the opt-in instrumentation after the Isaac Lab overhead it reports instead of the implementation detail used to derive it.

Synchronize before environment and simulation timing begins so policy and action-generation work queued earlier is not attributed to Isaac Lab overhead.

Recalculate throughput standard deviation around the aggregate effective mean so benchmark MeanStd values use one consistent center.

Validate workload counts before launch and require positive environment timing samples so empty synchronized runs fail with actionable errors instead of division by zero.

Run the training and play smoke matrix without synchronized overhead measurement, retain one focused synchronized replay, and verify pending work before env.step is excluded from its metric.

Keep the first runtime step outside throughput regardless of overhead instrumentation and floor derived overhead at zero when nested wall timers cross due to measurement noise.

Rename the disputed overhead metric as a serialized synchronized step breakdown and expose explicit measurement modes. Mark every instrumented timing and rate as diagnostic, validate the additive partition, and document why outside-simulation time is not removable overhead.

Follow-up polish from review of the runtime/synchronized-timing changes; no behavior change. - Drop the redundant ``dest=``/``default=False`` on runtime.py's ``--measure_synchronized_step_breakdown`` (argparse derives both) so the flag matches the eight RL adapter scripts. - Remove the single-use ``timer_context`` alias and enter the step-timing recorder alongside ``BenchmarkMonitor`` on one line, matching the adapters' context-manager pattern. - Reorder the ``build_runtime`` Args block to signature order (``aggregate_throughput`` before ``frames_per_environment_step``). - Clarify ``MeanStd.mean``/``std``: for effective-throughput fields these are the aggregate rate and the dispersion about the effective mean, not the arithmetic sample mean and standard deviation.

Bring PR isaac-sim#6553 on top of the benchmark timing work from PR isaac-sim#6474 so the benchmark library can dispatch the new RL entrypoints directly.

Move the supported benchmark workflows into isaaclab.benchmark and expose typed in-process dispatch for runtime, startup, training, and play benchmarks. Keep deprecated script and test namespace shims so existing callers can migrate without an immediate break.

Use one device-wide synchronization per timing boundary and keep mock backend state preallocated between samples. Initialize ordering and dynamics buffers so PhysX and Newton accessors are measured without hidden skips or allocation work.

Replace per-file Python shims with lazy module proxies so legacy imports remain compatible while Git can recognize the benchmark implementation moves. Retain type forwarding stubs and document removal in Isaac Lab 3.1.

Add a warmup_steps parameter to EnvironmentStepTimingRecorder that drops the first N ``env.step()`` calls (CUDA graph capture, lazy kernel compilation) from the recorded environment-step timing. In serialized mode the skipped steps' simulation call-count and time are rolled back so the recorded aggregate stays consistent. Expose it as an opt-in ``--warmup_steps`` flag (default 0, validated non-negative, recorded in metadata) on the eight RL train/play benchmark adapters. ``runtime.py`` is unchanged: its ``--warmup_frames`` already excludes the startup step before the recorder is installed. Add recorder unit tests for the host-return and serialized exclusion paths and a parse-level test rejecting a negative ``--warmup_steps``.

Preserve the destination Warp dtype when copying mock binding data so structured pose and spatial-vector buffers can be exercised by asset data benchmarks.

Cover articulation, rigid object, and rigid object collection method and data access performance using kitless mock bindings comparable to the Newton benchmark matrix.

Apply the repository Ruff format to the newly tracked benchmark scripts and keep fill-ratio selection conditions within the line-length limit.

Measure synchronized updates, submission latency, and backend reads for contact, frame-transformer, IMU, PVA, joint-wrench, and ray-caster sensors without charging physics stepping to sensor timing.

# Conflicts: # scripts/benchmarks/rl_games/benchmark_rl_games_play.py # scripts/benchmarks/rl_games/benchmark_rl_games_train.py # scripts/benchmarks/rsl_rl/benchmark_rsl_rl_play.py # scripts/benchmarks/rsl_rl/benchmark_rsl_rl_train.py # scripts/benchmarks/runtime.py # scripts/benchmarks/sb3/benchmark_sb3_play.py # scripts/benchmarks/sb3/benchmark_sb3_train.py # scripts/benchmarks/skrl/benchmark_skrl_play.py # scripts/benchmarks/skrl/benchmark_skrl_train.py

Make runtime and startup help bypass the required task check so the unified benchmark CLI can render its full options. Cover the forwarded-argv path that exposed the parser ordering bug.

Target the public benchmark backend modules after the standalone scripts became compatibility wrappers. This keeps parser validation attached to the implementation that owns it.

Keep typed training and play requests equivalent to the CLI added by PR isaac-sim#6474. Translate warm-up steps through dispatch so callers do not need backend-specific arguments.

Scope sys.argv to the typed request while benchmark workflows run so nested AppLauncher parsing sees the same arguments as the workflow entry point. Restore the caller arguments after dispatch.

…er/ovphysx-asset-micro-benchmarks

…ovphysx-sensor-micro-benchmarks

Share synchronized latency sampling across PhysX and OVPhysX sensor workloads. Emit summary and JSON results with observer-floor, submission, completion, and native-read statistics so runs are directly comparable and failures remain visible.

greptile-apps · 2026-07-20T15:21:12Z

Too many files changed for review. (148 files found, 100 file limit)

Bypass the limit by tagging @greptile-apps to review.

AntoineRichard and others added 25 commits July 13, 2026 09:48

Improve runtime benchmark sampling

97a00b1

Exclude configurable warmup frames from runtime metrics and compute throughput from total measured work over total wall time. Use longer defaults so the quick benchmark is less sensitive to startup and short sampling windows.

Measure simulation-step host time

8d0b99d

Add an opt-in runtime benchmark probe around the environment's SimulationContext step calls. Report host time inside and outside that boundary while preserving the original method after measurement.

Report synchronized benchmark overhead

7fb24c1

Measure environment and synchronized simulation stepping across runtime, training, and play benchmarks. Store the resulting Isaac Lab overhead in benchmark schema 1.1 while leaving regular training and play entry points uninstrumented.

Make synchronized benchmark timing optional

9521321

Require an explicit benchmark flag before wrapping environment and simulation steps. Disabled benchmarks now avoid method replacement and device synchronization while schema 1.1 continues to represent measured runs.

Rename benchmark overhead flag

ee04ef9

Name the opt-in instrumentation after the Isaac Lab overhead it reports instead of the implementation detail used to derive it.

Exclude pending work from overhead timing

4a59a05

Synchronize before environment and simulation timing begins so policy and action-generation work queued earlier is not attributed to Isaac Lab overhead.

Center throughput deviation on effective mean

ab28226

Recalculate throughput standard deviation around the aggregate effective mean so benchmark MeanStd values use one consistent center.

Reject empty benchmark workloads

e648170

Validate workload counts before launch and require positive environment timing samples so empty synchronized runs fail with actionable errors instead of division by zero.

Cover optional benchmark timing paths

7a2a118

Run the training and play smoke matrix without synchronized overhead measurement, retain one focused synchronized replay, and verify pending work before env.step is excluded from its metric.

Stabilize benchmark timing diagnostics

c292852

Keep the first runtime step outside throughput regardless of overhead instrumentation and floor derived overhead at zero when nested wall timers cross due to measurement noise.

Refactor train and play into rl subpackage

35a3777

Fix changelog fragment

2e8f35a

Merge RL entrypoint refactor dependency

b763ab5

Bring PR isaac-sim#6553 on top of the benchmark timing work from PR isaac-sim#6474 so the benchmark library can dispatch the new RL entrypoints directly.

Fix articulation data micro-benchmarks

4421ee0

Use one device-wide synchronization per timing boundary and keep mock backend state preallocated between samples. Initialize ordering and dynamics buffers so PhysX and Newton accessors are measured without hidden skips or allocation work.

Centralize deprecated benchmark aliases

63092f3

Replace per-file Python shims with lazy module proxies so legacy imports remain compatible while Git can recognize the benchmark implementation moves. Retain type forwarding stubs and document removal in Isaac Lab 3.1.

Fix OVPhysX mock structured reads

08abc71

Preserve the destination Warp dtype when copying mock binding data so structured pose and spatial-vector buffers can be exercised by asset data benchmarks.

Add OVPhysX asset micro-benchmarks

095acc4

Cover articulation, rigid object, and rigid object collection method and data access performance using kitless mock bindings comparable to the Newton benchmark matrix.

Format OVPhysX asset benchmarks

a12af09

Apply the repository Ruff format to the newly tracked benchmark scripts and keep fill-ratio selection conditions within the line-length limit.

Add OVPhysX sensor micro-benchmarks

8628973

Measure synchronized updates, submission latency, and backend reads for contact, frame-transformer, IMU, PVA, joint-wrench, and ray-caster sensors without charging physics stepping to sensor timing.

github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team labels Jul 16, 2026

AntoineRichard mentioned this pull request Jul 17, 2026

Document micro-benchmark workflows #6587

Open

7 tasks

AntoineRichard added 2 commits July 17, 2026 13:31

Allow benchmark help without task

e260c36

Make runtime and startup help bypass the required task check so the unified benchmark CLI can render its full options. Cover the forwarded-argv path that exposed the parser ordering bug.

Update benchmark smoke test imports

4067c87

Target the public benchmark backend modules after the standalone scripts became compatibility wrappers. This keeps parser validation attached to the implementation that owns it.

AntoineRichard mentioned this pull request Jul 17, 2026

Rewrite benchmark user guide #6595

Open

7 tasks

AntoineRichard added 3 commits July 20, 2026 11:16

Merge commit 'fba71b97d1a' into antoiner/benchmark-library-api

ce78a6e

Expose RL benchmark warm-up in Python API

ed5eae8

Keep typed training and play requests equivalent to the CLI added by PR isaac-sim#6474. Translate warm-up steps through dispatch so callers do not need backend-specific arguments.

Fix typed benchmark launcher arguments

ae91002

Scope sys.argv to the typed request while benchmark workflows run so nested AppLauncher parsing sees the same arguments as the workflow entry point. Restore the caller arguments after dispatch.

AntoineRichard mentioned this pull request Jul 20, 2026

Rewrite advanced benchmark framework guide #6627

Open

7 tasks

AntoineRichard added 9 commits July 20, 2026 14:47

Harden method micro-benchmark sampling

b2e801c

Merge latest develop into benchmark API

27acbc5

Adopt strict articulation micro-benchmarks

54da084

Merge branch 'antoiner/fix-articulation-micro-benchmarks' into antoin…

fdb7294

…er/ovphysx-asset-micro-benchmarks

Allow zero-joint micro-benchmarks

613876d

Merge zero-joint benchmark support

3c2ca66

Normalize OVPhysX asset benchmarks

75b7fcb

Merge branch 'antoiner/ovphysx-asset-micro-benchmarks' into antoiner/…

b7a3ef7

…ovphysx-sensor-micro-benchmarks

Harden sensor micro-benchmark timing

50e4852

Share synchronized latency sampling across PhysX and OVPhysX sensor workloads. Emit summary and JSON results with observer-floor, submission, completion, and native-read statistics so runs are directly comparable and failures remain visible.

AntoineRichard mentioned this pull request Jul 20, 2026

Add Newton sensor micro-benchmarks #6568

Open

7 tasks

AntoineRichard marked this pull request as ready for review July 20, 2026 15:21

AntoineRichard requested a review from a team July 20, 2026 15:21

AntoineRichard requested review from StafaH, kellyguo11, marcodiiga, matthewtrepte, myurasov-nv, ooctipus and pv-nvidia as code owners July 20, 2026 15:21

kellyguo11 approved these changes Jul 21, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add OVPhysX sensor micro-benchmarks#6567

Add OVPhysX sensor micro-benchmarks#6567
AntoineRichard wants to merge 40 commits into
isaac-sim:developfrom
AntoineRichard:antoiner/ovphysx-sensor-micro-benchmarks

AntoineRichard commented Jul 16, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

AntoineRichard commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Dependencies

Type of change

Validation

Checklist

Uh oh!

greptile-apps Bot commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AntoineRichard commented Jul 16, 2026 •

edited

Loading