Add OVPhysX asset micro-benchmarks by AntoineRichard · Pull Request #6566 · isaac-sim/IsaacLab

AntoineRichard · 2026-07-16T16:06:14Z

Description

Adds kitless OVPhysX method and data-property micro-benchmarks for articulations, rigid objects, and rigid object collections. The method matrix covers Torch list/tensor index paths and native Warp mask paths. Data benchmarks refresh cached properties by advancing timestamps without charging random-data allocation to accessor timing.

These micro-benchmarks isolate Isaac Lab wrapper and backend-adapter overhead. Benchmark-mode mocks reuse device-resident read sources and skip mock-only NumPy write scatter, so native OVPhysX binding and solver work is deliberately outside the measurement. Fill-ratio sensitivity therefore remains outside this PR because it requires a production binding to be meaningful.

The validation also exposed and fixed a mock binding defect: structured Warp destinations now preserve their dtype instead of being copied through a scalar float temporary.

The benchmarks require the optional ovphysx runtime wheel. Current workstation validation used ovphysx-0.5.2+head.f62c22207c. Sensor workloads remain in the following PR.

Dependencies

Direct dependency: Fix articulation data micro-benchmarks #6565.
Transitive dependencies: Improve benchmark sampling and synchronized timing diagnostics #6474 and Promote benchmarks to library API #6564.
Satisfied prerequisite: Refactor train and play into rl subpackage #6553 is already merged.

Type of change

Bug fix
New feature
Documentation update

Validation

./isaaclab.sh -f
./isaaclab.sh -d (build completed)
./isaaclab.sh -p -m pytest source/isaaclab_ovphysx/test/assets/test_articulation_helpers.py -q (5 passed)
Verified the structured-dtype regression test fails without the fix and passes with it.
Verified invalid --mode values fail during argument parsing.
Workstation CUDA method smokes at 747b63f238e: articulation (44), rigid object (15), rigid object collection (15).
Workstation CUDA data smokes at 747b63f238e: articulation (59), rigid object (40), rigid object collection (78).
All six workstation workflows completed without skips or tracebacks.

Checklist

I have read and understood the contribution guidelines
I have run the pre-commit checks
I have made corresponding documentation changes
My changes generate no new warnings
I have validated the benchmark paths on OVPhysX
I have added the required changelog fragment
My name already exists in CONTRIBUTORS.md

Exclude configurable warmup frames from runtime metrics and compute throughput from total measured work over total wall time. Use longer defaults so the quick benchmark is less sensitive to startup and short sampling windows.

Add an opt-in runtime benchmark probe around the environment's SimulationContext step calls. Report host time inside and outside that boundary while preserving the original method after measurement.

Measure environment and synchronized simulation stepping across runtime, training, and play benchmarks. Store the resulting Isaac Lab overhead in benchmark schema 1.1 while leaving regular training and play entry points uninstrumented.

Require an explicit benchmark flag before wrapping environment and simulation steps. Disabled benchmarks now avoid method replacement and device synchronization while schema 1.1 continues to represent measured runs.

Delegate synchronization and elapsed-time measurement to the existing Isaac Lab Timer. Keep the interception layer focused on recording per-step samples so environment, simulation, and derived overhead metrics include standard deviation and peak values.

Record the env.step boundary in every benchmark workflow without adding explicit synchronization. Keep the nested synchronized simulation probe optional, and expose environment-step FPS in schema 1.2 so runtime, play, and training results can be compared on the same scope.

Name the opt-in instrumentation after the Isaac Lab overhead it reports instead of the implementation detail used to derive it.

Synchronize before environment and simulation timing begins so policy and action-generation work queued earlier is not attributed to Isaac Lab overhead.

Recalculate throughput standard deviation around the aggregate effective mean so benchmark MeanStd values use one consistent center.

Validate workload counts before launch and require positive environment timing samples so empty synchronized runs fail with actionable errors instead of division by zero.

Run the training and play smoke matrix without synchronized overhead measurement, retain one focused synchronized replay, and verify pending work before env.step is excluded from its metric.

Keep the first runtime step outside throughput regardless of overhead instrumentation and floor derived overhead at zero when nested wall timers cross due to measurement noise.

Rename the disputed overhead metric as a serialized synchronized step breakdown and expose explicit measurement modes. Mark every instrumented timing and rate as diagnostic, validate the additive partition, and document why outside-simulation time is not removable overhead.

Follow-up polish from review of the runtime/synchronized-timing changes; no behavior change. - Drop the redundant ``dest=``/``default=False`` on runtime.py's ``--measure_synchronized_step_breakdown`` (argparse derives both) so the flag matches the eight RL adapter scripts. - Remove the single-use ``timer_context`` alias and enter the step-timing recorder alongside ``BenchmarkMonitor`` on one line, matching the adapters' context-manager pattern. - Reorder the ``build_runtime`` Args block to signature order (``aggregate_throughput`` before ``frames_per_environment_step``). - Clarify ``MeanStd.mean``/``std``: for effective-throughput fields these are the aggregate rate and the dispersion about the effective mean, not the arithmetic sample mean and standard deviation.

Bring PR isaac-sim#6553 on top of the benchmark timing work from PR isaac-sim#6474 so the benchmark library can dispatch the new RL entrypoints directly.

Move the supported benchmark workflows into isaaclab.benchmark and expose typed in-process dispatch for runtime, startup, training, and play benchmarks. Keep deprecated script and test namespace shims so existing callers can migrate without an immediate break.

Replace per-file Python shims with lazy module proxies so legacy imports remain compatible while Git can recognize the benchmark implementation moves. Retain type forwarding stubs and document removal in Isaac Lab 3.1.

Add a warmup_steps parameter to EnvironmentStepTimingRecorder that drops the first N ``env.step()`` calls (CUDA graph capture, lazy kernel compilation) from the recorded environment-step timing. In serialized mode the skipped steps' simulation call-count and time are rolled back so the recorded aggregate stays consistent. Expose it as an opt-in ``--warmup_steps`` flag (default 0, validated non-negative, recorded in metadata) on the eight RL train/play benchmark adapters. ``runtime.py`` is unchanged: its ``--warmup_frames`` already excludes the startup step before the recorder is installed. Add recorder unit tests for the host-return and serialized exclusion paths and a parse-level test rejecting a negative ``--warmup_steps``.

AntoineRichard · 2026-07-16T16:10:52Z

Workstation measurement follow-up (ovphysx-0.5.2+head.f62c22207c, RTX workstation, 4,096 instances, 10 warmups, 100 measured iterations, 3 independent runs):

Representative operation	Mean of run means (us)	Between-run std (us)
Articulation `write_joint_state_to_sim` / tensor indices	981.24	3.81
Articulation `write_joint_state_to_sim_mask`	627.12	32.17
Rigid object `write_root_link_pose_to_sim` / tensor indices	409.23	43.05
Rigid object `write_root_link_pose_to_sim_mask`	421.41	40.02
Collection `write_body_link_pose_to_sim` / tensor indices	488.18	1.24
Collection `write_body_link_pose_to_sim_mask`	756.00	71.63
Articulation data `body_link_pose_w`	299.98	2.52
Rigid object data `body_link_pose_w`	227.18	83.55
Collection data `body_link_pose_w`	286.45	1.82

End-to-end per-run durations were 26 s for articulation methods, 8-9 s for rigid-object methods, 10-11 s for collection methods, and 2-5 s per data suite. All 18 runs completed with zero skips and zero tracebacks. Raw JSON and logs are retained on the workstation under /home/antoiner/benchmarks/results/ovphysx_asset_microbench_20260716.

Make runtime and startup help bypass the required task check so the unified benchmark CLI can render its full options. Cover the forwarded-argv path that exposed the parser ordering bug.

Target the public benchmark backend modules after the standalone scripts became compatibility wrappers. This keeps parser validation attached to the implementation that owns it.

greptile-apps · 2026-07-20T15:21:10Z

Too many files changed for review. (133 files found, 100 file limit)

Bypass the limit by tagging @greptile-apps to review.

Keep typed training and play requests equivalent to the CLI added by PR isaac-sim#6474. Translate warm-up steps through dispatch so callers do not need backend-specific arguments.

Scope sys.argv to the typed request while benchmark workflows run so nested AppLauncher parsing sees the same arguments as the workflow entry point. Restore the caller arguments after dispatch.

Restore warm-up handling after rebasing the benchmark adapters. Preserve caller backend state for in-process runs, expose workflow-specific result types, fail clearly when schema output has no bundle, and complete the public API documentation.

Keep mock state buffers stable so timed samples measure accessor work instead of random-data allocation or mock-side copies. Initialize Newton dynamics metadata and make unsupported or deprecated properties explicit.

Benchmark articulation, rigid-object, and collection wrapper and data paths using the optional kitless OVPhysX runtime. Keep mock storage work outside timed operations so results reflect Isaac Lab overhead.

AntoineRichard and others added 20 commits July 13, 2026 09:48

Improve runtime benchmark sampling

97a00b1

Exclude configurable warmup frames from runtime metrics and compute throughput from total measured work over total wall time. Use longer defaults so the quick benchmark is less sensitive to startup and short sampling windows.

Measure simulation-step host time

8d0b99d

Add an opt-in runtime benchmark probe around the environment's SimulationContext step calls. Report host time inside and outside that boundary while preserving the original method after measurement.

Report synchronized benchmark overhead

7fb24c1

Measure environment and synchronized simulation stepping across runtime, training, and play benchmarks. Store the resulting Isaac Lab overhead in benchmark schema 1.1 while leaving regular training and play entry points uninstrumented.

Make synchronized benchmark timing optional

9521321

Require an explicit benchmark flag before wrapping environment and simulation steps. Disabled benchmarks now avoid method replacement and device synchronization while schema 1.1 continues to represent measured runs.

Rename benchmark overhead flag

ee04ef9

Name the opt-in instrumentation after the Isaac Lab overhead it reports instead of the implementation detail used to derive it.

Exclude pending work from overhead timing

4a59a05

Synchronize before environment and simulation timing begins so policy and action-generation work queued earlier is not attributed to Isaac Lab overhead.

Center throughput deviation on effective mean

ab28226

Recalculate throughput standard deviation around the aggregate effective mean so benchmark MeanStd values use one consistent center.

Reject empty benchmark workloads

e648170

Validate workload counts before launch and require positive environment timing samples so empty synchronized runs fail with actionable errors instead of division by zero.

Cover optional benchmark timing paths

7a2a118

Run the training and play smoke matrix without synchronized overhead measurement, retain one focused synchronized replay, and verify pending work before env.step is excluded from its metric.

Stabilize benchmark timing diagnostics

c292852

Keep the first runtime step outside throughput regardless of overhead instrumentation and floor derived overhead at zero when nested wall timers cross due to measurement noise.

Refactor train and play into rl subpackage

35a3777

Fix changelog fragment

2e8f35a

Merge RL entrypoint refactor dependency

b763ab5

Bring PR isaac-sim#6553 on top of the benchmark timing work from PR isaac-sim#6474 so the benchmark library can dispatch the new RL entrypoints directly.

Centralize deprecated benchmark aliases

63092f3

Replace per-file Python shims with lazy module proxies so legacy imports remain compatible while Git can recognize the benchmark implementation moves. Retain type forwarding stubs and document removal in Isaac Lab 3.1.

github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team labels Jul 16, 2026

This was referenced Jul 16, 2026

Add OVPhysX sensor micro-benchmarks #6567

Open

Document micro-benchmark workflows #6587

Open

AntoineRichard added 2 commits July 17, 2026 13:43

Allow benchmark help without task

e260c36

Make runtime and startup help bypass the required task check so the unified benchmark CLI can render its full options. Cover the forwarded-argv path that exposed the parser ordering bug.

Update benchmark smoke test imports

4067c87

Target the public benchmark backend modules after the standalone scripts became compatibility wrappers. This keeps parser validation attached to the implementation that owns it.

This was referenced Jul 17, 2026

Rewrite benchmark user guide #6595

Open

Rewrite advanced benchmark framework guide #6627

Open

Add Newton sensor micro-benchmarks #6568

Open

AntoineRichard marked this pull request as ready for review July 20, 2026 15:21

AntoineRichard requested a review from a team July 20, 2026 15:21

AntoineRichard requested review from StafaH, kellyguo11, marcodiiga, matthewtrepte, myurasov-nv, ooctipus and pv-nvidia as code owners July 20, 2026 15:21

AntoineRichard added 11 commits July 20, 2026 18:14

Fix benchmark timing validation

6594704

Merge latest benchmark warm-up changes

e262b31

Merge commit 'fba71b97d1a' into antoiner/benchmark-library-api

7b54533

Expose RL benchmark warm-up in Python API

46ef7b5

Keep typed training and play requests equivalent to the CLI added by PR isaac-sim#6474. Translate warm-up steps through dispatch so callers do not need backend-specific arguments.

Fix typed benchmark launcher arguments

5f39662

Scope sys.argv to the typed request while benchmark workflows run so nested AppLauncher parsing sees the same arguments as the workflow entry point. Restore the caller arguments after dispatch.

Harden method micro-benchmark sampling

07f3fb9

Merge latest develop into benchmark API

5de8246

Allow zero-joint micro-benchmarks

17c5a75

Harden benchmark library lifecycle

b47d375

Restore warm-up handling after rebasing the benchmark adapters. Preserve caller backend state for in-process runs, expose workflow-specific result types, fail clearly when schema output has no bundle, and complete the public API documentation.

Fix articulation data micro-benchmarks

b9a264d

Keep mock state buffers stable so timed samples measure accessor work instead of random-data allocation or mock-side copies. Initialize Newton dynamics metadata and make unsupported or deprecated properties explicit.

Add OVPhysX asset micro-benchmarks

747b63f

Benchmark articulation, rigid-object, and collection wrapper and data paths using the optional kitless OVPhysX runtime. Keep mock storage work outside timed operations so results reflect Isaac Lab overhead.

AntoineRichard force-pushed the antoiner/ovphysx-asset-micro-benchmarks branch from 75b7fcb to 747b63f Compare July 20, 2026 20:28

kellyguo11 approved these changes Jul 21, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add OVPhysX asset micro-benchmarks#6566

Add OVPhysX asset micro-benchmarks#6566
AntoineRichard wants to merge 33 commits into
isaac-sim:developfrom
AntoineRichard:antoiner/ovphysx-asset-micro-benchmarks

AntoineRichard commented Jul 16, 2026 •

edited

Loading

Uh oh!

AntoineRichard commented Jul 16, 2026

Uh oh!

greptile-apps Bot commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

AntoineRichard commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Dependencies

Type of change

Validation

Checklist

Uh oh!

AntoineRichard commented Jul 16, 2026

Uh oh!

greptile-apps Bot commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AntoineRichard commented Jul 16, 2026 •

edited

Loading