Fix articulation data micro-benchmarks by AntoineRichard · Pull Request #6565 · isaac-sim/IsaacLab

AntoineRichard · 2026-07-16T15:03:29Z

Description

Fixes articulation-data micro-benchmarks uncovered while evaluating cached read paths.

The previous harness regenerated large random buffers before every sample, modeled stable PhysX reads as mock-side clones, and silently skipped dynamics accessors whose mock backend data was incomplete. Those effects made accessor latency and total benchmark runtime misleading.

This change:

synchronizes the configured CUDA device once per timing boundary through the shared runner;
keeps backend mock state preallocated and invalidates timestamps between cold-read samples;
models PhysX state, mass-property, and dynamics reads with stable buffers;
initializes Newton model dimensions, dynamics stubs, and post-resolution ordering state;
documents that Newton dynamics stubs isolate Isaac Lab accessor and gather overhead;
excludes Newton gravity compensation and deprecated properties;
supports zero-joint workloads through the shared public benchmark runner introduced by the preceding PR.

This PR changes only benchmark infrastructure, mock fidelity, and workloads. It does not contain articulation ordering or launch-caching implementation.

Dependencies

Direct dependency: Promote benchmarks to library API #6564.
Transitive dependency: Improve benchmark sampling and synchronized timing diagnostics #6474.
Satisfied prerequisite: Refactor train and play into rl subpackage #6553 is already merged.

Type of change

Bug fix

Validation

Stable PhysX mass/inertia view regression: failed for both getters before the fix and passes after it.
Focused tests: 58 passed across the PhysX Warp mock and shared method benchmark runner.
Local CUDA smokes: PhysX reports 65 properties; Newton reports 64 after excluding default_root_state; zero skips and zero tracebacks.
Zero-joint CUDA smokes pass for both backends.
./isaaclab.sh -f
./isaaclab.sh -p tools/changelog/cli.py check develop

Checklist

I have read and understood the contribution guidelines
I have run the pre-commit checks
My changes generate no new warnings
I have added regression coverage
I have added the required changelog fragments
My name already exists in CONTRIBUTORS.md

Exclude configurable warmup frames from runtime metrics and compute throughput from total measured work over total wall time. Use longer defaults so the quick benchmark is less sensitive to startup and short sampling windows.

Add an opt-in runtime benchmark probe around the environment's SimulationContext step calls. Report host time inside and outside that boundary while preserving the original method after measurement.

Measure environment and synchronized simulation stepping across runtime, training, and play benchmarks. Store the resulting Isaac Lab overhead in benchmark schema 1.1 while leaving regular training and play entry points uninstrumented.

Require an explicit benchmark flag before wrapping environment and simulation steps. Disabled benchmarks now avoid method replacement and device synchronization while schema 1.1 continues to represent measured runs.

Delegate synchronization and elapsed-time measurement to the existing Isaac Lab Timer. Keep the interception layer focused on recording per-step samples so environment, simulation, and derived overhead metrics include standard deviation and peak values.

Record the env.step boundary in every benchmark workflow without adding explicit synchronization. Keep the nested synchronized simulation probe optional, and expose environment-step FPS in schema 1.2 so runtime, play, and training results can be compared on the same scope.

Name the opt-in instrumentation after the Isaac Lab overhead it reports instead of the implementation detail used to derive it.

Synchronize before environment and simulation timing begins so policy and action-generation work queued earlier is not attributed to Isaac Lab overhead.

Recalculate throughput standard deviation around the aggregate effective mean so benchmark MeanStd values use one consistent center.

Validate workload counts before launch and require positive environment timing samples so empty synchronized runs fail with actionable errors instead of division by zero.

Run the training and play smoke matrix without synchronized overhead measurement, retain one focused synchronized replay, and verify pending work before env.step is excluded from its metric.

Keep the first runtime step outside throughput regardless of overhead instrumentation and floor derived overhead at zero when nested wall timers cross due to measurement noise.

Rename the disputed overhead metric as a serialized synchronized step breakdown and expose explicit measurement modes. Mark every instrumented timing and rate as diagnostic, validate the additive partition, and document why outside-simulation time is not removable overhead.

Follow-up polish from review of the runtime/synchronized-timing changes; no behavior change. - Drop the redundant ``dest=``/``default=False`` on runtime.py's ``--measure_synchronized_step_breakdown`` (argparse derives both) so the flag matches the eight RL adapter scripts. - Remove the single-use ``timer_context`` alias and enter the step-timing recorder alongside ``BenchmarkMonitor`` on one line, matching the adapters' context-manager pattern. - Reorder the ``build_runtime`` Args block to signature order (``aggregate_throughput`` before ``frames_per_environment_step``). - Clarify ``MeanStd.mean``/``std``: for effective-throughput fields these are the aggregate rate and the dispersion about the effective mean, not the arithmetic sample mean and standard deviation.

Bring PR isaac-sim#6553 on top of the benchmark timing work from PR isaac-sim#6474 so the benchmark library can dispatch the new RL entrypoints directly.

Move the supported benchmark workflows into isaaclab.benchmark and expose typed in-process dispatch for runtime, startup, training, and play benchmarks. Keep deprecated script and test namespace shims so existing callers can migrate without an immediate break.

Replace per-file Python shims with lazy module proxies so legacy imports remain compatible while Git can recognize the benchmark implementation moves. Retain type forwarding stubs and document removal in Isaac Lab 3.1.

Add a warmup_steps parameter to EnvironmentStepTimingRecorder that drops the first N ``env.step()`` calls (CUDA graph capture, lazy kernel compilation) from the recorded environment-step timing. In serialized mode the skipped steps' simulation call-count and time are rolled back so the recorded aggregate stays consistent. Expose it as an opt-in ``--warmup_steps`` flag (default 0, validated non-negative, recorded in metadata) on the eight RL train/play benchmark adapters. ``runtime.py`` is unchanged: its ``--warmup_frames`` already excludes the startup step before the recorder is installed. Add recorder unit tests for the host-return and serialized exclusion paths and a parse-level test rejecting a negative ``--warmup_steps``.

Make runtime and startup help bypass the required task check so the unified benchmark CLI can render its full options. Cover the forwarded-argv path that exposed the parser ordering bug.

Target the public benchmark backend modules after the standalone scripts became compatibility wrappers. This keeps parser validation attached to the implementation that owns it.

greptile-apps · 2026-07-20T15:21:09Z

Too many files changed for review. (124 files found, 100 file limit)

Bypass the limit by tagging @greptile-apps to review.

Keep typed training and play requests equivalent to the CLI added by PR isaac-sim#6474. Translate warm-up steps through dispatch so callers do not need backend-specific arguments.

Scope sys.argv to the typed request while benchmark workflows run so nested AppLauncher parsing sees the same arguments as the workflow entry point. Restore the caller arguments after dispatch.

Restore warm-up handling after rebasing the benchmark adapters. Preserve caller backend state for in-process runs, expose workflow-specific result types, fail clearly when schema output has no bundle, and complete the public API documentation.

Keep mock state buffers stable so timed samples measure accessor work instead of random-data allocation or mock-side copies. Initialize Newton dynamics metadata and make unsupported or deprecated properties explicit.

AntoineRichard and others added 18 commits July 13, 2026 09:48

Improve runtime benchmark sampling

97a00b1

Exclude configurable warmup frames from runtime metrics and compute throughput from total measured work over total wall time. Use longer defaults so the quick benchmark is less sensitive to startup and short sampling windows.

Measure simulation-step host time

8d0b99d

Add an opt-in runtime benchmark probe around the environment's SimulationContext step calls. Report host time inside and outside that boundary while preserving the original method after measurement.

Report synchronized benchmark overhead

7fb24c1

Measure environment and synchronized simulation stepping across runtime, training, and play benchmarks. Store the resulting Isaac Lab overhead in benchmark schema 1.1 while leaving regular training and play entry points uninstrumented.

Make synchronized benchmark timing optional

9521321

Require an explicit benchmark flag before wrapping environment and simulation steps. Disabled benchmarks now avoid method replacement and device synchronization while schema 1.1 continues to represent measured runs.

Rename benchmark overhead flag

ee04ef9

Name the opt-in instrumentation after the Isaac Lab overhead it reports instead of the implementation detail used to derive it.

Exclude pending work from overhead timing

4a59a05

Synchronize before environment and simulation timing begins so policy and action-generation work queued earlier is not attributed to Isaac Lab overhead.

Center throughput deviation on effective mean

ab28226

Recalculate throughput standard deviation around the aggregate effective mean so benchmark MeanStd values use one consistent center.

Reject empty benchmark workloads

e648170

Validate workload counts before launch and require positive environment timing samples so empty synchronized runs fail with actionable errors instead of division by zero.

Cover optional benchmark timing paths

7a2a118

Run the training and play smoke matrix without synchronized overhead measurement, retain one focused synchronized replay, and verify pending work before env.step is excluded from its metric.

Stabilize benchmark timing diagnostics

c292852

Keep the first runtime step outside throughput regardless of overhead instrumentation and floor derived overhead at zero when nested wall timers cross due to measurement noise.

Refactor train and play into rl subpackage

35a3777

Fix changelog fragment

2e8f35a

Merge RL entrypoint refactor dependency

b763ab5

Bring PR isaac-sim#6553 on top of the benchmark timing work from PR isaac-sim#6474 so the benchmark library can dispatch the new RL entrypoints directly.

github-actions Bot added bug Something isn't working isaac-lab Related to Isaac Lab team labels Jul 16, 2026

AntoineRichard added 2 commits July 16, 2026 17:11

Centralize deprecated benchmark aliases

63092f3

Replace per-file Python shims with lazy module proxies so legacy imports remain compatible while Git can recognize the benchmark implementation moves. Retain type forwarding stubs and document removal in Isaac Lab 3.1.

This was referenced Jul 16, 2026

Add OVPhysX asset micro-benchmarks #6566

Open

Add OVPhysX sensor micro-benchmarks #6567

Open

Document micro-benchmark workflows #6587

Open

AntoineRichard added 2 commits July 17, 2026 13:43

Allow benchmark help without task

e260c36

Make runtime and startup help bypass the required task check so the unified benchmark CLI can render its full options. Cover the forwarded-argv path that exposed the parser ordering bug.

Update benchmark smoke test imports

4067c87

Target the public benchmark backend modules after the standalone scripts became compatibility wrappers. This keeps parser validation attached to the implementation that owns it.

This was referenced Jul 17, 2026

Rewrite benchmark user guide #6595

Open

Rewrite advanced benchmark framework guide #6627

Open

github-actions Bot added the documentation Improvements or additions to documentation label Jul 20, 2026

AntoineRichard mentioned this pull request Jul 20, 2026

Add Newton sensor micro-benchmarks #6568

Open

7 tasks

AntoineRichard marked this pull request as ready for review July 20, 2026 15:21

AntoineRichard requested a review from a team July 20, 2026 15:21

AntoineRichard requested review from StafaH, kellyguo11, matthewtrepte, myurasov-nv, ooctipus and pv-nvidia as code owners July 20, 2026 15:21

AntoineRichard added 10 commits July 20, 2026 18:14

Fix benchmark timing validation

6594704

Merge latest benchmark warm-up changes

e262b31

Merge commit 'fba71b97d1a' into antoiner/benchmark-library-api

7b54533

Expose RL benchmark warm-up in Python API

46ef7b5

Keep typed training and play requests equivalent to the CLI added by PR isaac-sim#6474. Translate warm-up steps through dispatch so callers do not need backend-specific arguments.

Fix typed benchmark launcher arguments

5f39662

Scope sys.argv to the typed request while benchmark workflows run so nested AppLauncher parsing sees the same arguments as the workflow entry point. Restore the caller arguments after dispatch.

Harden method micro-benchmark sampling

07f3fb9

Merge latest develop into benchmark API

5de8246

Allow zero-joint micro-benchmarks

17c5a75

Harden benchmark library lifecycle

b47d375

Restore warm-up handling after rebasing the benchmark adapters. Preserve caller backend state for in-process runs, expose workflow-specific result types, fail clearly when schema output has no bundle, and complete the public API documentation.

Fix articulation data micro-benchmarks

b9a264d

Keep mock state buffers stable so timed samples measure accessor work instead of random-data allocation or mock-side copies. Initialize Newton dynamics metadata and make unsupported or deprecated properties explicit.

AntoineRichard force-pushed the antoiner/fix-articulation-micro-benchmarks branch from 3c2ca66 to b9a264d Compare July 20, 2026 19:18

kellyguo11 approved these changes Jul 21, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix articulation data micro-benchmarks#6565

Fix articulation data micro-benchmarks#6565
AntoineRichard wants to merge 32 commits into
isaac-sim:developfrom
AntoineRichard:antoiner/fix-articulation-micro-benchmarks

AntoineRichard commented Jul 16, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

AntoineRichard commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Dependencies

Type of change

Validation

Checklist

Uh oh!

greptile-apps Bot commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AntoineRichard commented Jul 16, 2026 •

edited

Loading