Skip to content

[Test] Optimize Newton test runtime - #7306

Merged
ooctipus merged 2 commits into
isaac-sim:developfrom
StafaH:test/optimize-newton-tests
Aug 24, 2026
Merged

[Test] Optimize Newton test runtime#7306
ooctipus merged 2 commits into
isaac-sim:developfrom
StafaH:test/optimize-newton-tests

Conversation

@StafaH

@StafaH StafaH commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Description

Reduces isaaclab_newton CI runtime while preserving distinct behavioral coverage.

  • Tests actuator target-mode configuration directly instead of launching a simulation for pure configuration behavior.
  • Preserves singleton and batched coverage at initialization boundaries, then uses representative batched layouts for downstream behavior.
  • Consolidates redundant rigid-object and rigid-object-collection parameter matrices.
  • Keeps the complete contact lifecycle shape matrix while narrowing the secondary horizontal-collision matrix to representative primitive and mesh cases.
  • Combines contact-sensor metadata checks into one scene.
  • Removes simulation-backed print/string smoke tests and replaces tautological self-comparisons with cross-view behavioral assertions.

Before and after measurement

Measured with the CI-style per-file test runner on the same workstation, using a new empty Warp cache for each revision. Both revisions use the repository-default Warp backward configuration:

Revision Cases Result Wall time
upstream/develop (c4a275975) 1,180 0 failures, 0 errors 30m 44.38s
This PR (552b329a8) 907 0 failures, 0 errors 21m 41.56s

This saves 9m 02.82s (29.4%) in a cold-cache run. Both measurements executed all 35 Newton test files.

Measurement command:

TEST_FILTER_PATTERN=isaaclab_newton \
WARP_CACHE_PATH=<new-empty-cache> \
uv run --no-sync python -m pytest tools -q

Type of change

  • Test and CI performance improvement (non-breaking)

Validation

  • Full optimized Newton suite: 35 files, 907 cases, 0 failures, 0 errors.
  • Full untouched baseline suite: 35 files, 1,180 cases, 0 failures, 0 errors.
  • Focused changed suites all pass: articulation, rigid object, rigid object collection, contact sensor, frame transformer, IMU, joint wrench, and PVA.
  • Ruff and Ruff format hooks pass for all changed Python files.
  • git diff --check passes.
  • uv run --no-sync isaaclab -f passes all applicable hooks; the repository-wide changelog checker reports pre-existing fragment immutability/missing-fragment issues in unrelated packages. This PR includes an isaaclab_newton .skip fragment.

Checklist

  • I have read and followed the contribution guidelines.
  • I have run the relevant tests and included the results above.
  • I have formatted and linted the changed files.
  • I have added the appropriate changelog fragment.

Screenshots

Not applicable.

@StafaH
StafaH requested a review from a team August 23, 2026 02:11
@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Aug 23, 2026

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The Newton test-runtime optimization reduces redundant simulation and parameter coverage, but the new suite-level Warp configuration leaks beyond the Newton tests in shared pytest processes.

  • Design and architecture: The package conftest sets a process-global Warp option during collection without restoring or isolating it. Directory scoping of conftest discovery does not scope the resulting Warp state, so other collected packages can inherit Newton-specific behavior.
  • API: No production or public API surface changes. The test-only private helper import and optional scene configuration field do not affect users, and the package includes an appropriate .skip changelog fragment.
  • Implementation: The narrowed matrices and cross-view assertions retain representative batched coverage, but wp.config.enable_backward = False must be confined to isolated Newton execution or safely saved and restored so broad pytest invocations remain order-independent.

Minor fixes needed. Posted 1 actionable finding inline.

Automated review; human maintainers own approval decisions.

Comment thread source/isaaclab_newton/test/conftest.py Outdated
# Newton tests do not exercise Warp autodiff. Set this before pytest imports test
# modules so kernels are compiled without backward support, matching the training,
# play, and benchmark entry points.
wp.config.enable_backward = False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning · Implementation — Package conftest mutates global Warp config

wp.config.enable_backward is process-global, and pytest imports a directory's conftest.py during collection, before any test runs. A session that collects source/ therefore disables Warp backward codegen for every other package's tests as well, and the previous value is never restored, so behavior depends on which directories are collected. Scope this to Newton-only execution (save/restore around the suite, or set it in the isolated Newton runner).

@greptile-apps

greptile-apps Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR reduces Newton test-suite runtime by narrowing redundant parameter matrices, replacing simulation-backed configuration checks with direct unit checks, consolidating sensor scenes, disabling unused Warp backward generation, and removing low-value smoke assertions.

  • Preserves singleton and batched initialization coverage while using representative batched layouts downstream.
  • Retains complete contact-lifecycle coverage and narrows secondary collision-shape combinations.
  • Replaces tautological rigid-body assertions with cross-view behavioral comparisons.

Confidence Score: 5/5

The PR appears safe to merge because the changes are confined to test optimization and no concrete correctness or security defect remains.

Representative boundary coverage is retained, the new cross-view assertions strengthen behavior checks, and the global Warp setting is consistent with the repository’s lack of Warp-autodiff tests.

Important Files Changed

Filename Overview
source/isaaclab_newton/test/conftest.py Disables unused Warp backward-code generation for Newton tests at collection time, consistent with existing repository test configuration.
source/isaaclab_newton/test/assets/test_articulation.py Converts target-mode checks to direct configuration tests and substantially reduces redundant downstream batch-size combinations.
source/isaaclab_newton/test/assets/test_rigid_object.py Narrows batch matrices and replaces self-comparisons with meaningful root/body cross-view assertions.
source/isaaclab_newton/test/assets/test_rigid_object_collection.py Consolidates environment/object-count matrices while retaining representative singleton and batched initialization boundaries.
source/isaaclab_newton/test/sensors/test_contact_sensor.py Narrows secondary collision-shape coverage and consolidates three distinct metadata configurations into one multi-sensor scene.

Reviews (1): Last reviewed commit: "Optimize Newton test coverage" | Re-trigger Greptile

@AntoineRichard AntoineRichard left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ooctipus
ooctipus merged commit 6aad90b into isaac-sim:develop Aug 24, 2026
47 of 48 checks passed
ooctipus pushed a commit to ooctipus/IsaacLab that referenced this pull request Aug 24, 2026
# Description

Reduces `isaaclab_newton` CI runtime while preserving distinct
behavioral coverage.

- Tests actuator target-mode configuration directly instead of launching
a simulation for pure configuration behavior.
- Preserves singleton and batched coverage at initialization boundaries,
then uses representative batched layouts for downstream behavior.
- Consolidates redundant rigid-object and rigid-object-collection
parameter matrices.
- Keeps the complete contact lifecycle shape matrix while narrowing the
secondary horizontal-collision matrix to representative primitive and
mesh cases.
- Combines contact-sensor metadata checks into one scene.
- Removes simulation-backed print/string smoke tests and replaces
tautological self-comparisons with cross-view behavioral assertions.

## Before and after measurement

Measured with the CI-style per-file test runner on the same workstation,
using a new empty Warp cache for each revision. Both revisions use the
repository-default Warp backward configuration:

| Revision | Cases | Result | Wall time |
|---|---:|---|---:|
| `upstream/develop` (`c4a275975`) | 1,180 | 0 failures, 0 errors | 30m
44.38s |
| This PR (`552b329a8`) | 907 | 0 failures, 0 errors | 21m 41.56s |

This saves **9m 02.82s (29.4%)** in a cold-cache run. Both measurements
executed all 35 Newton test files.

Measurement command:

```bash
TEST_FILTER_PATTERN=isaaclab_newton \
WARP_CACHE_PATH=<new-empty-cache> \
uv run --no-sync python -m pytest tools -q
```

## Type of change

- Test and CI performance improvement (non-breaking)

## Validation

- Full optimized Newton suite: 35 files, 907 cases, 0 failures, 0
errors.
- Full untouched baseline suite: 35 files, 1,180 cases, 0 failures, 0
errors.
- Focused changed suites all pass: articulation, rigid object, rigid
object collection, contact sensor, frame transformer, IMU, joint wrench,
and PVA.
- Ruff and Ruff format hooks pass for all changed Python files.
- `git diff --check` passes.
- `uv run --no-sync isaaclab -f` passes all applicable hooks; the
repository-wide changelog checker reports pre-existing fragment
immutability/missing-fragment issues in unrelated packages. This PR
includes an `isaaclab_newton` `.skip` fragment.

## Checklist

- [x] I have read and followed the contribution guidelines.
- [x] I have run the relevant tests and included the results above.
- [x] I have formatted and linted the changed files.
- [x] I have added the appropriate changelog fragment.

## Screenshots

Not applicable.

(cherry picked from commit 6aad90b)
ooctipus added a commit that referenced this pull request Aug 24, 2026
…#7303 (#7313)

# Description

Backports six merged PRs to `release/3.0.0` as separate
provenance-preserving cherry-picks:

| Source PR | Source commit | Backport commit | Scope |
| --- | --- | --- | --- |
| #7272 | `0081477fea2` | `c9fc1997476` | Fix scene-wide gravity
distributions for PhysX and OvPhysX. |
| #7121 | `9f65e3d4fd2` | `cce8acd84e1` | Fix backend-factory fallback
before simulator initialization. |
| #7183 | `3fcc9c6b824` | `99d58d3d3c4` | Streamline the getting-started
documentation. |
| #7306 | `6aad90be0ae` | `9a786cc028a` | Optimize Newton test runtime.
|
| #7305 | `1c9ba908cf1` | `a2dc8ce948e` | Disable Warp autodiff in the
test suite. |
| #7303 | `393fc37d2b0` | `249a5cb97d8` | Remove post-Hydra preset
resolution. |

#7272 forwards the validated `uniform`, `log_uniform`, or `gaussian`
distribution through both scene-wide backend paths. Newton behavior is
unchanged.

#7121 makes factory resolution fall back to Newton when no
`SimulationContext` exists while retaining the visualizer contract of
reporting no active backend before context creation.

#7183 is documentation-only. #7306 and #7305 change test infrastructure
only; they do not change runtime behavior.

#7303 makes `resolve_task_config` and `parse_env_cfg` the
task-composition boundary. Runtime consumers now require concrete
physics, renderer, and camera configurations and no longer attempt late
preset fallback after Hydra composition.

The branch was synchronized with the current `release/3.0.0` tip
containing #7301 before #7303 was applied. All six source cherry-picks
applied without conflicts. A file-by-file existence, mode, and blob
audit confirmed that every #7303 path matches its merged source commit
exactly.

## Validation

### #7272

- Regression verification against the unpatched `release/3.0.0` tip —
both PhysX and OvPhysX cases failed as expected.
- `uv run --extra test --frozen python -m pytest -q
source/isaaclab/test/envs/test_gravity_randomization.py
source/isaaclab/test/envs/test_mdp_event_selectors.py` — 7 passed.
- `uv run --extra test --extra ovphysx --frozen python -m pytest -q
source/isaaclab_ov/test/physics/test_ovphysx_gravity.py` — 1 passed.

### #7121

- Regression verification before the #7121 cherry-pick reproduced the
original `NoneType.physics_manager` failure.
- `uv run --extra test --frozen python -m pytest -q
source/isaaclab/test/utils/test_backend_utils.py
source/isaaclab/test/visualizers/test_visualizer.py` — 18 passed.

### #7183

- `uv run --frozen --extra test python -m pytest --noconftest -q
tools/test/test_environ_docs.py` — 26 passed.

### #7306

- All 9 affected paths match the merged source PR exactly.
- The full simulator-backed Newton suite is left to backport CI because
this local worktree does not include the full Isaac Sim runtime.

### #7305

- Loaded the root `conftest.py` successfully with Warp unavailable.
- Loaded it with Warp installed and confirmed `wp.config.enable_backward
is False`.

### #7303

- Hydra and Shadow Hand camera suites — 129 passed.
- Benchmark capture and RL entrypoint suites — 31 passed.
- Experimental frontend and custom-coupling suites — 74 passed, 1
skipped.
- Ruff and Ruff-format passed for every changed Python path.
- Source/backport audit — 54 paths checked, 0 mismatches.
- The Isaac Sim-backed integration case is left to CI because this local
worktree is kitless.

### Repository gates

- `git diff --check upstream/release/3.0.0...HEAD` — passed.

---------

Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Co-authored-by: Antoine RICHARD <antoiner@nvidia.com>
Co-authored-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Co-authored-by: Mustafa H <34825877+StafaH@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants