[Test] Optimize Newton test runtime - #7306
Conversation
There was a problem hiding this comment.
Isaac Lab Review Bot
The Newton test-runtime optimization reduces redundant simulation and parameter coverage, but the new suite-level Warp configuration leaks beyond the Newton tests in shared pytest processes.
- Design and architecture: The package conftest sets a process-global Warp option during collection without restoring or isolating it. Directory scoping of conftest discovery does not scope the resulting Warp state, so other collected packages can inherit Newton-specific behavior.
- API: No production or public API surface changes. The test-only private helper import and optional scene configuration field do not affect users, and the package includes an appropriate
.skipchangelog fragment. - Implementation: The narrowed matrices and cross-view assertions retain representative batched coverage, but
wp.config.enable_backward = Falsemust be confined to isolated Newton execution or safely saved and restored so broad pytest invocations remain order-independent.
Minor fixes needed. Posted 1 actionable finding inline.
Automated review; human maintainers own approval decisions.
| # Newton tests do not exercise Warp autodiff. Set this before pytest imports test | ||
| # modules so kernels are compiled without backward support, matching the training, | ||
| # play, and benchmark entry points. | ||
| wp.config.enable_backward = False |
There was a problem hiding this comment.
🟡 Warning · Implementation — Package conftest mutates global Warp config
wp.config.enable_backward is process-global, and pytest imports a directory's conftest.py during collection, before any test runs. A session that collects source/ therefore disables Warp backward codegen for every other package's tests as well, and the previous value is never restored, so behavior depends on which directories are collected. Scope this to Newton-only execution (save/restore around the suite, or set it in the isolated Newton runner).
Greptile SummaryThis PR reduces Newton test-suite runtime by narrowing redundant parameter matrices, replacing simulation-backed configuration checks with direct unit checks, consolidating sensor scenes, disabling unused Warp backward generation, and removing low-value smoke assertions.
Confidence Score: 5/5The PR appears safe to merge because the changes are confined to test optimization and no concrete correctness or security defect remains. Representative boundary coverage is retained, the new cross-view assertions strengthen behavior checks, and the global Warp setting is consistent with the repository’s lack of Warp-autodiff tests. Important Files Changed
Reviews (1): Last reviewed commit: "Optimize Newton test coverage" | Re-trigger Greptile |
# Description Reduces `isaaclab_newton` CI runtime while preserving distinct behavioral coverage. - Tests actuator target-mode configuration directly instead of launching a simulation for pure configuration behavior. - Preserves singleton and batched coverage at initialization boundaries, then uses representative batched layouts for downstream behavior. - Consolidates redundant rigid-object and rigid-object-collection parameter matrices. - Keeps the complete contact lifecycle shape matrix while narrowing the secondary horizontal-collision matrix to representative primitive and mesh cases. - Combines contact-sensor metadata checks into one scene. - Removes simulation-backed print/string smoke tests and replaces tautological self-comparisons with cross-view behavioral assertions. ## Before and after measurement Measured with the CI-style per-file test runner on the same workstation, using a new empty Warp cache for each revision. Both revisions use the repository-default Warp backward configuration: | Revision | Cases | Result | Wall time | |---|---:|---|---:| | `upstream/develop` (`c4a275975`) | 1,180 | 0 failures, 0 errors | 30m 44.38s | | This PR (`552b329a8`) | 907 | 0 failures, 0 errors | 21m 41.56s | This saves **9m 02.82s (29.4%)** in a cold-cache run. Both measurements executed all 35 Newton test files. Measurement command: ```bash TEST_FILTER_PATTERN=isaaclab_newton \ WARP_CACHE_PATH=<new-empty-cache> \ uv run --no-sync python -m pytest tools -q ``` ## Type of change - Test and CI performance improvement (non-breaking) ## Validation - Full optimized Newton suite: 35 files, 907 cases, 0 failures, 0 errors. - Full untouched baseline suite: 35 files, 1,180 cases, 0 failures, 0 errors. - Focused changed suites all pass: articulation, rigid object, rigid object collection, contact sensor, frame transformer, IMU, joint wrench, and PVA. - Ruff and Ruff format hooks pass for all changed Python files. - `git diff --check` passes. - `uv run --no-sync isaaclab -f` passes all applicable hooks; the repository-wide changelog checker reports pre-existing fragment immutability/missing-fragment issues in unrelated packages. This PR includes an `isaaclab_newton` `.skip` fragment. ## Checklist - [x] I have read and followed the contribution guidelines. - [x] I have run the relevant tests and included the results above. - [x] I have formatted and linted the changed files. - [x] I have added the appropriate changelog fragment. ## Screenshots Not applicable. (cherry picked from commit 6aad90b)
…#7303 (#7313) # Description Backports six merged PRs to `release/3.0.0` as separate provenance-preserving cherry-picks: | Source PR | Source commit | Backport commit | Scope | | --- | --- | --- | --- | | #7272 | `0081477fea2` | `c9fc1997476` | Fix scene-wide gravity distributions for PhysX and OvPhysX. | | #7121 | `9f65e3d4fd2` | `cce8acd84e1` | Fix backend-factory fallback before simulator initialization. | | #7183 | `3fcc9c6b824` | `99d58d3d3c4` | Streamline the getting-started documentation. | | #7306 | `6aad90be0ae` | `9a786cc028a` | Optimize Newton test runtime. | | #7305 | `1c9ba908cf1` | `a2dc8ce948e` | Disable Warp autodiff in the test suite. | | #7303 | `393fc37d2b0` | `249a5cb97d8` | Remove post-Hydra preset resolution. | #7272 forwards the validated `uniform`, `log_uniform`, or `gaussian` distribution through both scene-wide backend paths. Newton behavior is unchanged. #7121 makes factory resolution fall back to Newton when no `SimulationContext` exists while retaining the visualizer contract of reporting no active backend before context creation. #7183 is documentation-only. #7306 and #7305 change test infrastructure only; they do not change runtime behavior. #7303 makes `resolve_task_config` and `parse_env_cfg` the task-composition boundary. Runtime consumers now require concrete physics, renderer, and camera configurations and no longer attempt late preset fallback after Hydra composition. The branch was synchronized with the current `release/3.0.0` tip containing #7301 before #7303 was applied. All six source cherry-picks applied without conflicts. A file-by-file existence, mode, and blob audit confirmed that every #7303 path matches its merged source commit exactly. ## Validation ### #7272 - Regression verification against the unpatched `release/3.0.0` tip — both PhysX and OvPhysX cases failed as expected. - `uv run --extra test --frozen python -m pytest -q source/isaaclab/test/envs/test_gravity_randomization.py source/isaaclab/test/envs/test_mdp_event_selectors.py` — 7 passed. - `uv run --extra test --extra ovphysx --frozen python -m pytest -q source/isaaclab_ov/test/physics/test_ovphysx_gravity.py` — 1 passed. ### #7121 - Regression verification before the #7121 cherry-pick reproduced the original `NoneType.physics_manager` failure. - `uv run --extra test --frozen python -m pytest -q source/isaaclab/test/utils/test_backend_utils.py source/isaaclab/test/visualizers/test_visualizer.py` — 18 passed. ### #7183 - `uv run --frozen --extra test python -m pytest --noconftest -q tools/test/test_environ_docs.py` — 26 passed. ### #7306 - All 9 affected paths match the merged source PR exactly. - The full simulator-backed Newton suite is left to backport CI because this local worktree does not include the full Isaac Sim runtime. ### #7305 - Loaded the root `conftest.py` successfully with Warp unavailable. - Loaded it with Warp installed and confirmed `wp.config.enable_backward is False`. ### #7303 - Hydra and Shadow Hand camera suites — 129 passed. - Benchmark capture and RL entrypoint suites — 31 passed. - Experimental frontend and custom-coupling suites — 74 passed, 1 skipped. - Ruff and Ruff-format passed for every changed Python path. - Source/backport audit — 54 paths checked, 0 mismatches. - The Isaac Sim-backed integration case is left to CI because this local worktree is kitless. ### Repository gates - `git diff --check upstream/release/3.0.0...HEAD` — passed. --------- Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com> Co-authored-by: Antoine RICHARD <antoiner@nvidia.com> Co-authored-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com> Co-authored-by: Mustafa H <34825877+StafaH@users.noreply.github.com>
Description
Reduces
isaaclab_newtonCI runtime while preserving distinct behavioral coverage.Before and after measurement
Measured with the CI-style per-file test runner on the same workstation, using a new empty Warp cache for each revision. Both revisions use the repository-default Warp backward configuration:
upstream/develop(c4a275975)552b329a8)This saves 9m 02.82s (29.4%) in a cold-cache run. Both measurements executed all 35 Newton test files.
Measurement command:
Type of change
Validation
git diff --checkpasses.uv run --no-sync isaaclab -fpasses all applicable hooks; the repository-wide changelog checker reports pre-existing fragment immutability/missing-fragment issues in unrelated packages. This PR includes anisaaclab_newton.skipfragment.Checklist
Screenshots
Not applicable.