Stabilize standalone demo smoke tests

[kellyguo11]

Description

Standalone demo smoke tests had two independent flaky failure modes:

1. The supervisor kept classifying fatal-looking output after it intentionally sent SIGTERM. Kit can emit Material Library, USD caching, and Replicator asyncio tracebacks during that forced teardown, after the demo reached readiness and completed its healthy soak.
2. The first camera-enabled Kit/RTX process consistently needs roughly 526?558 seconds to initialize on CI, but the camera case had a fixed 600-second startup budget. That left only 42?74 seconds for runner, cache, and service variation. In the timeout example, approximately 10 seconds of failed OmniHub launch retries contributed to the case crossing the limit at 605.8 seconds.

A CI experiment that disabled image saving and deferred the camera demo's Matplotlib import still took 526 seconds. That rules out optional plotting and confirms the long phase belongs to cold camera-enabled Kit/RTX initialization; the experimental demo change has been removed from the final diff.

This change fixes both test boundaries:

• Fatal-pattern detection stops immediately before intentional teardown, while shutdown output remains captured for diagnostics. Fatal output during startup or the soak still fails the test.
• The camera case receives a 900-second startup timeout, providing headroom for its known cold initialization while preserving the 300-second default hang detector for other demos.

Example failures:

• Intentional-teardown traceback: https://github.com/isaac-sim/IsaacLab/actions/runs/31243355943/job/93068486798?pr=6831
• Same teardown race on another unrelated PR: https://github.com/isaac-sim/IsaacLab/actions/runs/31230902685/job/93048260426
• Same teardown race on PR 6979: https://github.com/isaac-sim/IsaacLab/actions/runs/31247096260/job/93077436474?pr=6979
• Camera startup budget exhausted: https://github.com/isaac-sim/IsaacLab/actions/runs/31243047225/job/93069642023?pr=6979

No new dependencies are required.

Type of change

• Bug fix (non-breaking change which fixes an issue)

Screenshots

Not applicable; this changes headless smoke-test supervision and a case-specific timeout.

Validation

• Teardown regression confirmed failing before the fix and passing after it:
  uv run --frozen python -m pytest source/isaaclab/test/app/test_standalone_scripts.py::test_subprocess_supervisor_ignores_fatal_output_after_intentional_teardown -q
• Camera timeout contract confirmed failing at the previous 600-second value and passing at 900 seconds:
  uv run --frozen python -m pytest source/isaaclab/test/app/test_standalone_scripts.py::test_commands_respect_script_launcher_capabilities -q
• Buffered pre-teardown output regression confirmed failing before the drain fix and passing after it:
  uv run --frozen python -m pytest source/isaaclab/test/app/test_standalone_scripts.py::test_subprocess_supervisor_classifies_buffered_fatal_output_before_intentional_teardown -q
• Platform-independent standalone harness subset: 35 passed, 298 simulator launches skipped, and 7 POSIX process-control tests deselected on Windows.
• Linux standalone Kit CI passed with the teardown boundary; the camera case took 526 seconds, providing the direct timing evidence used for the final timeout fix: https://github.com/isaac-sim/IsaacLab/actions/runs/31247940912/job/93079622769
• Changelog fragment gate against current upstream develop: passed.
• uv run --frozen isaaclab -f: passed after staging and again after commit.

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with uv run isaaclab -f
• Documentation changes are not applicable to this test stabilization
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump extension.toml ? CI handles that)
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[greptile-apps]

Greptile Summary

Stabilizes standalone demo smoke tests by disabling optional camera image saving in the camera smoke case and suppressing fatal-pattern classification during intentional teardown.

• Adds a camera CLI option that skips output-directory creation, image saving, and the deferred Matplotlib import.
• Updates the camera smoke-test override to use the new option.
• Adds phase-sensitive fatal-output monitoring and regression coverage for teardown tracebacks.
• Leaves a narrow boundary where pre-teardown fatal output buffered at the soak deadline can be ignored.

Confidence Score: 4/5

The PR should not merge until fatal output produced just before intentional teardown is distinguished from output caused by teardown.

Disabling monitoring before draining stdout creates a reachable race where a genuine soak-time traceback is collected only after monitoring is off and therefore does not fail the smoke test.

Files Needing Attention: source/isaaclab/test/app/standalone_script_cases.py

Important Files Changed

Filename	Overview
scripts/demos/sensors/cameras.py	Adds an optional image-saving bypass and defers Matplotlib initialization until an image is actually saved.
source/isaaclab/test/app/standalone_script_cases.py	Updates camera smoke arguments and teardown monitoring, but disables fatal classification before pre-teardown stdout is guaranteed to be drained.
source/isaaclab/test/app/test_standalone_scripts.py	Adds focused regressions for deferred plotting and teardown output, but the teardown test covers only output returned after termination.
source/isaaclab/changelog.d/kellyguo11-fix-standalone-demo-teardown.skip	Adds the required empty changelog skip fragment.

_{Reviews (1): Last reviewed commit: "Reduce camera demo smoke startup time" | Re-trigger Greptile}

[greptile-apps]

Pre-teardown output escapes classification

When the child emits a fatal pattern after the final selector read but before the soak deadline check, monitoring is disabled before that buffered output is collected by communicate(), causing a genuinely unhealthy demo to pass the smoke test.

[isaaclab-review-bot]

Isaac Lab Review Bot

The camera-specific 900-second startup budget is appropriately scoped, but the teardown change can suppress fatal output emitted during the healthy-soak window if that output remains buffered when monitoring is disabled.

• Design and architecture: Keeping the extended timeout in the camera demo’s per-script override preserves the default timeout for other demos. The supervisor also owns the correct teardown boundary, but that boundary must distinguish output produced before SIGTERM from teardown-generated output.
• API: No public API or data-structure changes are introduced: run_until_ready, SmokeResult, and assert_smoke_passed retain their existing interfaces. The .skip changelog fragment is appropriate for this test-only change.
• Implementation: At the end of the soak, monitor_fatal_patterns is disabled before pending stdout is drained. Because each selector iteration reads at most 65536 bytes, output already emitted during the soak can remain queued and then be consumed by communicate() without fatal classification. Drain currently available output while monitoring remains enabled, then disable monitoring immediately before intentional termination.

Minor fixes needed. Posted 1 actionable finding inline.

Automated review; human maintainers own approval decisions.

[isaaclab-review-bot]

🔵 Suggestion · Implementation — Disable fatal detection after draining pending output

Each loop iteration reads at most 65536 bytes, and monitoring is cleared as soon as the soak elapses. Output the demo already emitted during the healthy soak but still queued in the pipe is later consumed by communicate() with detection off, so a genuine soak-time failure can be classified as teardown noise. Drain the currently available stdout with monitoring still enabled, then clear the flag immediately before sending SIGTERM.