[CI] Cross-platform — Part 5: Dynamic work-stealing across multi-GPU shards

[hujc7]

1. Summary

• Replaces the static round-robin file split across multi-GPU shards with dynamic work-stealing: all discovered test files go into one shared queue, and each shard container claims the next file under flock. A shard that lands a slow file simply claims fewer, so the GPUs stay balanced — no per-file timing data needed, and every file still runs exactly once on one non-default GPU.
• 2-file delta on top of [CI] Cross-platform — Part 4: Multi-GPU pytest workflow with cuda:1 coverage #5823: tools/conftest.py (+claim-from-queue) and .github/workflows/test-multi-gpu-pytest.yaml (+seed/mount queue, −round-robin).
• Keeps the per-shard cache-isolated containers from [CI] Cross-platform — Part 4: Multi-GPU pytest workflow with cuda:1 coverage #5823 unchanged, so there is no new shared-cache contention.

Stacked on #5823 — the multi-GPU workflow file and the conftest summary code both originate there. Review only the two files above; the rest of the diff is #5823 and disappears once it merges.

2. Why work-stealing

The static round-robin split (#5823) assigns file i to shard i mod N. File durations vary widely (Kit-boot tests ~60–90 s, pure-math tests ~ms), so one shard routinely becomes the long pole while others idle. Work-stealing removes the imbalance with no duration model: whichever shard is free claims the next file.

3. Mechanism

3.1 Shared queue (conftest)

tools/conftest.py gains _claim_queued_file(queue_path) — an flock-guarded pop of the first line of a shared queue file — and _queued_files(), a generator that drains it. When ISAACLAB_TEST_QUEUE is set, run_individual_tests iterates the queue instead of the static include list; each file still runs once, on this container's pinned GPU. The end-of-run summary reports the files this container actually claimed.

3.2 Workflow

The discover step also emits the full relative paths; the run step seeds them into queue.txt in a dir mounted into every shard container at /mgpu, sets ISAACLAB_TEST_QUEUE=/mgpu/queue.txt, and drops the round-robin partition. flock on the shared mount serializes claims across containers (same host inode via the bind mount).

4. Design note — why not one container with N GPU worker threads

An earlier prototype ran a single container with one thread per GPU. The blocker: Kit's shader/compute cache lives at the absolute path /isaac-sim/.nv/ComputeCache, which is shared across concurrent Kit boots in one container — and per-worker HOME cannot relocate an absolute path. Concurrent shader compiles would race there. The existing N-container design exists precisely to isolate that cache, so this PR keeps N containers and only changes how files are distributed to them.

5. Validation

Queue logic validated locally: interleaved claimers each get every file exactly once, no double-claim, a missing queue file returns cleanly. conftest compiles, workflow YAML parses, pre-commit clean. The single-GPU CI path is untouched (ISAACLAB_TEST_QUEUE unset ⇒ the static iteration as before).

---

Note (2026-05-30): Cherry-picked #5881 for hypothesis test

This branch currently contains a cherry-pick of #5881 (Honor device kwarg over sim_cfg.device in build_simulation_context) to validate whether the Kit lifecycle hang in dynamic 3-shard mode is exacerbated by the device-drift bug. CI on this push exercises both fixes together. When #5881 lands on develop, this branch will rebase and the duplicate commit will drop automatically.

[isaaclab-review-bot]

---

Update (commit 74d1ad1): New commit replaces the workflow-level grep -vE exclusion list with a file-level marker convention.

Changes in This Update

1. .github/workflows/test-multi-gpu-pytest.yaml

The discovery step now checks each candidate file for a MULTI_GPU_SKIP_REASON module-level constant instead of maintaining a hardcoded exclusion list:

for f in "${candidates[@]}"; do
  if grep -q '^MULTI_GPU_SKIP_REASON' "$f"; then
    skipped+=("$f")
  else
    discovered+=("$f")
  fi
done

Skipped files emit a ::notice:: annotation with the reason for CI visibility.

2. Test File Annotations

source/isaaclab_newton/test/assets/test_articulation.py and source/isaaclab_physx/test/assets/test_articulation.py both add:

MULTI_GPU_SKIP_REASON = "Kit lifecycle bug: SIGHUP ~80% into the file under concurrent multi-GPU"

Assessment

This is a clean improvement over the previous grep -vE approach:

✅ Decoupled — Test owners add/remove skip annotations without touching workflow files

✅ Self-documenting — The reason is co-located with the test file

✅ Visible — CI ::notice:: annotations surface skipped files and reasons

✅ Maintainable — Adding a new skip is one line in the test file, not editing YAML

Minor Observations

1. Regex anchoring: grep -q '^MULTI_GPU_SKIP_REASON' requires the constant at line start. If someone indents it accidentally (e.g., inside a conditional), it won't be detected. This is actually a feature — it forces module-level declaration.

2. Reason extraction: The sed pipeline handles both "quoted" and unquoted values, but might truncate if the reason contains embedded quotes. Low risk since reasons are simple strings.

Outstanding Items (Unchanged)

The previous review's open items still apply:

• run_docker_tests: 'false' hardcoding in build.yaml (marked as "TEMP")
• 45-minute timeout (documented as diagnostic, target is 30)

Verdict

LGTM ✅ — Nice incremental improvement. The file-level marker convention is cleaner and more maintainable than workflow-level exclusion lists.

---

Update (commit e2bfa1b): New commit adds --ulimit nofile=65535:65535 to the workflow's docker run and comments out the MULTI_GPU_SKIP_REASON markers in both test files as a CI experiment to validate the fix.

Assessment: Clean diagnostic change. The ulimit bump targets the root cause (Kit/Vulkan needing >1024 fds for multi-GPU shared resources), and the commented-out markers preserve easy rollback. No new issues introduced. Previous outstanding items (build.yaml run_docker_tests: 'false' hardcoding, 45-min timeout) still apply but are pre-existing.

---

Update (commit 5464c1e): Experiment rolled back — --ulimit nofile=65535:65535 removed from docker run and MULTI_GPU_SKIP_REASON markers restored in both test files. The ulimit bump did not resolve the Kit lifecycle failures. Comments upgraded from "TEMP CI EXPERIMENT" to proper documentation explaining the skip convention. No new issues introduced.

[isaaclab-review-bot]

---

Update (commit e7d1af7): New commit implements a 2-shard cap as a concurrency-threshold workaround for the Kit lifecycle bug.

Changes in This Commit

1. .github/workflows/test-multi-gpu-pytest.yaml

Shard count is now capped at 2, regardless of available GPUs:

# TEMP EXPERIMENT: cap shards at 2 to test concurrency-threshold workaround for
# the Kit lifecycle bug (SIGHUP/shutdown-hang fires at >=3 concurrent Kit
# instances; 2 is below the threshold per local 2-MIG cross-GPU repro).
shards=$(python3 -c "import json; cap=2; print(json.dumps(list(range(1, min($N, cap+1)))))")
count=$(python3 -c "cap=2; print(min($N-1, cap))")

2. Test Files Re-enabled

Both test_articulation.py files (isaaclab_newton and isaaclab_physx) now have their MULTI_GPU_SKIP_REASON markers commented out to test whether the 2-shard cap avoids the Kit lifecycle bug.

Assessment

✅ Hypothesis-driven debugging — The approach systematically tests the theory that ≥3 concurrent Kit instances trigger the lifecycle bug, while ≤2 are stable.

✅ Conservative cap — 2 shards still provides multi-GPU coverage (cuda:1, cuda:2) while staying below the suspected threshold.

⚠️ GPU Utilization — On 4+ GPU systems, only 2 of the non-default GPUs will be utilized. This is acceptable as a diagnostic step but should be reverted once the root cause is understood.

Comments on Implementation

1. Inline Python for cap logic: The cap calculation uses inline Python:

   count=$(python3 -c "cap=2; print(min($N-1, cap))")

   This works but is slightly fragile. Consider extracting cap=2 to a workflow-level variable for easier adjustment:

   env:
     SHARD_CAP: 2  # TEMP: workaround for Kit lifecycle bug

2. Experiment lifecycle: This is now the third commit in the diagnostic cycle (enable → test → observe). The clear comments document the experiment purpose, which is good practice.

Outstanding Items (Cumulative)

Item	Status	Location
run_docker_tests: 'false' hardcoding	Pre-existing	build.yaml
45-minute timeout (target 30)	Pre-existing	test-multi-gpu-pytest.yaml
2-shard cap	New (diagnostic)	test-multi-gpu-pytest.yaml
Commented MULTI_GPU_SKIP_REASON markers	New (experiment)	Both test_articulation.py

Verdict

LGTM for experiment ✅ — Clean diagnostic change. The 2-shard cap is a reasonable approach to isolate the concurrency threshold. Results from this CI run will inform whether the cap resolves the issue or if the problem lies elsewhere.

Pre-merge checklist (once experiments conclude):

• Revert or formalize the shard cap based on findings
• Restore or remove MULTI_GPU_SKIP_REASON markers accordingly
• Revert run_docker_tests: 'false' in build.yaml
• Adjust 45-minute timeout if validated

---

Update (commit 4bc8b33): The 2-shard cap experiment has concluded. This commit:

1. Removes the shard cap — all available GPUs are now used (shards=$(python3 -c "...range(1, $N)"), count=$((N - 1))). The inline Python is simplified and the bash arithmetic is cleaner.
2. Re-enables MULTI_GPU_SKIP_REASON in both test_articulation.py files — with improved documentation explaining the opt-out mechanism. These files are excluded from concurrent multi-GPU CI while the upstream Kit lifecycle bug persists.

Previous Concerns Status

• ⚠️ Inline Python for cap logic → ✅ Moot — the cap is removed entirely. The remaining inline Python is minimal and the count uses native bash arithmetic.
• ⚠️ GPU Utilization → ✅ Fixed — all non-default GPUs are now utilized.
• 📋 2-shard cap → ✅ Reverted (experiment concluded).
• 📋 Commented MULTI_GPU_SKIP_REASON markers → ✅ Markers restored and properly documented.

Assessment

Clean resolution of the diagnostic cycle. The approach is sound: use all GPUs but exclude known-bad test files via a well-documented constant. The improved comments on MULTI_GPU_SKIP_REASON clearly explain the mechanism and removal criteria. No new issues introduced.

---

Update (commit 3d3f136): New commit adds a fix for build_simulation_context silently ignoring the device kwarg when sim_cfg is also provided.

Changes

1. simulation_context.py — device parameter default changed from "cuda:0" to None. When device is explicitly passed alongside sim_cfg, it now overrides sim_cfg.device. This fixes warp kernel device-mismatch errors on non-default GPUs.
2. Changelog entry added documenting the fix.
3. Test updates — both headless and non-headless test files split the test case to separately verify: (a) sim_cfg values are preserved when no device kwarg is given, and (b) explicit device kwarg wins when both are passed.

Assessment

✅ Correct fix — The None default + explicit override pattern is clean and backward-compatible. Callers not passing device get SimulationCfg's default; callers passing it explicitly get what they asked for.

✅ Well-tested — Both branches (sim_cfg-only and device-override) are covered.

✅ Good documentation — The inline comment explains the rationale and the failure mode it prevents.

⚠️ Scope note — This commit fixes a real bug but is unrelated to the CI multi-GPU sharding work in the PR title. Consider whether this should be a separate PR for cleaner git history, though it is understandable if the bug was discovered during multi-GPU testing.

No blocking issues. LGTM.

---

Update (commit 3bd01c4): Removes MULTI_GPU_SKIP_REASON from both test_articulation.py files (isaaclab_newton and isaaclab_physx), fully re-enabling them in concurrent multi-GPU CI.

This is the logical conclusion of the diagnostic cycle: the build_simulation_context device-override fix from the previous commit likely resolved the warp kernel failures that were triggering Kit lifecycle issues under concurrent execution. With the root cause addressed, the opt-out markers are no longer needed.

Changes

Both files simply remove the 6-line MULTI_GPU_SKIP_REASON constant block:

# File-level opt-out from concurrent multi-GPU CI. The multi-GPU workflow's
# discover step skips any test file declaring this module-level constant.
# ...
MULTI_GPU_SKIP_REASON = "Kit lifecycle bug: ..."

Assessment

✅ Clean removal — No code changes beyond deleting the opt-out marker.

✅ Logical progression — The diagnostic cycle is complete: 2-shard cap (experiment) → device fix (root cause) → re-enable tests (validation).

⚠️ Risk note — If the Kit lifecycle issues resurface in CI, these markers can be quickly re-added. The workflow's MULTI_GPU_SKIP_REASON discovery mechanism remains in place.

Outstanding Pre-merge Items

Item	Status	Notes
run_docker_tests: 'false'	⚠️ Still hardcoded	build.yaml — revert before final merge
45-minute timeout	ℹ️ Diagnostic	Can tighten to 30 once validated
Cherry-picked #5881	ℹ️ Noted in PR description	Will drop on rebase after #5881 merges

No new issues introduced. LGTM. ✅

Update (commit 7d04e41): Adds SIGHUP signal handling and fixes _abort_signal_handle_callback to properly exit after cleanup.

Changes

1. app_launcher.py — Registers SIGHUP handler so child processes shut down cleanly when the parent shell (supervising sibling shards) exits. Rewrites the callback to call sys.exit(128 + signum) after app.close(), preventing Python from resuming with half-torn-down Kit state. Uses contextlib.suppress(Exception) for robustness in signal context.
2. Changelog entry — Documents both fixes clearly.

Assessment

✅ Correct fix — SIGHUP cascading to child shards was the missing piece causing "Stage X already attached" failures. The explicit sys.exit() after cleanup is the right pattern for replaced signal dispositions.

✅ Clean implementation — Both contextlib and sys were already imported. Parameter rename from signal to signum avoids shadowing the module. Comments are thorough.

✅ No new issues introduced.

Outstanding Pre-merge Items

Item	Status	Notes
run_docker_tests: 'false'	⚠️ Still hardcoded	build.yaml — revert before final merge
45-minute timeout	ℹ️ Diagnostic	Can tighten to 30 once validated

---

Update (commit 2b4530d): Two new diagnostic commits (7251b74 + 2b4530d). The latest restores 3 shards and replaces --gpus all with --gpus "device=$cuda" to give each container exclusive access to a single physical GPU. This eliminates the cross-GPU visibility that was triggering the SIGHUP cascade / "Stage already attached" pattern.

Assessment: Clean diagnostic change. The per-container GPU isolation via --gpus device=N is the correct approach — it mirrors MIG-level hardware isolation and prevents Kit processes from interfering with each other's GPU contexts. The explanatory comment is clear and well-reasoned.

No new issues introduced. Previous outstanding items unchanged:

Item	Status	Notes
run_docker_tests: 'false'	⚠️ Still hardcoded	build.yaml — revert before final merge
45-minute timeout	ℹ️ Diagnostic	Can tighten to 30 once validated

---

Update (commit 6137b1b): Significant pivot — instead of per-container GPU isolation (--gpus device=N), the new approach eliminates Kit entirely from newton tests (kitless mode) and re-adds MULTI_GPU_SKIP_REASON for the physx test that still requires Kit.

Changes

1. Workflow — Reverts --gpus "device=$cuda" → --gpus all. The per-container isolation is no longer needed since newton tests no longer boot Kit.

2. schemas.py — New _create_fixed_joint_to_world() helper replaces the from omni.physx.scripts import utils import with pure pxr.UsdPhysics calls. Well-implemented: handles instance proxy / prototype prim climbing, unique joint naming, and correctly sets body0 (world) / body1 (articulation) with proper transforms via UsdGeom.XformCache.

3. Newton test files — test_articulation.py and test_rigid_object_collection.py remove AppLauncher boot and test_devices() import, replacing with hardcoded ["cuda:0", "cpu"] or ["cuda:0"]. This makes them fully kitless, avoiding the Kit lifecycle bug entirely.

4. PhysX test_articulation.py — Re-adds MULTI_GPU_SKIP_REASON to opt out of concurrent multi-GPU CI (this file still requires Kit).

5. Changelog entries — Properly added for the kitless fix.

Assessment

✅ Better root-cause solution — Rather than working around Kit concurrency bugs with container isolation, this removes the Kit dependency where it isn't needed. Kitless newton tests run faster (~30s saved per file) and are immune to Kit lifecycle issues.

✅ _create_fixed_joint_to_world() is well-written — Correctly mirrors the omni.physx.scripts.utils.createJoint single-selection Fixed branch. The prototype/instance climb, unique naming, and transform extraction are all correct.

✅ Clean test migration — Removing AppLauncher is safe for newton-only tests since SimulationContext gates Kit-specific paths on has_kit().

Outstanding Pre-merge Items

Item	Status	Notes
run_docker_tests: 'false'	⚠️ Still hardcoded	build.yaml — revert before final merge
45-minute timeout	ℹ️ Diagnostic	Can tighten to 30 once validated
MULTI_GPU_SKIP_REASON in physx test	ℹ️ Intentional	Stays until upstream Kit fix (IsaacLab #3475)

No new issues introduced. LGTM. ✅