[BUG] Fix Isaac Sim 6.0 streaming crash from prebundle torch shadowing

[hujc7]

Summary

Fixes a crash (nvbugs 6343978) when launching the Isaac Sim streaming app
(isaac-sim.streaming.sh / runheadless.sh) from an Isaac Lab install against Isaac Sim 6.0:

[Error] [isaacsim.core.deprecation_manager.api] Import error:
.../omni.isaac.ml_archive/pip_prebundle/torch/lib/libtorch_cuda.so:
undefined symbol: ncclDevCommCreate

Root cause (corrected after investigation)

Not a broken Isaac Sim package — a cross-stack NCCL collision:

• Isaac Sim 6.0's deprecated omni.isaac.ml_archive prebundle ships a self-consistent
  torch + NCCL (torch 2.11 + nccl 2.28.9, which does export ncclDevCommCreate).
• Isaac Lab installs its own pinned torch (2.10) + nccl (2.27.5) into site-packages.
• torch's Python-side CUDA preload resolves libnccl.so.2 by sys.path/namespace order
  (RTLD_GLOBAL), before libtorch_cuda.so's RPATH. On launch paths that don't import
  isaaclab (the streaming app — isaaclab/__init__.py deprioritizes the prebundle only on
  import), the older pip nccl 2.27.5 loads, the prebundle torch 2.11 binds to it, and
  ncclDevCommCreate is missing → crash. LD_DEBUG confirms the pip nccl wins the load.

_repoint_prebundle_packages() already exists to prevent this (it symlinks the prebundle to
the active env, collapsing to one stack), but it silently no-op'd inside the Docker image
build: the prebundle is on an overlayfs lower layer where os.rename→.bak raises EXDEV
and shutil.rmtree raises EINVAL (its dir_fd walk), both swallowed as warnings. It works
on the host (single filesystem), which is why only the Docker image was affected.

Fix

Make _repoint_prebundle_packages() overlayfs-safe via a single uniform path:

• Replace rename-to-.bak + shutil.rmtree with _force_remove (plain-path
  os.unlink/os.rmdir, which create the proper overlayfs whiteout) + symlink. No .bak
  (the env copy is the symlink target), no subprocess, no fallback.
• Add a fail-loud post-check: raise if a real prebundle torch still shadows the pip torch,
  so the broken state can't ship silently again.

This collapses the two torch/nccl stacks into one on every launch path, host and Docker.

Validation (x86 RTX A6000, exact isaac-lab container)

• Before: streaming crashes at ~19s with undefined symbol: ncclDevCommCreate.
• After (repoint runs at build): prebundle torch → symlink to env 2.10; streaming boots to
  Isaac Sim Full Streaming Version: 6.0.0-rc.47, 0 ncclDevCommCreate.

Tests

• test_no_shadowing_prebundled_torch_in_isaac_sim (filesystem invariant): RED on the
  un-fixed image (real prebundle torch), GREEN on the fixed image / host (symlink).
• Updated _repoint_prebundle_packages unit tests: no .bak; new fail-loud test.

Fixes nvbugs 6343978.

[greptile-apps]

Greptile Summary

Fixes a Docker image build-time crash where Isaac Sim 6.0's prebundled PyTorch (omni.isaac.ml_archive/extsDeprecated/.../torch) contains libtorch_cuda.so that references an NCCL symbol absent from its co-bundled NCCL, causing streaming launchers that bypass isaaclab's sys.path guard to crash. The fix removes the broken artifact during docker build, mirrors the existing Dockerfile.curobo pattern, and adds a regression test.

• docker/Dockerfile.base: Adds rm -rf for the extsDeprecated/ (Isaac Sim 6.0) and exts/ (Isaac Sim 5.x) torch prebundle paths, followed by an unconditional pip reinstall from bootstrap.pypa.io to recover pip vendor files deleted as a side-effect.
• test_install.py: New regression test test_no_shadowing_prebundled_torch_in_isaac_sim walks all pip_prebundle/torch directories under the Isaac Sim root and asserts none are real (non-symlink) directories; skips gracefully when Isaac Sim is not installed.
• Dockerfile.curobo (unchanged by this PR) only removes the exts/ path, leaving the extsDeprecated/ gap present for curobo images built against Isaac Sim 6.0.

Confidence Score: 4/5

The base image fix is correct and validated. The curobo image has a parallel gap that would expose Isaac Sim 6.0 curobo users to the same streaming crash.

The Dockerfile.base change correctly removes the broken prebundle from both the Isaac Sim 6.0 (extsDeprecated/) and 5.x (exts/) paths and the regression test is sound. However, Dockerfile.curobo only removes the exts/ path and would still leave the extsDeprecated/ prebundle in place on Isaac Sim 6.0, meaning curobo-image streaming launches would still crash with the same undefined-symbol error this PR is fixing.

docker/Dockerfile.curobo — the extsDeprecated/ removal added to Dockerfile.base was not applied here, leaving a gap for curobo + Isaac Sim 6.0 streaming builds.

Important Files Changed

Filename	Overview
docker/Dockerfile.base	Adds two RUN layers: one removes the broken prebundled torch from both extsDeprecated/ (Isaac Sim 6.0) and exts/ (Isaac Sim 5.x) paths; a second reinstalls pip via bootstrap.pypa.io to recover pip vendor files deleted as a side-effect. Pattern mirrors the existing Dockerfile.curobo approach. Minor concern: the pip reinstall runs unconditionally even when no torch was removed.
docker/Dockerfile.curobo	Not changed by this PR but has a gap: only removes torch* from the Isaac Sim 5.x exts/ path, not from the extsDeprecated/ path used by Isaac Sim 6.0. This means curobo images built against Isaac Sim 6.0 would still crash on streaming launches.
source/isaaclab/test/cli/test_install.py	Adds a regression test that walks all pip_prebundle/torch directories under the Isaac Sim root and asserts none are real directories (as opposed to symlinks). Correctly skips when Isaac Sim is not installed. Test logic and use of extract_isaacsim_path(required=False) are correct.
source/isaaclab/changelog.d/jichuanh-fix-sim6-prebundle-torch-streaming.rst	New changelog entry accurately describes the root cause and fix.

_{Reviews (1): Last reviewed commit: "Add regression test for prebundled torch..." | Re-trigger Greptile}

[greptile-apps]

Network fetch of get-pip.py introduces build-time internet dependency

wget https://bootstrap.pypa.io/get-pip.py is called unconditionally during every image build, even when neither extsDeprecated/ nor exts/ torch* paths exist (e.g. Isaac Sim 5.x without curobo, where the first rm -rf is a no-op and pip was never disturbed). If bootstrap.pypa.io is unreachable or the script is updated with a breaking change the build silently breaks at this step. Consider guarding the pip reinstall with a check for whether the prior rm -rf actually removed anything, or pin to a known-good hash of the script.