[SOLVED] DGX Spark Support

[robmurrer]

If you are submitting a bug report, please fill in the following details and use the tag [bug].

Describe the bug

Cannot run IsaacLab 2.3.1 on DGX Spark

Steps to reproduce

build isaacsim 5.1 from source and then setup symbolic link. sourced the setup_conda_env.py and then ran ant example:

Traceback (most recent call last):
  File "/home/goat/deve/IsaacLab2p3p0/scripts/reinforcement_learning/skrl/train.py", line 70, in <module>
    app_launcher = AppLauncher(args_cli)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/goat/deve/IsaacLab2p3p0/source/isaaclab/isaaclab/app/app_launcher.py", line 127, in __init__
    self._create_app()
  File "/home/goat/deve/IsaacLab2p3p0/source/isaaclab/isaaclab/app/app_launcher.py", line 811, in _create_app
    self._app = SimulationApp(self._sim_app_config, experience=self._sim_experience_file)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goat/deve/IsaacSim/_build/linux-aarch64/release/exts/isaacsim.simulation_app/isaacsim/simulation_app/simulation_app.py", line 279, in __init__
    from .utils import create_new_stage, open_stage
  File "/home/goat/deve/IsaacSim/_build/linux-aarch64/release/exts/isaacsim.simulation_app/isaacsim/simulation_app/utils.py", line 23, in <module>
    from omni.kit.usd import layers
  File "/home/goat/deve/IsaacSim/_build/linux-aarch64/release/extscache/omni.kit.usd.layers-2.2.11+69cbf6ad.la64.r.cp311/omni/kit/usd/layers/__init__.py", line 9, in <module>
    from ._impl import *
  File "/home/goat/deve/IsaacSim/_build/linux-aarch64/release/extscache/omni.kit.usd.layers-2.2.11+69cbf6ad.la64.r.cp311/omni/kit/usd/layers/_impl/__init__.py", line 9, in <module>
    from .extension import *
  File "/home/goat/deve/IsaacSim/_build/linux-aarch64/release/extscache/omni.kit.usd.layers-2.2.11+69cbf6ad.la64.r.cp311/omni/kit/usd/layers/_impl/extension.py", line 18, in <module>
    import omni.usd
  File "/home/goat/deve/IsaacSim/_build/linux-aarch64/release/extscache/omni.usd-1.13.10+69cbf6ad.la64.r.cp311/omni/usd/__init__.py", line 43, in <module>
    from ._impl import *
  File "/home/goat/deve/IsaacSim/_build/linux-aarch64/release/extscache/omni.usd-1.13.10+69cbf6ad.la64.r.cp311/omni/usd/_impl/__init__.py", line 9, in <module>
    from .._usd import *
ImportError: libomni.usd.so: cannot open shared object file: No such file or directory

System Info

• Commit: e06a067
• Isaac Sim Version: 5.1.0
• OS: DGX OS
• GPU:DGX
• CUDA:| NVIDIA-SMI 580.82.09 Driver Version: 580.82.09 CUDA Version: 13.0 |

Additional context

Add any other context about the problem here.

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.

• Run training and inference on DGX Spark :)

[RandomOakForest]

Thank you for posting this. The team will review this.

[robmurrer]

So not sure why, but if you do not install inside a virtual environment and just install it in issacsim's kit python it works... Someone tried this and I was kind of skeptical. It seems perhaps the LD_LIBRARY needs to be configured to pick up the files?... seems the file exists here:

goat@spark-8fb9:~/deve/IsaacLab2p3p0$ sudo find / -iname "libomni.usd.so" 2>/dev/null /home/goat/.local/share/ov/data/exts/v2/omni.usd.core-1f9667570e265c62/bin/libomni.usd.so

alas, when i try to train the ant I get this error. I think this worked on my colleagues machine though. Not sure why i'm
./isaaclab.sh -p scripts/reinforcement_learning/skrl/train.py --task=Isaac-Ant-v0

    env = env_creator(**env_spec_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goat/deve/IsaacLab2p3p0/source/isaaclab/isaaclab/envs/manager_based_rl_env.py", line 82, in __init__
    super().__init__(cfg=cfg)
  File "/home/goat/deve/IsaacLab2p3p0/source/isaaclab/isaaclab/envs/manager_based_env.py", line 173, in __init__
    self.load_managers()
  File "/home/goat/deve/IsaacLab2p3p0/source/isaaclab/isaaclab/envs/manager_based_rl_env.py", line 118, in load_managers
    super().load_managers()
  File "/home/goat/deve/IsaacLab2p3p0/source/isaaclab/isaaclab/envs/manager_based_env.py", line 298, in load_managers
    self.observation_manager = ObservationManager(self.cfg.observations, self)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goat/deve/IsaacLab2p3p0/source/isaaclab/isaaclab/managers/observation_manager.py", line 82, in __init__
    super().__init__(cfg, env)
  File "/home/goat/deve/IsaacLab2p3p0/source/isaaclab/isaaclab/managers/manager_base.py", line 173, in __init__
    self._prepare_terms()
  File "/home/goat/deve/IsaacLab2p3p0/source/isaaclab/isaaclab/managers/observation_manager.py", line 549, in _prepare_terms
    obs_dims = tuple(term_cfg.func(self._env, **term_cfg.params).shape)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goat/deve/IsaacLab2p3p0/source/isaaclab/isaaclab/envs/utils/io_descriptors.py", line 195, in wrapper
    out = func(env, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goat/deve/IsaacLab2p3p0/source/isaaclab/isaaclab/envs/mdp/observations.py", line 67, in base_ang_vel
    return asset.data.root_ang_vel_b
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goat/deve/IsaacLab2p3p0/source/isaaclab/isaaclab/assets/articulation/articulation_data.py", line 1020, in root_ang_vel_b
    return self.root_com_ang_vel_b
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goat/deve/IsaacLab2p3p0/source/isaaclab/isaaclab/assets/articulation/articulation_data.py", line 809, in root_com_ang_vel_b
    return math_utils.quat_apply_inverse(self.root_link_quat_w, self.root_com_ang_vel_w)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)

nvrtc compilation failed: 

#define NAN __int_as_float(0x7fffffff)
#define POS_INFINITY __int_as_float(0x7f800000)
#define NEG_INFINITY __int_as_float(0xff800000)
...

[RandomOakForest]

Following up, you will need to install Isaac Lab from the release/2.3.0 branch for Spark support. Let us know if you need further help.

[robmurrer]

ok great. you are right. I was on wrong branch!

[nahidalam]

@robmurrer how did you solve it? I installed it with release/2.3.0 following this link https://isaac-sim.github.io/IsaacLab/release/2.3.0/source/setup/installation/source_installation.html

The installation seems to work but when I train like ./isaaclab.sh -p scripts/reinforcement_learning/skrl/train.py --task=Isaac-Ant-v0 I get the same error nvtrc error

RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)

nvrtc compilation failed: 

#define NAN __int_as_float(0x7fffffff)
#define POS_INFINITY __int_as_float(0x7f800000)
#define NEG_INFINITY __int_as_float(0xff800000)

[robmurrer]

Are you sure you did a git checkout origin/release/2.3.0? I also did a sudo apt update && sudo apt upgrade inbetween but i'm pretty sure the main issue was I was on wrong branch.

[robmurrer]

one thing to note is that branch doesn't have uv available as environment creator in ./isaaclab.sh --uv :( I solved this with installing uv and then doing uv venv venv -p 3.11 then installed isaaclab to that environment.

[nahidalam]

ah @robmurrer okay I was in the main branch. switched to release/2.3.0 and the training seems to be working.

Although after installation when I verify with scripts/tutorials/00_sim/create_empty.py I see IsaacSim screen loading but below message in the terminal. Not sure if it matters

[7.837s] Simulation App Starting
2025-10-24T22:15:15Z [8,205ms] [Warning] [omni.fabric.plugin] Warning: attribute viewPickingId not found for bucket id 9

[8.319s] app ready
ERROR: ld.so: object '/home/nahidalam/IsaacLab/_isaac_sim/exts/omni.isaac.ml_archive/pip_prebundle/torch/lib/libgomp.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
[10.722s] Simulation App Startup Complete
/home/nahidalam/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/cuda/__init__.py:283: UserWarning:
    Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.
    Minimum and Maximum cuda capability supported by this version of PyTorch is
    (8.0) - (12.0)
   
  warnings.warn(
[10.772s] [ext: omni.physx.fabric-107.3.26] startup
[INFO]: Setup complete...