[BUG] nemoclaw onboard forces --gpu on WSL2, sandbox DOA (workaround included)

[mattezell]

Environment

• OS: Windows 11 → WSL2 Ubuntu 24.04 LTS
• Docker: Docker Desktop 4.x with WSL2 integration
• GPU: NVIDIA RTX 5090 Laptop (also confirmed on RTX 5070 Ti — see NVIDIA Forums #363769
• NemoClaw: v0.0.7
• OpenShell: v0.0.7
• OpenClaw: 2026.3.11

Problem

nemoclaw onboard detects nvidia-smi on WSL2 and forces --gpu on both openshell gateway start and openshell sandbox create. On WSL2 with Docker Desktop, the GPU cannot be passed through to the k3s cluster inside the gateway container. The sandbox reports as created but is immediately dead — every subsequent command returns "sandbox not found".

There is no --no-gpu flag, no environment variable to skip GPU detection, and no config option to override this behavior.

Steps to Reproduce

1. Install NemoClaw on WSL2 with any NVIDIA GPU present
2. Run nemoclaw onboard
3. openshell doctor check passes all prerequisites
4. Step [3/7] "Creating sandbox" reports success
5. Immediately after, any sandbox command returns status: NotFound

Expected Behavior

--gpu should be optional. Either:

• Add a --no-gpu or --skip-gpu flag to nemoclaw onboard
• Detect WSL2 via /proc/version and skip GPU passthrough automatically
• Fall back gracefully when GPU passthrough fails instead of creating a dead sandbox

Workaround

Bypass nemoclaw onboard entirely and drive openshell directly without --gpu:

# 1. Clean stale state (critical — failed onboard runs corrupt k3s)
openshell sandbox delete <name> 2>/dev/null
openshell gateway destroy --name nemoclaw 2>/dev/null
docker volume rm openshell-cluster-nemoclaw 2>/dev/null

# 2. Start gateway WITHOUT --gpu
openshell gateway start --name nemoclaw

# 3. Create provider BEFORE sandbox (credentials injected at creation time)
openshell provider create --name nvidia-nim --type nvidia \
  --credential NVIDIA_API_KEY=nvapi-xxx

# 4. Set inference route
openshell inference set --provider nvidia-nim \
  --model nvidia/nemotron-3-super-120b-a12b

# 5. Create sandbox WITHOUT --gpu
openshell sandbox create --name my-sandbox --from openclaw

Inside the sandbox, openclaw onboard must use https://inference.local/v1 as the base URL (Custom Provider → OpenAI-compatible) since the sandbox blocks direct outbound network.

Full workaround documentation and automated deploy scripts for WSL2 and macOS: thenewguardai/tng-nemoclaw-quickstart — WSL2-WORKAROUND.md

Additional Notes

• Stale gateway state from failed nemoclaw onboard runs requires docker volume rm openshell-cluster-nemoclaw to fully clear — openshell gateway destroy alone is not sufficient
• Provider must be created before sandbox creation, otherwise the sandbox lacks inference credentials
• ANTHROPIC_API_KEY in the sandbox environment causes OpenClaw to silently default to Claude regardless of configured model

[mattezell]

PR: #209

[tyeth]

Just to add my llm (cc opus4.6) added a fix to the nvidia-ctk (nvidia container tool kit) to correctly support the GPU via WSL.

issue.md

Issue / Resolution Details

NemoClaw/OpenShell GPU Failure on WSL2 Dual-GPU Device

Environment

• OS: Windows WSL2 (Linux 6.6.87.2-microsoft-standard-WSL2)
• GPUs: Dual GPU laptop - integrated (non-NVIDIA) + NVIDIA GeForce RTX 5070 (8GB)
• Driver: NVIDIA 595.71, CUDA 13.2
• Docker: 29.1.3
• OpenShell: 0.0.7 (gateway: nemoclaw)
• k3s: v1.35.2+k3s1
• NVIDIA Device Plugin Helm Chart: v0.18.2

Symptom

OpenShell gateway starts successfully with --gpu flag. All core pods run fine.
However, the nvidia-device-plugin DaemonSet has 0/0 desired/ready replicas, meaning
no nvidia.com/gpu resource is ever advertised on the node. GPU sandbox creation fails
because Kubernetes has no GPU capacity to schedule against.

Root Cause

WSL2 virtualises GPU access through /dev/dxg (DirectX GPU abstraction) rather than
exposing native /dev/nvidia* device nodes. This causes three cascading failures:

1. No native NVIDIA device nodes

Inside the gateway container:

/dev/dxg          exists (WSL2 DirectX GPU shim)
/dev/nvidia*      does NOT exist
/proc/driver/nvidia/gpus/*  does NOT exist

nvidia-smi works at the gateway container level (via the WSL2 driver shim), but
the nvidia container runtime's auto mode uses legacy injection which fails to
properly pass GPU libraries into k3s pods.

2. NFD cannot detect NVIDIA PCI device

Node Feature Discovery sees only Microsoft Hyper-V virtual PCI:

feature.node.kubernetes.io/pci-1414.present=true   (Microsoft Hyper-V)
feature.node.kubernetes.io/pci-10de.present=true    MISSING (NVIDIA vendor ID)

WSL2 does not expose PCI topology to the guest kernel, so NFD never discovers
the physical NVIDIA GPU.

3. DaemonSet affinity never satisfied

The nvidia-device-plugin DaemonSet requires one of these node labels:

• feature.node.kubernetes.io/pci-10de.present=true — not present (WSL2 hides PCI)
• feature.node.kubernetes.io/cpu-model.vendor_id=NVIDIA — node has AMD CPU
• nvidia.com/gpu.present=true — never set (GFD also has 0 pods for the same reason)

Result: 0 desired replicas. The device plugin never runs.

4. Missing libdxcore.so in CDI spec

Even when the DaemonSet is forced to schedule (via manual label), the device plugin
crashes with Failed to initialize NVML: Not Supported. This is because:

• The nvidia runtime's auto mode uses legacy injection which doesn't mount
  libdxcore.so into pods
• libdxcore.so is the critical WSL2 library that bridges Linux NVML to the
  Windows DirectX GPU Kernel via /dev/dxg
• Without it, NVML nvmlInit() fails even though /dev/dxg and the driver
  store libraries are present

Failure Chain

WSL2 /dev/dxg (not /dev/nvidia*)
  -> NFD can't see PCI vendor 10de
    -> Node never gets pci-10de.present label
      -> nvidia-device-plugin DaemonSet affinity not satisfied (0 pods)

Even when forced to schedule:
  -> nvidia runtime auto mode uses legacy injection
    -> libdxcore.so not injected into pods
      -> NVML init fails: "Not Supported"
        -> device plugin crashes, no nvidia.com/gpu resource advertised

Fix (Verified Working)

Two changes are needed to make GPU work on WSL2:

Step 1: Patch the CDI spec to include libdxcore.so

The nvidia-ctk cdi generate command auto-detects WSL mode but misses libdxcore.so.
Add it to the generated /var/run/cdi/nvidia.yaml:

# Add to containerEdits.mounts:
- hostPath: /usr/lib/x86_64-linux-gnu/libdxcore.so
  containerPath: /usr/lib/x86_64-linux-gnu/libdxcore.so
  options:
    - ro
    - nosuid
    - nodev
    - rbind
    - rprivate

Also update the update-ldcache hook to include the directory:

# Add --folder /usr/lib/x86_64-linux-gnu to the update-ldcache hook args

Step 2: Switch nvidia runtime from auto to cdi mode

In /etc/nvidia-container-runtime/config.toml:

[nvidia-container-runtime]
mode = "cdi"     # was "auto"

This makes the nvidia container runtime use the CDI spec (which includes
/dev/dxg, WSL driver store libs, AND libdxcore.so) instead of the
legacy injection path (which fails on WSL2).

Step 3: Add the missing PCI label

openshell doctor exec -- kubectl label node <node> \
  feature.node.kubernetes.io/pci-10de.present=true

This satisfies the DaemonSet's node affinity requirement since WSL2's NFD
can't detect the GPU via PCI.

Result

After all three steps:

• nvidia-device-plugin DaemonSet: 1/1 Running
• gpu-feature-discovery DaemonSet: 1/1 Running
• Node capacity: nvidia.com/gpu: 1
• nvidia-smi works inside pods with runtimeClassName: nvidia
• RTX 5070 (8GB) fully accessible for GPU sandboxes

Upstream Fix Needed

For a permanent solution, OpenShell's cluster-entrypoint.sh should:

1. Detect WSL2 (check for /dev/dxg or kernel version containing WSL2)
2. Generate CDI spec with nvidia-ctk cdi generate (already auto-detects WSL mode)
3. Patch the CDI spec to include libdxcore.so (bug in nvidia-ctk)
4. Set runtime mode to cdi in /etc/nvidia-container-runtime/config.toml
5. Add the PCI label to the k3s node (or configure the device plugin Helm chart
   to not require NFD labels on WSL2)

The root bug is in nvidia-ctk cdi generate which logs
"Could not locate libdxcore.so: pattern libdxcore.so not found" but the library
IS present at /usr/lib/x86_64-linux-gnu/libdxcore.so. This should be reported
upstream to the NVIDIA Container Toolkit project.

Related:
NVIDIA/OpenShell#404
NVIDIA/nvidia-container-toolkit#1739

[mattezell]

Excellent root cause analysis from @tyeth — the full failure chain (NFD can't see PCI on WSL2 → device plugin never schedules → legacy runtime misses libdxcore.so) explains exactly why --gpu is broken on WSL2.

PR #209 addresses this at the NemoClaw layer by detecting WSL2 and skipping --gpu entirely so users can get working sandboxes immediately via cloud inference. The deeper fix in OpenShell (#404) to support actual GPU passthrough on WSL2 via CDI mode is the right long-term path — once that lands, NemoClaw's WSL2 detection could be updated to pass --gpu again.

[tyeth-ai-assisted]

PR for openshell to get GPU support: NVIDIA/OpenShell#411
Seems to run the models locally, but using the GPU not NPU and not shared unified memory (although does tolerate the larger models). The screenshot is a large while afterwards so ignore the time / cpu usage and low recent GPU usage.
I might be pushing it with the 42GB nemotron model (hence the truncated outputs maybe).

[mattezell]

Update: We got full GPU passthrough working on WSL2. RTX 5090, nvidia-smi inside the sandbox, device plugin running.

The CDI pipeline fix from OpenShell#404 works but required additional discoveries: UUID device entries in the CDI spec, enable_cdi=true in containerd config, and a containerd restart + pod reschedule cycle.

PR #209 (skip --gpu on WSL2) is still the right short-term fix for NemoClaw — it unblocks everyone immediately via cloud inference. The GPU path requires 11 steps of CDI surgery that should eventually be handled by OpenShell's cluster-entrypoint.sh.

Automated GPU deploy script: https://github.com/thenewguardai/tng-nemoclaw-quickstart/blob/main/scripts/wsl2-gpu-deploy.sh

[tyeth-ai-assisted]

100% agree, let's not pollute this codebase with hacky fixes for upstream issues, but glad we see working GPUs.

Did a PR to unblock anyone wanting a PR from this repo: #254

[manoharreddyporeddy]

100% agree, let's not pollute this codebase with hacky fixes for upstream issues, but glad we see working GPUs.

Did a PR to unblock anyone wanting a PR from this repo: #254

Just checking, is below related to this ?
#290