inference.local returns 404 for /v1/chat/completions and /v1/responses despite configured NVIDIA provider

[Plummere]

Summary

On a fresh local OpenShell gateway, inference.local inside a sandbox consistently returns 404 page not found for both:

• POST /v1/chat/completions (OpenAI-style)
• POST /v1/responses (per the docs’ “Verify from sandbox” example)

This happens even though:

• Gateway inference is configured with a valid NVIDIA provider and Nemotron 3 model.
• The sandbox proxy does intercept these calls and routes them through navigator_router to https://integrate.api.nvidia.com/v1 with the expected paths.

This effectively breaks the documented https://inference.local inference routing path.

---

Environment

• Host: Windows 11 + WSL2 (Ubuntu, Docker Engine in WSL)
• OpenShell CLI: installed via uv pip install openshell --pre from internal nv-shared-pypi
• Docker: logged in to ghcr.io with PAT (including SSO) and able to pull ghcr.io/nvidia/openshell/* images
• Gateway: started via openshell gateway start on WSL host
• Inference backend: NVIDIA Inference API, Nemotron 3 Nano 30B (works directly from WSL with my key)

---

Steps to Reproduce

1. Start gateway (host / WSL)

# In WSL
uv venv .venv
source .venv/bin/activate
uv pip install openshell --upgrade --pre \
  --index-url [https://urm.nvidia.com/artifactory/api/pypi/nv-shared-pypi/simple](https://urm.nvidia.com/artifactory/api/pypi/nv-shared-pypi/simple)
openshell gateway start

→ Gateway ready, e.g. Endpoint: https://127.0.0.1:8080

2. Configure NVIDIA provider + Nemotron 3 inference (host / WSL)

export NVIDIA_API_KEY="YOUR_INFERENCE_API_KEY"  # same key that works directly against inference-api.nvidia.com
openshell provider create \
  --name nvidia-prod \
  --type nvidia \
  --from-existing
openshell inference set \
  --provider nvidia-prod \
  --model nvidia/nvidia/Nemotron-3-Nano-30B-A3B
openshell inference get

Output:

Gateway inference:
  Provider: nvidia-prod
  Model:    nvidia/nvidia/Nemotron-3-Nano-30B-A3B
  Version:  1
System inference:
  Not configured

3. Create and connect to sandbox

openshell sandbox create --name test
openshell sandbox list   # wait until Ready
openshell sandbox connect test

prompt: sandbox@test:~$

4. Test /v1/chat/completions from sandbox

pip install openai
python - << 'EOF'
from openai import OpenAI
client = OpenAI(
    base_url="[https://inference.local/v1](https://inference.local/v1)",
    api_key="dummy",  # ignored by OpenShell; routing uses configured provider
)
resp = client.chat.completions.create(
    model="anything",  # should be rewritten to configured model
    messages=[{"role": "user", "content": "Hello from OpenShell sandbox!"}],
    temperature=0.7,
    max_tokens=128,
)
print(resp.choices[0].message.content)
EOF

Actual result:

openai.NotFoundError: 404 page not found

5. Test /v1/responses from sandbox (per docs)

pip install requests
python - << 'EOF'
import requests, json
url = "[https://inference.local/v1/responses](https://inference.local/v1/responses)"
payload = {
    "instructions": "You are a helpful assistant.",
    "input": "Hello from OpenShell sandbox!",
}
resp = requests.post(url, json=payload, timeout=60)
print("Status:", resp.status_code)
print("Body:", resp.text[:500])
EOF

Actual result:

Status: 404
Body: 404 page not found

---

What I Expected

Given:

• openshell inference get shows a configured NVIDIA provider + Nemotron model.
• Docs state that /v1/chat/completions and /v1/responses are recognized inference patterns for inference.local.
• The “Verify the Endpoint from a Sandbox” example uses POST /v1/responses.

I expected:

• POST https://inference.local/v1/chat/completions and
• POST https://inference.local/v1/responses

to return a normal model response (HTTP 200 + JSON) from inside the sandbox.

---

What Actually Happens

• Both endpoints return a simple 404 page not found from inside the sandbox.
• There is no obvious configuration error on the host/sandbox side (gateway, provider, and inference are all reported as healthy).

---

Relevant Logs (openshell logs -g openshell)

1773260787.772 INFO  Fetching inference route bundle from gateway endpoint=[https://openshell.openshell.svc.cluster.local:8080](https://openshell.openshell.svc.cluster.local:8080)
1773260787.822 INFO  Loaded inference route bundle revision=6ce65bfa03d7bff0 route_count=1
1773260787.822 INFO  Inference routing enabled with local execution route_count=1
1773260787.823 INFO  Proxy listening (tcp) addr=10.200.0.1:3128

... sandbox [navigator_sandbox::proxy] Intercepted inference request, routing locally kind=chat_completion method=POST path=/v1/chat/completions protocol=openai_chat_completions
1773260870.962 INFO  routing proxy inference request endpoint=[https://integrate.api.nvidia.com/v1](https://integrate.api.nvidia.com/v1) method=POST path=/v1/chat/completions protocols=openai_chat_completions,openai_completions,openai_responses,model_discovery

... sandbox [navigator_sandbox::proxy] Intercepted inference request, routing locally kind=responses method=POST path=/v1/responses protocol=openai_responses
1773261095.914 INFO  routing proxy inference request endpoint=[https://integrate.api.nvidia.com/v1](https://integrate.api.nvidia.com/v1) method=POST path=/v1/responses protocols=openai_chat_completions,openai_completions,openai_responses,model_discovery

Notes:

• The proxy does intercept inference.local and classifies both /v1/chat/completions and /v1/responses as inference requests.
• navigator_router is invoked with endpoint=https://integrate.api.nvidia.com/v1 and path=/v1/....
• Despite this, the sandbox receives 404 page not found for both URLs.

Separately, I’ve confirmed that my NVIDIA Inference API key + Nemotron 3 model work fine directly from WSL against https://inference-api.nvidia.com/v1/chat/completions with the same model ID.

---

Questions

• Is integrate.api.nvidia.com/v1 the intended upstream endpoint for the nvidia provider in this build?
• Should the router be constructing /v1/chat/completions and /v1/responses against that base as-is, or is there a known issue with the current OpenShell server image’s inference routing?
• Is there a different path or configuration I should be using to exercise inference.local from inside a sandbox on the current version?

Happy to provide more logs or try a specific build/tag if that helps narrow it down.

[johntmyers]

Please make sure you're using the right model ID for the right endpoint. From what I can tell:

integrate.api.nvidia.com uses nvidia/nemotron-3-nano-30b-a3b

inference-api.nvidia.com uses nvidia/nvidia/Nemotron-3-Nano-30B-A3B

The default URL for the nvidia provider is integrate.api.nvidia.com so I think you will need to try updating your provider command to:

openshell provider create \
  --name nvidia-prod \
  --type nvidia \
  --credential NVIDIA_API_KEY \
  --config NVIDIA_BASE_URL=https://inference-api.nvidia.com/v1

[Plummere]

Thank you John! good catch on the model ID / base URL pairing.

I retried locally using the explicit NVIDIA Inference API base URL and the matching model ID:

openshell provider create \
  --name nvidia-prod \
  --type nvidia \
  --credential NVIDIA_API_KEY \
  --config NVIDIA_BASE_URL=https://inference-api.nvidia.com/v1

openshell inference set \
  --provider nvidia-prod \
  --model nvidia/nvidia/Nemotron-3-Nano-30B-A3B

That changed the failure mode, but it still does not work end-to-end in my local setup.

What I’m seeing now:

• inference.local from inside the sandbox reaches the OpenShell gateway path.
• The gateway pod then fails trying to reach inference-api.nvidia.com.
• The root cause appears to be pod-level DNS resolution, not the model ID anymore.

A few concrete datapoints from the local k3s/OpenShell environment:

• Inside openshell/openshell-0, internal service DNS works:
  • kubernetes.default.svc.cluster.local resolves
  • openshell.openshell.svc.cluster.local resolves
• But external DNS does not:
  • getent hosts inference-api.nvidia.com returns nothing
  • openssl s_client -connect inference-api.nvidia.com:443 ... fails with Temporary failure in name resolution

CoreDNS is forwarding via /etc/resolv.conf, and CoreDNS logs show repeated timeouts to public resolvers:

• 8.8.8.8:53
• 8.8.4.4:53

So at this point it looks like there are two separate issues:

1. The earlier integrate vs inference-api / model ID mismatch, and
2. A local pod/CoreDNS external DNS resolution problem in the OpenShell k3s deployment.

If helpful, I can add the exact DNS/CoreDNS logs to the issue as a follow-up. Also curious whether local OpenShell is expected to inherit host DNS/VPN resolver settings rather than forwarding to public resolvers in this environment.

[johntmyers]

Thanks for the thorough writeup! As John mentioned, the first issue was the integrate.api.nvidia.com vs inference-api.nvidia.com base URL / model ID mismatch. The second issue (external DNS failing inside the cluster) looks like a Docker/WSL2 DNS problem rather than an OpenShell bug. Here are some tips to narrow it down:

Diagnosing the DNS chain

1. Check if the DNS proxy initialized correctly:

openshell doctor exec -- cat /etc/rancher/k3s/resolv.conf

If this shows 8.8.8.8 / 8.8.4.4 instead of a 172.x.x.x address, the iptables DNS proxy failed to set up and fell back to public resolvers. Check the entrypoint logs for details:

openshell doctor logs --lines 50

Look for setup_dns_proxy messages or warnings.

2. Test DNS from inside the container directly (bypassing CoreDNS/k3s):

openshell doctor exec -- nslookup inference-api.nvidia.com

3. Check WSL2's own DNS resolution (from your WSL terminal, outside OpenShell):

cat /etc/resolv.conf
nslookup inference-api.nvidia.com

If WSL2's own DNS is broken or pointing at a nameserver that doesn't work for the Docker network namespace, everything downstream will fail.

Likely fixes

Explicit Docker DNS config — If Docker's embedded DNS isn't inheriting working resolvers, you can force it in /etc/docker/daemon.json:

{
  "dns": ["8.8.8.8", "8.8.4.4"]
}

Then restart Docker and run openshell gateway start again. On native Docker Engine in WSL2 (vs Docker Desktop), outbound UDP should route correctly through WSL2's virtual NIC, so this has a good chance of working.

WSL2 networking mode — Newer WSL2 versions support networkingMode=mirrored in .wslconfig, which changes how DNS and networking work entirely. If you're using mirrored mode, Docker's assumptions about 127.0.0.11 and iptables chains may not hold. Worth checking your .wslconfig to see if that's set.

---

The openshell doctor commands above should quickly pinpoint where the DNS chain is breaking. Let us know what you find!

[Plummere]

Thanks — I tested the Docker DNS override path, and it looks like that is not the layer OpenShell/k3s is using for its generated resolver config.

What I tested

I created /etc/docker/daemon.json with:

{
  "dns": ["10.255.255.254"]
}

Then I:

1. restarted Docker
2. destroyed the OpenShell gateway
3. started the OpenShell gateway fresh

Result

After the fresh recreate:

openshell doctor exec -- cat /etc/rancher/k3s/resolv.conf

still shows:

nameserver 8.8.8.8
nameserver 8.8.4.4

So /etc/rancher/k3s/resolv.conf is being regenerated during OpenShell/k3s startup and is not inheriting the Docker daemon DNS override in my environment.

At the same time:

openshell doctor exec -- nslookup inference-api.nvidia.com

still succeeds via Docker’s embedded resolver (127.0.0.11) and returns the expected internal 10.48.x.x addresses.

Current picture

At this point the split looks like:

• WSL host DNS works
• Docker/container DNS works
• OpenShell/k3s resolver bootstrap does not — it keeps writing 8.8.8.8 / 8.8.4.4 into /etc/rancher/k3s/resolv.conf

That seems to be the actual broken link behind the inference.local failure in my setup.

Question

Is there a specific OpenShell bootstrap path or config source that determines how /etc/rancher/k3s/resolv.conf is generated?

Right now, even a fresh gateway recreate after setting Docker daemon DNS still ends up with public Google DNS in the k3s resolver file.

[johntmyers]

I'm unable to reproduce. Could you try the debug-navigator-cluster skill (will be re-named soon) with your running deployment and see what it weeds out and report back? I suspect we'll need to see the iptables that were created? I suspect that the discovery we try to do isn't working w/ your WSL2 + Docker setup.