Gateway crash on v0.0.16: k3s fatal error 'flag provided but not defined: -resolv-conf'

[yanggf8]

Description

OpenShell gateway v0.0.16 fails to start. The k3s process inside the cluster container crashes immediately with:

time="2026-03-31T06:17:23Z" level=fatal msg="Error: flag provided but not defined: -resolv-conf"

The container exits before the openshell namespace is ever created, causing nemoclaw onboard to fail with "K8s namespace not ready" after all retry attempts.

Environment

• OS: Ubuntu (WSL2 on Windows)
• Docker: 29.3.0
• OpenShell CLI: 0.0.16
• OpenShell image: ghcr.io/nvidia/openshell/cluster:0.0.16
• k3s version in image: v1.35.2+k3s1

Steps to reproduce

docker run -d --name test --privileged ghcr.io/nvidia/openshell/cluster:0.0.16
docker logs test
# Container exits immediately with the -resolv-conf fatal error

Or via NemoClaw:

node bin/nemoclaw.js onboard
# Fails at step 2/7 "Starting OpenShell gateway"

Root cause

The cluster entrypoint passes -resolv-conf to k3s, but k3s v1.35.2 no longer recognizes this flag. It was likely renamed or removed in a recent k3s release.

Workaround

Downgrading to v0.0.15 works:

OPENSHELL_CLUSTER_IMAGE=ghcr.io/nvidia/openshell/cluster:0.0.15 openshell gateway start --name nemoclaw

Expected behavior

Gateway should start successfully and create the openshell k8s namespace.

[yanggf8]

Additional finding: overlayfs snapshotter failure on WSL2

Even after working around the -resolv-conf flag issue (by passing server as the subcommand), k3s gets stuck in a loop unable to initialize containerd:

Waiting to retrieve agent configuration; server is not ready: "overlayfs" snapshotter cannot be enabled
for "/var/lib/rancher/k3s/agent/containerd", try using "fuse-overlayfs" or "native": failed to mount
overlay: mount source: "overlay", target: "...", fstype: overlay, flags: 0, data: "...", err: invalid argument

WSL2's kernel does not support nested overlayfs (overlay-on-overlay) when Docker itself is already using overlayfs as its storage driver. k3s's embedded containerd defaults to overlayfs and fails.

Suggested fix

The entrypoint should detect when overlayfs is unavailable and fall back to fuse-overlayfs or native snapshotter:

# In cluster-entrypoint.sh, before exec:
if ! mount -t overlay overlay -o lowerdir=/tmp/test-lower1:/tmp/test-lower2,upperdir=/tmp/test-upper,workdir=/tmp/test-work /tmp/test-merged 2>/dev/null; then
    EXTRA_KUBELET_ARGS="$EXTRA_KUBELET_ARGS --snapshotter=native"
fi

Or pass --snapshotter=native via k3s server args when running inside Docker (detected by /.dockerenv existence).

Summary of issues

So there are two distinct problems blocking WSL2:

1. -resolv-conf as global flag — k3s v1.35.2 rejects it outside the server subcommand (original report)
2. overlayfs snapshotter — nested overlayfs fails on WSL2, needs native or fuse-overlayfs fallback

Environment

• WSL2 kernel: 6.6.87.2-microsoft-standard-WSL2
• Docker: 29.3.0 (Docker Desktop WSL backend)
• Docker storage driver: overlay2

[cluster2600]

Verification: cluster image rebuilt and tested

Built the cluster image from branch 696-k3s-resolv-conf-flag/mc on an NVIDIA DGX Spark (aarch64, Docker 29.1.3) and ran it with docker run --privileged:

Before (v0.0.16 stock):

time="…" level=fatal msg="Error: flag provided but not defined: -resolv-conf"

Container exits immediately.

After (this fix):

time="2026-03-31T07:22:52Z" level=info msg="Starting k3s v1.35.2+k3s1 (13563feb)"
time="2026-03-31T07:22:52Z" level=info msg="Configuring sqlite3 database connection pooling…"
time="2026-03-31T07:22:52Z" level=info msg="Database tables and indexes are up to date"
time="2026-03-31T07:22:52Z" level=info msg="Kine available at unix://kine.sock"
…

k3s starts successfully, generates all certificates, and begins agent configuration. Zero fatal messages in the container logs. Container remains running.

The fix is a one-liner in the entrypoint: --resolv-conf= → --kubelet-arg=resolv-conf=. Regression test added to catch this if the flag format changes again.

Build: mise run docker:build:cluster (5m 25s)
Image: openshell/cluster:dev (sha256:167550c6…)
Test: sh deploy/docker/test-cluster-entrypoint.sh — 6/6 assertions pass

[drew]

Please make sure to launch the cluster using the openshell gateway start command. Direct launching via the docker container is currently unsupported.