Debugging Minimal Containers

Debugging Minimal Containers

Images built from this project's layouts have no shell - scratch as the final stage means there's no /bin/sh, no busybox, nothing docker exec -it <container> sh could attach to. That's a deliberate consequence of minimizing attack surface (see Security Model), but it means the usual "just exec in and look around" debugging workflow doesn't work. This page covers what does.

nsenter, in depth

What it actually does

Every Linux process belongs to a set of namespaces - mount, UTS (hostname), IPC, network, PID, user, and cgroup - and a container is, mechanically, just a process (or process tree) running inside its own set of these, distinct from the host's. nsenter doesn't attach to a container in any container-runtime-specific sense at all - it's a generic utility (part of util-linux, present on essentially every Linux system) that takes a target PID and joins the calling process into that PID's namespaces, then execs a command inside them. Because it operates purely on namespaces and doesn't need anything from the target's filesystem to run, it works identically whether the target container has a full distro image or, like this project's images, nothing beyond an entrypoint binary and (optionally) the handful of extra files it explicitly needs.

The commands

# Docker
PID=$(docker inspect -f '{{.State.Pid}}' <container>)
sudo nsenter --target "$PID" --mount --uts --ipc --net --pid

This drops you into a new shell process running on the host, but inside the container's mount/UTS/IPC/network/PID namespaces - so ps, ls /, ip addr, and so on all show you the container's view, using tools that exist on your host, not inside the (possibly shell-less) container.

# containerd (no Docker involved)
ctr -n <namespace> tasks ls
sudo nsenter --target <PID> --mount --uts --ipc --net --pid

ctr tasks ls gives you the running task's PID the same way docker inspect does for Docker; the nsenter invocation afterward is identical.

# Kubernetes
kubectl debug node/<node> -it --image=busybox
# then, from inside that debug pod (which runs on the node, in the host's
# root namespaces), find the target container's PID via the node's
# container runtime and nsenter into it the same way.

kubectl debug node/... gives you an authorized, audited session running in the host's own namespaces on that node - from there, the same nsenter technique applies once you've identified the target process's PID via the node's runtime (crictl inspect, ctr, etc., depending on what the cluster runs).

What you can inspect once inside

• /proc/1 - the container's PID 1 (the entrypoint itself): its /proc/1/status, open file descriptors (/proc/1/fd/), and so on.
• /proc/1/mountinfo - exactly what's mounted where inside the container, useful for confirming --read-only/tmpfs settings actually took effect.
• /proc/1/environ (subject to permission - root inside the container's user namespace, or host root, can usually read this; a fully unprivileged caller often can't) - the entrypoint's actual environment variables.
• ip addr, ip route - the container's network namespace, exactly as the container sees it.
• the mounted root filesystem itself, via the --mount namespace join - you can ls, cat, file anything in it using host-provided tools, even though the container itself has none of those tools.

Why nsenter instead of the alternatives

Tool	Needs a shell in the target image?	Scope	When to prefer it
docker exec / podman exec	Yes - execs a new process using the target image's own binaries	one container	normal day-to-day debugging of an image that has a shell
kubectl exec	Yes, same reason	one pod/container	same, in-cluster
nsenter	No - uses the host's own tools, only needs a PID	namespaces of one process (usually the container's PID 1)	any image without a shell (this project's images, distroless, scratch), or when you specifically need namespace/mount/network-level inspection rather than an application-level shell
ctr/crictl (runtime CLIs)	Varies (ctr exec still needs a shell in-image; ctr/crictl alone are for runtime-level inspection: listing tasks, images, state)	one runtime's containers	getting the PID to feed nsenter, or runtime-level state (not a substitute for nsenter - complementary to it)
kubectl debug node/...	N/A - gives you a node-level debug pod, not a target-container shell	the whole node	when you need node-level access (to then nsenter into a specific container) in a cluster where you can't SSH to nodes directly
SSH directly to the host + ps/nsenter	No	whatever you can see from the host	when it's available and authorized - kubectl debug node exists precisely for when it isn't
strace -p <pid> / gdb -p <pid> (from the host)	No	one process, syscall/debugger level	deep single-process debugging (a hang, a crash, a specific syscall failure) - complements nsenter, isn't a replacement: you'd typically nsenter to look around, then strace/gdb a specific PID once you know which one

The short version: docker exec/kubectl exec work by running a new process using the target image's own filesystem - if that filesystem has no shell, there is nothing for them to exec. nsenter sidesteps this entirely by joining the target's namespaces while still using the host's binaries to actually do the inspecting. For any image built from this project's layouts - which by design never contain a shell - nsenter (or the runtime-CLI-plus-nsenter combination) is the only one of these that works at all, not merely the more convenient option.

nsenter's own cost - read this before reaching for it

Great power, usual disclaimer. nsenter requires host root or CAP_SYS_ADMIN-equivalent privileges, and once inside, it exposes namespaces, processes, mounts, network state, and potentially secrets (environment variables, mounted files) with the same visibility the container's own PID 1 has. This is real privilege, not a read-only inspection tool. Restrict its use to:

• incident responders,
• audited hosts (i.e. somewhere with logging/session-recording on privileged access),
• approved maintenance windows.

For normal, non-emergency application-level debugging of an image that does have a shell, prefer docker exec/kubectl exec - they're scoped to exactly that one container and don't require host-level privilege. Reach for nsenter specifically when you're debugging namespaces, mounts, or networking themselves, or the target image has no shell at all - not as a default habit.

Debugging a microVM instead

None of the above applies to MicroVM Support - there is no shared host kernel to nsenter into; the guest is a genuinely separate kernel. run-microvm.sh streams the guest's console live to your terminal and prints the full console log on any boot failure - that log (kernel boot messages, then whatever the entrypoint itself logs to its own console/stdout) is the primary debugging surface. There is currently no built-in equivalent of "exec a shell inside the running guest" - the guest image, like the container images this project produces, has no shell either, and the tiny cmd/microvm-init PID 1 doesn't provide one.

Inspecting a layout before you ever run it

You don't need a running container or VM at all to check a layout is what you expect - scripts/ci/verify-oci-layout.py (the same one CI itself uses; see Scripts Reference) runs standalone against any layout directory:

python3 scripts/ci/verify-oci-layout.py ./oci-image

It independently re-derives and checks every descriptor, digest, blob, and layer path - the same verification the Dockerfile's verify stage and CI both rely on - without needing Docker, Podman, or root at all.