Problem
When envbox runs inside a Kubernetes pod, the inner Docker daemon creates container cgroups at /sys/fs/cgroup/docker/<id>/... — a sibling of the pod's cgroup tree (/kubepods.slice/.../<pod>.slice/<envbox-container>.scope/...) rather than a descendant. There's no path through the cgroup tree from the inner container's cgroup to the pod's cgroup.
This breaks any host-level tool that uses the cgroup hierarchy for pod attribution.
Symptoms
We hit this with Cilium Tetragon: processes running inside a Coder workspace (i.e. inside the inner Docker container) generate process_exec events with no pod field — no namespace, no name, no container ID, no pod_labels. Processes running directly in the envbox container (e.g. sysbox-fs, dockerd itself) ARE correctly attributed.
This means we cannot answer "which workspace ran this process?" from the security audit log without manually walking the host process tree by PID, which is impractical at scale.
The same issue would affect Falco, custom eBPF agents, or any other tool that uses cgroup hierarchy for k8s pod resolution.
Root cause
The inner Docker daemon (started in cli/docker.go::dockerCmd) runs with no --cgroup-parent. On cgroupv2 it creates /sys/fs/cgroup/docker/<id>/... at the cgroup root rather than under envbox's own .scope.
Setting CgroupParent on the inner container via the Docker API alone doesn't solve this either: the cgroupv2 "no internal processes" rule prevents creating processes-bearing child cgroups under a parent that already has processes (envbox's .scope always has processes from sysbox-mgr, sysbox-fs, dockerd, etc.).
Why it can't be fixed in the observability tool alone
We considered fixing this in Tetragon — e.g. adding a fallback that walks up the cgroup hierarchy when a container ID lookup fails (similar to cilium/tetragon#3381). But:
- Even with that, Tetragon needs a tracked ancestor in the cgroup tree to walk up to. Today the inner Docker daemon's containers aren't placed anywhere under a tracked envbox container, so there's nothing to walk up to.
- Falco / custom eBPF agents have the same limitation. Fixing this in envbox makes Coder workspaces visible to all cgroup-aware tooling at once.
Proposed solution
Wrap the dockerd invocation with unshare --cgroup + a fresh cgroup2 mount + cgroup-delegation setup (move existing processes into a sibling /init cgroup, enable cgroup.subtree_control). This is the canonical pattern moby uses for running cgroup-v2 Docker inside a privileged container. Isovalent (Cilium/Tetragon vendor) specifically recommended this for our scenario in moby/moby#45378 (comment).
After the wrapper runs, dockerd's view of /sys/fs/cgroup is rooted at the envbox container's own cgroup — all inner container cgroups become descendants of that scope on the host. Cgroup-walking tools can then trace inner-container processes back to the parent pod.
Environment
- containerd 2.2.1
- Linux 6.8 / Ubuntu 22.04
- cgroup v2 (unified hierarchy)
- envbox with sysbox-runc as the inner-container runtime
- Verified with Cilium Tetragon v1.18.1 (with
--enable-cgtrackerid=true)
Problem
When envbox runs inside a Kubernetes pod, the inner Docker daemon creates container cgroups at
/sys/fs/cgroup/docker/<id>/...— a sibling of the pod's cgroup tree (/kubepods.slice/.../<pod>.slice/<envbox-container>.scope/...) rather than a descendant. There's no path through the cgroup tree from the inner container's cgroup to the pod's cgroup.This breaks any host-level tool that uses the cgroup hierarchy for pod attribution.
Symptoms
We hit this with Cilium Tetragon: processes running inside a Coder workspace (i.e. inside the inner Docker container) generate
process_execevents with nopodfield — no namespace, no name, no container ID, nopod_labels. Processes running directly in the envbox container (e.g.sysbox-fs, dockerd itself) ARE correctly attributed.This means we cannot answer "which workspace ran this process?" from the security audit log without manually walking the host process tree by PID, which is impractical at scale.
The same issue would affect Falco, custom eBPF agents, or any other tool that uses cgroup hierarchy for k8s pod resolution.
Root cause
The inner Docker daemon (started in
cli/docker.go::dockerCmd) runs with no--cgroup-parent. On cgroupv2 it creates/sys/fs/cgroup/docker/<id>/...at the cgroup root rather than under envbox's own.scope.Setting
CgroupParenton the inner container via the Docker API alone doesn't solve this either: the cgroupv2 "no internal processes" rule prevents creating processes-bearing child cgroups under a parent that already has processes (envbox's.scopealways has processes from sysbox-mgr, sysbox-fs, dockerd, etc.).Why it can't be fixed in the observability tool alone
We considered fixing this in Tetragon — e.g. adding a fallback that walks up the cgroup hierarchy when a container ID lookup fails (similar to cilium/tetragon#3381). But:
Proposed solution
Wrap the dockerd invocation with
unshare --cgroup+ a freshcgroup2mount + cgroup-delegation setup (move existing processes into a sibling/initcgroup, enablecgroup.subtree_control). This is the canonical pattern moby uses for running cgroup-v2 Docker inside a privileged container. Isovalent (Cilium/Tetragon vendor) specifically recommended this for our scenario in moby/moby#45378 (comment).After the wrapper runs, dockerd's view of
/sys/fs/cgroupis rooted at the envbox container's own cgroup — all inner container cgroups become descendants of that scope on the host. Cgroup-walking tools can then trace inner-container processes back to the parent pod.Environment
--enable-cgtrackerid=true)