Supporting Nix-built challenge images: three options for the /nix store conflict

[lacraig2]

Splitting this out of #1121 per CONTRIBUTING.md, since it's a design question rather than the bug report itself.

The bug: the workspace store is mounted over /nix in every challenge container, so a challenge image that is itself Nix-built loses its entire userland. Any dockerTools-built image is affected, and there's nothing the image can do about it from inside the container.

This reads as unfinished business from #306. The goal there was that custom images shouldn't need to derive from the challenge base — and mounting the store in achieves that for Dockerfile images, but leaves Nix-built ones as the one case that can't work at all. Since Nix is also the most likely way someone builds a self-contained toolchain image, that's an awkward gap to leave in place.

The useful fact underneath all three options below: Nix store paths are content-addressed and hash-named, so two stores can coexist safely — they just can't both be mounted at /nix. Two different closures can never collide on the same path name with different content.

Cheapest first.

1. Import challenge-image closures into the workspace store

workspace-builder (docker-compose.yml:68-78, workspace/Dockerfile) already has the nix CLI and a writable store bind-mounted from /data/workspace/nix. Add a step that imports a Nix-built challenge image's store closure into that store. The existing single read-only /nix mount then contains both userlands and nothing else changes: no runtime privileges, no new mounts, no OCI hooks, no entrypoint change, no relocation.

Using Nix's own machinery rather than a file copy: pkgs.closureInfo emits a registration file that is exactly the input to nix-store --load-db, and nixpkgs' dockerTools.buildImageWithNixDb is built on this. So the image ships its closure registration at a known path and the import is a copy plus a --load-db, producing properly registered paths so nix-store --verify, GC accounting and dedup all behave.

Why this looks like the best long-term shape for the platform, independent of any one image:

• Disk use improves as adoption grows. Two Nix-built dojos on the same nixpkgs generation share those store paths automatically. Options 2 and 3 leave every image carrying its own full closure.
• It opens the door to thin challenge images. Once the platform can import closures, an image could ship a manifest and let the platform substitute from a binary cache — smaller images, faster pulls.

Things it needs, which seem better flagged up front than found in review:

• Import must be trust-checked. This makes the store shared mutable state fed by image content. Hash-naming prevents accidental collisions, but a hostile image could pre-plant a path whose name matches something legitimate; nix would then treat it as valid and skip substituting the real one — store poisoning across challenges. Mitigations: verify the NAR hash against the path hash on import (what nix-store --import does for signed closures, and the reason trusted-users exists), or give each dojo its own store directory.
• Imported paths need a GC root. workspace/Dockerfile runs nix-collect-garbage -d when NIX_GARBAGE_COLLECT=true, and GC removes store entries not reachable from a root. Each imported closure needs an indirect root or a symlink under /nix/var/nix/gcroots.
• The suid mechanism must not extend to imported closures. The workspace entrypoint chmods u+s over paths listed in $NIX_PROFILE/suid; that must stay scoped to the workspace profile.
• Import has to happen before first container start — i.e. at image registration, which already exists.

2. Union both stores at /nix at container start

Semantically cleanest: lazy, no store growth, no import step. The mount needs the runtime's privileges inside the container's mount namespace before the entrypoint runs, i.e. an OCI createContainer hook. Docker's API won't attach hooks per container, so this means a thin custom runtime shim that injects the hook into config.json and execs runc. There's precedent in that a second runtime is already selected for privileged challenges (docker.py:190), but this is the most infrastructure work of the three.

3. Move the workspace store off /nix

Relocate the deployed workspace store to a different prefix and mount it there, leaving /nix inside the container to the image. Every Nix-built image then works untouched — no per-image step, no detection.

The catch is that the workspace's own binaries have /nix/store baked in: ELF PT_INTERP/DT_RUNPATH, compiled-in string constants, shebangs, .pyc paths. That's fixable with a byte-for-byte in-place rewrite provided the replacement prefix is the same byte length as /nix/store (10 bytes, e.g. /opt/dojonx). Equal length keeps all of those references structurally valid with no patchelf and no per-format handling. We do this for our own image and can hand over the script — ~130 lines of stdlib Python that fails closed if any reference to the old prefix survives.

Concretely: run the rewrite over the deployed copy at /data/workspace/nix after nix build, then change two string literals in docker.py — the mount target (:115-120) and the entrypoint path (:144-148). Builds still go through cache.nixos.org normally, since relocation happens post-build on the deployed copy only.

Detection, if you want it opt-in

For options 1 and 2 the image can declare itself rather than being probed: a label such as pwn.college/nix-store=/nix/store, read from image.attrs["Config"]["Labels"] at create time. There's precedent — docker.py:110-113 already reads image.attrs["Config"] for the image's PATH.

Where we'd land

Our instinct is 3 now, 1 as the direction: 3 is a small diff plus a script we can provide and it unblocks Nix-built images immediately, and it doesn't foreclose doing 1 later. But we're outside-in here and don't want to presume on your roadmap.

Happy to do the work either way — a PR for 3, or the import step plus the image-side registration/label plumbing for 1. Also entirely fine to hear this is lower priority than it looks from where we're standing.