feat(incus-vm): Incus VM support, k3s/template fixes, docs#9
Open
bpmct wants to merge 10 commits into
Open
Conversation
Ports PR bpmct/coder-nixos#4 to coder/box. ### NixOS fixes - **`hosts/incus-vm/`** — new template host for any Incus VM provisioned by the incus-vm Coder template. `incus-vm.nix` handles QEMU guest agents, networkd DHCP on enp5s0, and disables the desktop stack. `default.nix` imports incus-vm.nix plus the two runtime files written by the provisioner (`/etc/nixos/incus.nix`, `/etc/nixos/coder.nix`). `README.md` covers manual setup and how the provisioner flow works. - **`configuration.nix`** — two bugs fixed during Incus VM testing: 1. `coder-init-admin`: bare `hostname -s` fails in systemd units (not in PATH). Use `${pkgs.nettools}/bin/hostname -s` (fully qualified). 2. `coder-template-sync` activation script: `mkdir -p "$STATE_DIR"` runs as root but terraform runs as `coder`. Add `chown coder:coder "$STATE_DIR"` so the coder user can write `.terraform` state into that directory. Also cleans up the mangled single-line `coder-reset` and `coder-workspace-reaper` shell commands into readable multiline form. - **`nixos/k3s-sysbox.nix`** — add `rsync` to `environment.systemPackages`. `sysbox-mgr` checks for rsync at startup; if absent it exits immediately: `preflight check failed: rsync is not installed on host` and pods stay stuck in ContainerCreating. ### k3s-sysbox template fixes - **`coderd/templates/k3s-sysbox/main.tf`**: - `local.kubectl`: was a hardcoded Nix store path (`/nix/store/-k3s-.../bin/k3s`). Store hashes change on every nixos-rebuild that touches the k3s derivation. Use `/run/current-system/sw/bin/k3s` (stable per-generation symlink). - `hostAliases`: was `[{}]` (empty object), which Kubernetes rejects with `Invalid value: "": must be a valid IP address`. Now conditional: entry only added when `coder_lan_ip` is non-empty. ### Docs - **`coderd/templates/coder-cli/README.md`** — new; was the only template without one.
- nixos/k3s-sysbox.nix: add rsync to systemPackages; sysbox-mgr does a
preflight check for rsync and exits if absent, leaving pods stuck in
ContainerCreating
- coderd/templates/k3s-sysbox/main.tf: replace hardcoded Nix store path
for kubectl with /run/current-system/sw/bin/k3s (stable across rebuilds)
- coderd/templates/k3s-sysbox/main.tf: fix hostAliases — was [{}] (empty
object) which k8s rejects; now only added when coder_lan_ip is non-empty
…ership
Two bugs found during Incus VM testing:
1. coder-init-admin: bare `hostname -s` fails in systemd units because
/usr/bin is not in PATH. Use ${pkgs.nettools}/bin/hostname -s.
2. coder-template-sync activation script: `mkdir -p "$STATE_DIR"` runs
as root but terraform runs as the `coder` user. Add
`chown coder:coder "$STATE_DIR"` so coder can write .terraform state.
Also cleans up mangled single-line shell in coder-reset and
coder-workspace-reaper into readable multiline form.
nixos/k3s-sysbox.nix: only change is adding rsync to systemPackages configuration.nix: only changes are the hostname fix, chown STATE_DIR, and multiline shell reformatting of the mangled single-line curl calls (no comment banner changes)
…x does, fix coder.nix confusion
- README: step 2 — warn that the copied default.nix imports coder.nix (only needed when the VM is also a coder-agent workspace); show how to remove it for a pure box host - README: step 3 — add aarch64 note: set nixpkgs.hostPlatform = "aarch64-linux" in default.nix if the VM is ARM; flake defaults to x86_64-linux - README: step 5 — add local.nix creation (copy from local.nix.example, set admin creds + LAN IP) before nixos-rebuild switch; this is what actually triggers coder-init-admin auto-bootstrap, not a browser wizard - README: step 6 — rewrite to reflect that coder-init-admin.service handles bootstrap automatically if local.nix was set up; browser/CLI flow is the fallback, not the primary path - hosts/incus-vm/default.nix — remove /etc/nixos/coder.nix import (that file is for coder-agent workspace VMs, not box hosts); add local.nix import with a comment; add nixpkgs.hostPlatform placeholder comment - hosts/incus-vm/incus-vm.nix — clarify enp5s0 applies to both x86_64 and aarch64 Incus VMs (confirmed on aarch64)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes DEVREL-69
Ports
bpmct/coder-nixos#4tocoder/box. Validated end-to-end on a live Incus VM.What's here
hosts/incus-vm/— newConfig for running box on any headless NixOS host (Incus VM, ThinkStation, etc.). After
nixos-rebuild switchthe machine runs a full self-contained Coder deployment: Coder server + PostgreSQL + k3s + sysbox + template-sync.incus-vm.nix— QEMU guest agents + virtio drivers (via upstreamincus-virtual-machine.nix),systemd-networkdDHCP onenp5s0, disables the desktop/PipeWire/Avahi stack. Does not disable k3s — hosts opt in viadefault.nix.default.nix— per-host entrypoint; importsincus-vm.nix+ the two runtime files (/etc/nixos/incus.nix,/etc/nixos/coder.nix).README.md— explains what box does, the difference between this and thebpmct/incus-nixosregistry template, and a validated step-by-step setup guide (including bootstrap, git-add gotcha, admin first-run).configuration.nix— two bugs fixedFound and reproduced on a fresh Incus VM:
coder-init-admin: barehostname -sfails in a systemd unit because/usr/binisn't inPATH. Fixed to${pkgs.nettools}/bin/hostname -s.coder-template-syncactivation script:mkdir -p "$STATE_DIR"runs as root but terraform runs as thecoderuser. Addedchown coder:coder "$STATE_DIR"so terraform can write state.Also cleans up mangled single-line shell commands in
coder-resetandcoder-workspace-reaperinto readable multiline form.nixos/k3s-sysbox.nix— addrsyncsysbox-mgrdoes a preflight check forrsyncat startup. If absent it exits immediately and every pod stays stuck inContainerCreating. Fix is one package inenvironment.systemPackages.coderd/templates/k3s-sysbox/main.tf— two fixeslocal.kubectl: was a hardcoded Nix store path. Replaced with/run/current-system/sw/bin/k3s(stable per-generation symlink).hostAliases: was[{}](empty object), which Kubernetes rejects. Made conditional oncoder_lan_ipbeing non-empty.coderd/templates/coder-cli/README.md— newWas the only template missing a README.
Validation
Tested on a live Incus VM (NixOS 25.11, x86_64):
nixos-rebuild switch --flake /etc/nixos-repo#$(hostname -s) --impure— exit 0coder.service,postgresql.service,k3s.service,sysbox-mgr.service,sysbox-fs.service— all activetemplate-sync— pushed all 4 templates (k3s-sysbox,k3s-podman,k3s-dev,coder-cli)k3s-sysboxworkspace via API → podRunningwithruntimeClassName: sysbox-runc, agentconnected