feat(orch): premade NixOS base-image support

[tomassrnka]

Stacks on #3411. Adds a nixos distro profile — premade images whose packages/services are declared in the image's own NixOS configuration rather than installed at provision time — plus the E2B NixOS base-image definition.

Verification status

Verified at tip on real KVM: the image was built with a nix builder from the committed configuration.nix (build.sh fixed — it was non-executable and staged from a hardcoded out-of-repo path, so the committed config was previously never the one built) and booted as sandboxes through the template pipeline. All previously inspection-only fixes now have observed evidence: envd unit parity (Nice=-20, OOMScoreAdjust=-1000, GOMEMLIMIT=512MiB, IO/CPU/memory weights), chrony source-select oneshot (pool branch live; PHC branch replay-verified — a real PHC lock needs kvm-ptp, not exposed on the verification host), firewall off with envd's port connected, CA bundle a regular file + https=200, default user + NOPASSWD sudo, hostname/hosts kept, shadow tools present, nix store DB valid (--check-validity OK, 532 paths).

Note: sshd accepts connections ~1–2s after sandbox start — NixOS generates host keys per sandbox on first boot (unique keys per sandbox; the package-manager families bake shared per-template keys at provision time). Inherent to NixOS's design, left as-is.

Known NixOS limitations

• Sudoers for non-default users. A USER build step naming an account other than user fails: commands/user.go appends to /etc/sudoers, which is a read-only store symlink on NixOS. The fix is a drop-in under /etc/sudoers.d, which changes behaviour for every family and belongs in its own PR.
• No at-tip real-KVM coverage. Building the premade image needs a nix builder; none is available on the current box or dev VM, so the NixOS-side changes here are reviewed but not runtime-verified.

Draft: on-the-fly (nix-build at template-build time) is still open design; merge after #3411.

[cursor]

PR Summary

Medium Risk
Changes sit on the critical template base provisioning and boot path; mistakes in systemd cleanup or pre-activation bootstrap could break sandboxes, though impact is scoped to the new NixOS family.

Overview
Adds a premade NixOS path for template base images: the orchestrator treats ID=nixos as a declarative profile that does not install packages at provision time, runs an optional Bootstrap hook in the generated shell selector (busybox on PATH before activation), and uses InitNixOS provisioning that strips baked /etc/systemd/system drop-ins so NixOS activation can own systemd. A committed NixOS system definition and build/publish script produce a single-layer OCI rootfs with init glue, static os-release, and nix store DB registration for first boot.

^{Reviewed by Cursor Bugbot for commit 4596940. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1b75df9. Configure here.}

[cursor]

Stale comment

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7d6c806208

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Derive envd's memory limit from sandbox RAM

For supported templates below 1 GiB, this hard-coded value gives envd a much larger soft heap limit than the existing min(MemoryMB/2, 512) policy in rootfs/templates.go; for example, a 128 MiB sandbox receives 512 MiB instead of 64 MiB. Under allocation pressure envd can therefore retain far more of the guest's memory before the limit drives aggressive GC, causing workload or VM OOMs on the 128/256/512 MiB configurations, so the unit should calculate the limit at boot or receive the template-specific value.

Useful? React with 👍 / 👎.

[claude]

I didn't find any additional bugs beyond what's already been found and fixed in this thread, but given the scope and security-adjacent surface of this change, it's worth a human look before merging.

What was reviewed: the NixOS profile wiring in distro.go/init.go (bootstrap busybox PATH, systemd-unit removal for activation), and the base-image configuration.nix/build.sh. Checked the envd systemd unit for missing boot-ordering directives (Wants/After chronyd etc.) — not needed since envd doesn't depend on time-sync to start. Checked build.sh's nixpkgs channel pin (channel:nixos-24.05) for reproducibility — it is pinned, just not to a flake lockfile hash. Checked the nix-DB load-db activation script for a permanent-breakage risk if the first attempt fails — it's idempotent and guarded with || true, so a failed load just leaves the DB unset for a retry on the next activation, not a permanent lockout.

Extended reasoning...

Overview

This PR adds a fifth, fundamentally different distro profile (nixos) to the orchestrator's provisioning system: NixOS images are premade and declaratively configured via configuration.nix rather than provisioned imperatively like the other four families. It touches the shared selector contract (distro.go, init.go) plus new NixOS-specific files (configuration.nix, build.sh, README).

Security risks

The configuration.nix declares PermitRootLogin = yes, PermitEmptyPasswords = true, security.sudo.wheelNeedsPassword = false, and disables the guest firewall entirely. These mirror what the other distro profiles already do (isolation is enforced at the E2B network layer, not the guest), and the PR thread already surfaced and fixed the firewall-disable timing issue, so I don't see a new security regression relative to the existing profiles. The main residual risk is the acknowledged, still-open limitation that a USER build step naming a non-default account silently fails on NixOS (sudoers is a read-only store symlink) — tracked by the author as a known limitation for a follow-up PR rather than fixed here.

Level of scrutiny

This warrants more than a mechanical-change level of review: it's boot-path and init-system code for a new guest family, self-verified on real KVM only very late in the PR's life (several rounds of real bugs were found and fixed: missing firewall disable, missing socat/iptables/shadow packages, clobbered hostname/hosts, missing nix DB registration, and a PATH bug that broke pause-time fstrim reclaim). That iteration history is a good sign the process is working, but also indicates a change with enough surface area and subtlety that a human maintainer familiar with the orchestrator's provisioning contract should sign off before merge, particularly given the still-open sudoers limitation and the stacked dependency on #3411.

Other factors

A CI comment reports a failing test (TestSubscriptionManager_PubSubEndToEnd in packages/api/internal/sandbox/storage/redis) unrelated to the files this PR touches, which looks like pre-existing flakiness rather than something introduced here, but I did not independently investigate it.

[cursor]

Stale comment