Break up agent-box.nix

[lionello]

The main NixOS module is a single file. This was done for easy fetching in cloud-init using fetchurl (check history), no need for unzip/tar, but is harder and harder to maintain. Agents spend tons of tokens just figuring out which section to edit. We also can't use static the usual static analysis tools, since the embedded assets are not accessible without installing the module and checking in a built NixOS image.

• separate all/most embedded assets into separate files
• either re-embed all assets in CI workflow, or find an alternative to the fetchurl to fix multiple file fetch.

[defangdevs]

Picking this up. Investigated the constraints and prior art (issue 97 / PR 98, closed) before touching code.

Approach: split the source, re-embed in CI, keep the deploy fetch byte-for-byte unchanged

The single fetched file is load-bearing and stays — CFN user-data and agent-box-update.service builtins.fetchurl exactly modules/agent-box.nix at a pinned {rev, sha256}. That gives us dual-stack raw.githubusercontent (IPv6-only boxes via DNS64/NAT64), one content hash, and no unpack step on a journal-only first boot. I am not touching that, and I'm explicitly not pursuing the "multi-file fetch" alternative in the second checkbox — a tarball / multiple pins reintroduces exactly the IPv4-only + hash-fragility risk that design avoids.

The single source file was only ever a side effect of that fetch, and it's what PR 98 tripped over: it used builtins.readFile ./sibling, resolved at module-eval time on the box, which 404s the next rebuild (the box only has the one file). The fix is to resolve the split at build/CI time and commit a self-contained artifact.

Plan

1. modules/src/ holds real, tool-visible assets — settings daemon (.py, ~1941 lines / 45% of the file, and its embedded CSS + JS), AGENTS.md default, shell scripts, Caddyfile template.
2. A generator assembles the committed modules/agent-box.nix from a template + src/ (banner: GENERATED, edit src/). The daemon block has zero internal ${/'', so it embeds verbatim — same byte-identity property PR 98 verified.
3. CI guards:
   • regenerate then git diff --exit-code — committed artifact matches sources;
   • keep the existing module-single-file check — output stays self-contained;
   • round-trip fidelity — extract each embedded blob back out, diff vs source.
4. Deploy path, pin format, and fetch URL: unchanged.

Sequencing

• PR 1 — generator + CI guards + extract the settings daemon (lowest-risk, biggest slice, proven byte-identical).
• PR 2 — split the daemon's CSS/JS into their own files (unlocks eslint/prettier/stylelint), and pull the remaining shell/Caddyfile/AGENTS.md assets out.

Will validate each locally (nix build .#checks..., VM tests) before merge.

[defangdevs]

Done — both PRs merged to master.

• feat(build): generate agent-box.nix from split sources; extract settings daemon (#140) #143 — modules/agent-box.nix is now generated from modules/agent-box.nix.in + modules/src/* by bin/assemble-module.py (@@include:...@@ markers). Extracted the ~1930-line settings daemon to modules/src/settings-daemon.py.
• feat(build): split settings daemon CSS/JS into their own files (#140) #145 — split the daemon's embedded frontend into modules/src/settings.css + settings.js (now node --check/eslint/stylelint-visible), plus a CI-trigger fix so generator-input changes actually run the guard.

The two checkboxes

• Separate embedded assets into files — daemon, CSS, JS are out. The shell start/attach scripts and the Caddyfile stay inline on purpose: they're densely Nix-${…}-interpolated, so their content can't live in a plain file. (Only zero-interpolation assets can move.)
• Re-embed in CI vs. replace fetchurl — chose re-embed at build time. The fetchurl path is deliberately unchanged: deployed boxes still fetch the one self-contained modules/agent-box.nix at a pinned sha256. I explicitly did not pursue a multi-file/tarball fetch — it would give up dual-stack raw.githubusercontent (IPv6-only boxes), the single content hash, and the no-unpack-on-first-boot property (Deploy regression: module no longer single-file — 1-click first boot and self-update break on master #51). This also sidesteps what sank Extract the settings daemon into modules/settings-daemon.py #98 (readFile ./sibling resolved on-box → 404): the split is resolved before the file is committed.

Guarantee

Both PRs produce a modules/agent-box.nix that is byte-identical to the previous hand-written file except a 4-line GENERATED banner (#145 doesn't touch it at all). No derivation or behavior change — confirmed by the settings-page VM test and the master Deploy test (CFN create + destroy) after #143.

Guards in place

• nix run .#assemble to regenerate; edit .in/src, never the generated file (banner + .gitattributes linguist-generated).
• module-generated-up-to-date check fails on drift; module-single-file still guards the output is sibling-free; CI now triggers on .in / modules/src/** / the assembler.

Not done (offered as a separate follow-up if wanted): a caddy validate / caddy fmt --diff CI check on the rendered Caddyfile — the only way to get static-checking on the dynamic Caddyfile, since it can't be source-split.