appliance

Pre-Alpha. This page describes behavior that may change.

Ze ships a bootable VM appliance image: a minimal Linux system built with gokrazy that runs Ze as its only application. No package manager, no shell (except an emergency serial console), atomic A/B updates, automatic process supervision. The result is a 2 GB disk image suitable for N100-class mini PCs, Proxmox VMs, or QEMU testing. The default target is x86_64; set GOKRAZY_ARCH=arm64 for a native Apple Silicon QEMU image.

If you just want to run Ze on an existing Linux host, use the binary or container builds from the install page instead. The appliance is for when you want a locked-down, single-purpose networking box.

What's in the image

Component	Purpose
Linux kernel	Boot and hardware drivers.
Gokrazy init	Process supervisor, entropy seeding, watchdog heartbeat.
Ze	BGP daemon with DHCP client, NTP, and all internal plugins compiled in.
ze-serial-shell	Authenticated emergency shell on the serial console (login required).

Ze owns network configuration (DHCP) and time (NTP), so the gokrazy default DHCP and NTP packages are excluded from the image. interface { dhcp-auto true } discovers all ethernet interfaces and runs DHCP on each; environment { ntp { enabled true } } syncs the clock.

The root filesystem is a read-only SquashFS. Persistent data lives on a separate ext4 partition mounted at /perm.

The bundled gokrazy init is kept current. The pinned release pulls in golang.org/x/net 0.56.0 or newer, which clears CVE-2026-25680.

Prerequisites

Install once on the build machine. macOS:

brew install e2fsprogs    # ext4 filesystem tools
brew install qemu         # VM runtime (testing only)

For ISO creation, also install grub-mkstandalone (or grub2-mkstandalone) and xorriso; ze appliance iso checks for them before staging.

The three e2fsprogs tools the build needs (mkfs.ext4, debugfs, e2fsck) are resolved independently, each in its own right, with PATH consulted last. Distributions that split the package are therefore supported: Alpine ships debugfs in e2fsprogs-extra, so with both packages installed no single directory held all three, and the build refused with e2fsprogs not found while pointing at a package that was already installed. A missing tool is now named individually rather than blamed on the package.

The gokrazy build tool (gok) is vendored in the repo at vendor/github.com/gokrazy/ and built automatically by Make. A vendored gokrazy updater handles OTA pushes. No separate install is needed.

First-time setup

After cloning the repo, download gokrazy system packages (Linux kernel, init, serial console) into the Go module cache. This is a one-time ~42 MB download. The exact versions are pinned in gokrazy/ze/builddir/*/go.mod (tracked in git, verified by go.sum):

make ze-gokrazy-deps

After this, subsequent builds work offline.

The pinned kernel does not carry every module

The pinned rtr7/kernel package is a general-purpose kernel, and it ships without L2TP support. Ze's L2TP subsystem probes for l2tp_ppp and pppol2tp at startup and refuses to start when neither loads, which is the correct fail-closed behaviour for a BNG but reads badly on an appliance: the daemon exits at first boot and gokrazy restarts it in a loop, with nothing useful on the serial console to explain why.

If you are building an L2TP appliance you need an L2TP-capable kernel, not the default pin. Build one with a kernel profile that includes L2TP, and check ze doctor on the target before assuming a module is present. The same caution applies to any feature with a fail-closed module probe.

Build an image

The first build creates SSH credentials and a TLS certificate for the web UI:

make ze-gokrazy USER=admin PASS=secret

Subsequent rebuilds reuse the existing database (same credentials, same TLS cert):

make ze-gokrazy

To use a database from a running instance or from another machine:

make ze-gokrazy ZEFS=/path/to/database.zefs

The first build does these things:

1. Builds bin/ze for the host.
2. Runs ze init with the supplied credentials and generates a self-signed TLS certificate.
3. Cross-compiles Ze for the target arch (linux/amd64 by default, linux/arm64 with GOKRAZY_ARCH=arm64 or image.arch in appliance.json).
4. Builds a 2 GB disk image.
5. Formats the persistent /perm partition.
6. Injects database.zefs (credentials plus TLS cert) into /perm/ze/.
7. Bakes a build manifest (appliance, timestamp, arch, image name) into /perm/ze/build.json so ze version on the installed box can report which build it runs.

Build verification

After injecting the database, the builder verifies the write rather than trusting the tool's exit code. It reads the /perm image back and confirms the source database bytes are present (catching silent debugfs failures, which can exit 0 on internal errors), runs e2fsck -fn for ext4 structural integrity, and on a Linux root also loop-mounts the partition. The build fails loudly on the first error.

The database is kept at tmp/gokrazy/init/database.zefs between builds, so browsers that trust the certificate on first use will not prompt again after image rebuilds. The final image lands at tmp/gokrazy/ze.img.

Test in QEMU

make ze-gokrazy-run

This boots the image with the ports you care about forwarded to localhost:

Host port	Guest service	How to use it
18080	Gokrazy web UI (port 80)	http://localhost:18080/
28080	Ze web UI (port 8080)	http://localhost:28080/
2222	Ze SSH CLI (port 22)	ssh -p 2222 admin@localhost

Quit QEMU with Ctrl-A X.

The gokrazy web UI (:18080) shows process status, stdout/stderr ring buffers, and resource usage. Default credentials for it live in gokrazy/ze/config.json under Update.HTTPPassword.

Deploy to hardware

Write the image to a USB drive or internal disk on your target machine:

# Linux
sudo dd if=tmp/gokrazy/ze.img of=/dev/sdX bs=4M status=progress

# macOS
sudo dd if=tmp/gokrazy/ze.img of=/dev/rdiskN bs=4m

Or import into Proxmox:

qm importdisk <vmid> tmp/gokrazy/ze.img <storage>

The machine boots to a serial console (ttyS0, 115200 baud; the gokrazy config sets SerialConsole: ttyS0,115200 and loglevel=8). ze-serial-shell requires authentication with the local admin credentials before granting shell access. If the credentials database is missing or unreadable, access is granted without authentication for emergency recovery. When admin-enabled: false is set in the appliance config, the serial console denies the built-in admin (fail-closed) and prints "local admin login disabled".

On first boot the device auto-inits: if it has a database.zefs but no committed config, Ze enters bootstrap mode (DHCP client on every ethernet interface, SSH enabled) so the box is reachable before any operator config exists. See Zero-Touch Provisioning for the bootstrap details. Once configured, Ze starts automatically, gets a DHCP address, and listens for BGP connections per the effective config.

Configuration

Seed config

The initial Ze config is embedded in the read-only root filesystem at /etc/ze/ze.conf. It is baked in at build time from the ExtraFileContents field of gokrazy/ze/config.json. The shipped seed enables the web UI on :8080 and SSH on :22, with logging at info level:

environment {
    log {
        level info
    }

    web {
        enabled true
        server default {
            ip 0.0.0.0
            port 8080
        }
    }

    ssh {
        enabled true
        server default {
            ip 0.0.0.0
            port 22
        }
    }
}

To change the seed config, edit ExtraFileContents in gokrazy/ze/config.json and rebuild.

Runtime config

Once booted, use ze config edit over SSH to modify the running configuration. Changes are stored in /perm/ze/database.zefs and persist across reboots and image updates.

Environment variables

Ze's environment is set in gokrazy/ze/config.json under PackageConfig:

Variable	Value	Purpose
ze.config.dir	/perm/ze	Persistent storage for database.zefs.
ze.bgp.api.socketpath	/tmp/ze.socket	Unix socket for the local API.
ze.bgp.daemon.drop	false	No privilege dropping (no zeuser on gokrazy).
ze.log	info	Log level.
ze.log.backend	stderr	Logs land in the gokrazy ring buffer.

Updating

Gokrazy supports atomic A/B partition updates over the network. Push the most recent image to the running instance:

bin/ze-setup appliance push <name>

This pushes the new root filesystem without touching /perm. The machine reboots into the new version. If the update fails mid-flight, the previous root partition is still intact, so you cannot brick the box with a bad update.

For full image rebuilds (when you also want to update the kernel or partition layout), run ze appliance build <name> again and re-flash.

Architecture notes

Internal plugins only

Gokrazy has no shell and no PATH. Ze's external plugin mechanism uses /bin/sh -c to fork processes, so it does not work here. Instead, every Ze plugin (bgp-rib, bgp-gr, bgp-adj-rib-in, …) is compiled into the ze binary as an internal plugin and runs as a goroutine. This is the default build and covers all standard BGP functionality. If you need an external plugin, the gokrazy appliance is the wrong target.

Process supervision

Gokrazy's init restarts Ze if it exits with a non-zero status, except exit code 125 which means "don't restart". Ze handles SIGTERM for graceful shutdown. Logs (stdout/stderr) are captured in ring buffers visible through the gokrazy web UI.

Persistent storage

The /perm partition (ext4) survives image updates. Ze stores its database (database.zefs), TLS certificates, and config state there. The pointer is ze.config.dir=/perm/ze.

Startup resilience

Ze reaches out to external services from background workers, off the critical boot and commit path. An unreachable NTP time source, RADIUS change-of-authorization server, or managed fleet hub no longer stalls startup or a config commit/apply. The daemon comes up (or the commit completes) and the worker keeps retrying in the background.

A physical interface named in the config but absent from the machine is warned and skipped rather than aborting the whole apply. One image can boot on hardware where a configured NIC is missing or a cable is unplugged, without failing every other interface in the config. A genuine error on a present interface still aborts and rolls back.

Repo layout

gokrazy/
  .gitignore              # excludes *.img
  ze/
    config.json           # gokrazy instance config (what to build, how to start)
    builddir/
      github.com/ze-software/ze/
        go.mod            # ze dependency pins + relative replace directive
        go.sum
      github.com/rtr7/kernel/
        go.mod, go.sum    # Linux kernel version pin
      github.com/gokrazy/gokrazy/
        go.mod, go.sum    # gokrazy init version pin
        cmd/dhcp/         # DHCP client
        cmd/ntp/          # NTP client
        cmd/heartbeat/    # watchdog heartbeat
        cmd/randomd/      # entropy seeder
cmd/ze-serial-shell/      # serial console login gate (replaces serial-busybox)
cmd/ze-gok/               # gok wrapper (built by make bin/gok)

The gok build tool source is vendored in the main vendor/github.com/gokrazy/ directory. The builddir/ files are small text (go.mod + go.sum, ~27 KB). System packages (kernel, init) live in the Go module cache after make ze-gokrazy-deps.

ze-setup binary

Appliance build commands (ze appliance) and PXE provisioning (ze install remote) live in the build-host ze-setup binary, which keeps build-host tooling out of the on-device ze binary. Build it from the repo root:

make ze-setup              # produces bin/ze-setup

Appliance management CLI

The ze appliance command (in the ze-setup binary) provides a structured workflow for managing appliance fleets from a bastion host. Each appliance has its own directory under ~/.config/ze/appliances/<name>/ with a JSON config, encrypted secrets, and a TLS certificate (override the location with --dir or ZE_APPLIANCE_DIR).

Command	Purpose
bin/ze-setup appliance init <name>	Interactive wizard: set target IP, SSH user, passphrase. Creates the encrypted appliance directory.
bin/ze-setup appliance build <name>	Build a full disk image (assemble ZeFS + gok + ext4 inject + checksum + manifest). --all builds every appliance.
bin/ze-setup appliance kernel <name>	Download or build the installer kernel (reads arch/profile from config).
bin/ze-setup appliance initrd	Download or build the installer initrd.
bin/ze-setup appliance iso <name>	Build a bootable installer ISO from an existing image.
bin/ze-setup appliance run <name>	Boot the image in QEMU for local testing.
bin/ze-setup appliance push <name>	OTA update to the running appliance (A/B partition swap).
bin/ze-setup appliance config-push <name>	Push config to a running device. The device auto-reverts after 30s if BGP does not stabilize.
bin/ze-setup appliance config-push --all --parallel 4	Push config to all addressed appliances in parallel.
bin/ze-setup appliance list	List all configured appliances.
bin/ze-setup appliance show <name>	Show config summary, cert expiry, managed status.
bin/ze-setup appliance export <name>	Export encrypted archive for disaster recovery (--all for everything).
bin/ze-setup appliance import <archive>	Restore appliance from encrypted archive.

Appliance secrets are encrypted at rest using Argon2id + XChaCha20-Poly1305 when a passphrase is set. For fleet operations, ze appliance unlock starts a passphrase agent (like ssh-agent) holding the derived key in memory.

Device-side config behavior

When config is pushed to a device (config-push), the device:

1. Validates the new config against YANG schemas.
2. Applies the config atomically.
3. Waits 30 seconds for BGP sessions to stabilize.
4. If BGP does not stabilize, automatically reverts to the previous config.

This prevents a bad config push from leaving a device unreachable.

Reserving hugepages for VPP

When the appliance runs the VPP data plane, reserve hugepages at boot so VPP starts on a performant memory layout. Add image.hugepages to the appliance config and ze appliance build bakes the hugepages kernel arguments into the boot command line (the checked-in gokrazy config is left untouched):

"image": {
    "arch": "amd64",
    "size-bytes": 2147483648,
    "memory": "8gb",
    "hugepages": { "size": "1gb", "page-size": "2mb" }
}

hugepages.size is the total reservation and page-size is 2mb or 1gb. Declare image.memory so the build rejects a reservation over half of target RAM and ze appliance run sizes QEMU's memory to match. 1 GiB pages need CPU pdpe1gb support and CONFIG_HUGETLBFS in the kernel profile. ze doctor surfaces both, and warns when the reservation is missing, insufficient, or clamped.

ISO installer media

ze appliance iso builds a bootable UEFI installer ISO around an existing appliance image. The image is gzip-compressed inside the ISO (a 2 GiB image with ~100 MiB of content compresses to roughly 100 MiB), and a minimal initrd decompresses and writes it to the target disk on boot. The architecture comes from the appliance config (image.arch); amd64 produces BOOTX64.EFI, arm64 produces BOOTAA64.EFI.

bin/ze-setup appliance build prod
bin/ze-setup appliance kernel --profile hardware prod   # installer kernel (reads arch from config)
bin/ze-setup appliance initrd                           # installer initrd
bin/ze-setup appliance iso prod

Use ze appliance iso --check to verify prerequisites (kernel, initrd, grub, xorriso) before building. The Makefile shortcut make ze-iso CONFIG=prod.json SSH_PASSWORD='...' runs the whole pipeline (init, kernel, initrd, image, ISO).

The ISO verifies the embedded image checksum before writing and matches the booted media by a builder-generated ze.media-id token. It does not regenerate credentials or fetch a separate ZeFS database; the installed disk receives the selected image bytes, including the /perm/ze/database.zefs and /perm/ze/build.json that build already injected. After installation, the system powers off so removable media can be removed before the next boot.

With more than one fixed disk, pass an explicit whole-disk target at ISO creation time:

bin/ze-setup appliance iso --target /dev/vda prod

The ISO can be written with dd, Etcher, or Rufus in DD mode. Ventoy works when the installer kernel includes loop and FAT/exFAT support (the hardware profile has this; the qemu profile does not).

Installer initrd (pure Go)

The installer is a single statically-linked Go binary that runs as PID 1 in the initrd. There is no busybox and no shell: the old busybox shell initrd is gone. The same binary drives both install paths, downloading the disk image over HTTP for a PXE install or mounting the appliance ISO read-only for USB/optical media, then writing the selected image to a safe target disk.

It is built to survive an awkward first boot on unknown hardware:

Feature	Behavior
Boot-NIC pinning	The kernel command line can pin the installer to the NIC it booted from (by MAC), so it does not race DHCP across every interface.
DHCP recovery	If the pinned NIC has no carrier or DHCP fails, the installer retries rather than aborting.
MAC-pinned recovery	A retry reuses the same boot-NIC identity, so it stays on the interface that has connectivity.
Gated rescue shell	On repeated failure the installer offers a recovery console (retry network and install, show network state, reboot, power off) gated behind the local admin password. With no admin credential set, it reboots after a short delay instead of exposing an open shell.

After an HTTP install the installer injects the credentials database and reboots. After an ISO install it powers off so the media can be removed before the next boot.

Installer kernel profiles

The installer kernel (ze appliance kernel, for PXE and ISO installs) supports named profiles that select the driver set. This is separate from the on-device runtime kernel below.

Profile	Drivers	Use case
qemu (default)	virtio NIC + block	QEMU tests, fast build.
hardware	virtio + EFI + framebuffer + Intel/Realtek/Broadcom/Mellanox NICs + AHCI + NVMe	Bare-metal PXE/ISO install.
hardware-kms	hardware plus Intel iGPU KMS	Boxes with integrated graphics.

Set image.kernel-profile in appliance.json so ze appliance kernel <name> picks it up, or pass --profile. The build script verifies every required option resolved to =y and fails loudly otherwise. The build backend defaults to Docker when available and falls back to the shared QEMU backend; force one with --builder docker or --builder qemu.

Build outputs and incremental kernel builds

Installer artifacts land under a single build/ tree: the installer kernel at build/kernel/Image, the initrd at build/initrd/initrd.img.gz, and (for PXE) the assembled TFTP root under build/pxe/. Resolved kernel and initrd artifacts are also cached under ~/.cache/ze/ and reused on later runs.

Kernel rebuilds are incremental. The QEMU kernel builder keeps a compiler cache (ccache) across runs, so a second kernel build after a small config change is much faster than the first.

Runtime kernel (L2TP / PPP / tunnels)

The on-device runtime kernel is built separately from the installer kernel. Ze's L2TP LNS and PPPoE paths need PPP/L2TP support compiled into the runtime kernel, which the pinned upstream gokrazy kernel is not assumed to provide. The repo-local runtime kernel config keeps these built in (=y): CONFIG_PPP, CONFIG_PPPOL2TP, CONFIG_PPPOE, CONFIG_L2TP, and CONFIG_L2TP_V3 (the former CONFIG_L2TP_NETLINK was removed in kernel 7.0, where netlink is always built with L2TP). Tunnel and WireGuard support comes from CONFIG_NET_UDP_TUNNEL, CONFIG_WIREGUARD, and CONFIG_TUN. The image also sets a serial console (SerialConsole: ttyS0,115200) and loglevel=8 for full boot diagnostics on a headless box.

Build the runtime kernel before building an appliance that terminates L2TP subscribers:

make ze-kernel                                   # default: docker, amd64
make ze-kernel KERNEL_BUILDER=qemu               # force the shared QEMU backend
make ze-kernel KERNEL_ARCH=arm64                 # runtime arm64 kernel
make ze-gokrazy USER=admin PASS=secret

make ze-kernel emits vmlinuz, lib/modules/, and DTBs and overlays them into the module-cache copy used by make ze-gokrazy. The first overlay backs up the pinned cache; make ze-kernel-clean restores it. The pinned appliance kernel is version 7.1.1.

ze.wait kernel parameter

The ze.wait installer kernel parameter sets the maximum number of server probe attempts before giving up (default 30, 0 to skip the probe). It is useful for slow NIC carrier detection or debugging PXE boot sequences. See Zero-Touch Provisioning for the full installer kernel command line.

See also

• Install for non-appliance builds (binary, container, from source).
• Zero-Touch Provisioning for PXE-based bare-metal provisioning.
• Building for host builds and toolchain setup.
• CLI for editing the running config over SSH.
• Web UI for the browser interface reachable on port 8080 of the appliance.

Adapted from main/docs/guide/appliance.md.