Troglodyne Universal Automatic Provisioning

Automatically build and host pretty much any website

Where things are written down

docs/CONFIGURATION.md	the files in /etc/trog-provisioner, and what a guest's recipes look like
perldoc Provisioner::Recipe	writing a recipe: fragments, generated files, dependencies, tests
perldoc Trog::Credentials	handing passwords to a run that has nobody to ask
docs/APPROACH.md	the choices a recipe is expected to make
docs/BACKUPS.md	how state survives a guest being rebuilt
docs/NETWORK.md	the networks a guest is on, and what a static address means
MAINTENANCE.md	what is pinned here, and how to tell when it has gone stale
EXAMPLE.md	a worked deployment, end to end
STYLE.md and docs/POD.md	how we write the perl and document it
t/TESTING.md	how we test, here and on the guest
CLAUDE.md and AGENTS.md	working here with a coding agent: the procedure, and the guidelines it follows

WHY

Until your business is at the scale you need to distribute dynamic operations like statics with a CDN, stuff like openstack or kubernetes is overkill.

This gives you similar ease of orchestration with the supposition you are working with one or a few Hypervisors. You'd be surprised how well this works for 99% of business purposes. Even business units at giant corporations can do just fine with this approach.

How it works

1. Write a configuration file (Config::Simple format) in $DOMAIN/provision.conf which tells us:

   • ips: What IPs/Gw/Resolvers to use for said domain
   • size: How big the disk oughtta be
   • image: What base image to use
   • packages: What packages to install
   • distro: Which distribution the guest is, which decides the base image, the packager, and which version of each recipe names its packages. Written out of _global by bin/new_config; defaults to ubuntu.
   • release: Which release of that distribution the guest is, such as noble. Written out of _global by bin/new_config; defaults to the release that the distro recipe pins.
   • contact_email: Email address to send root's mail to
   • depends_on: Whether this system is to be provisioned on something that may or may not already exist
   • admin_user: What the name of the admin user is in the event we want to provision on already existing systems. This user needs passwordless sudo; when omitted we use root.
   • hypervisor: Pin this guest to a named hypervisor out of hypervisors.conf, when it genuinely has to live on a particular machine. Normally you leave this out and let it be placed. See "PROVISIONING A REMOTE HYPERVISOR" below.

   Nothing else here says anything about where the guest runs; that's what hypervisors.conf is for. A guest definition that names a machine is a guest definition you can't move.

   • cpu_mode: libvirt <cpu mode="..."> value. Defaults to host-passthrough so the guest sees the HV's real CPU (incl. AVX/AVX2 — required by anything that probes cpuid for vector extensions, e.g. the v8 snapshot bundled with the claude CLI). Set to host-model or a specific qemu CPU model if you need to migrate the guest to a differently-specced HV.

2. Write $DOMAIN/users.yaml describing the users to create. See cloud-init's documentation for examples.

3. Ensure tarball backups to restore (if they exist) are in the directory as data.tar.gz.

4. Run bin/provision $DOMAIN -- fully qualified, e.g. bin/provision mysql.example.net (add --hypervisor $NAME to build on a particular one)

5. It is the responsibility of data.tar.gz to have a Makefile in the TLD which sets up all relevant dependencies, loads up DBs, etc as the default target.

6. To set up new sites, have a skeleton site generator to build a blank site tarball.

This all works behind the scenes due to:

1. A small ISO, labelled cidata, mounted as a cdrom, carrying the network setup and cloud-config
2. libvirt's virtual networking allowing us to talk to this HV.
3. Spinning off an atd job to actually do the setup when cloud-config is done.

You'll need to have the user running this set up with at least one working authorized SSH key so we can add more. Aside from that, clone this in /opt where the user can see it.

It is not the responsibility of this tool to ensure inappropriate setup steps in the run makefile are skipped when applying to a dependent system, or deploying to an exisiting host.

DO NOT USE ROOT TO RUN THIS! Make sure that the user is in the qemu/kvm group (depending on running env) so you can talk to libvirt instead. The same goes for the login you use to reach a remote hypervisor.

From there the host should "Just Work"TM.

See example.test/ for example usage.

All the relevant resources will be stored in /vms on the HV.

PROVISIONING A REMOTE HYPERVISOR

By default the hypervisor is the machine you run bin/provision on. There are two ways to build somewhere else.

A fleet, in hypervisors.conf

Describe the machines once, in /etc/trog-provisioner/hypervisors.conf. See hypervisors.conf.example; the short version is one [block] per machine:

[hv1]
libvirt_uri    = qemu+ssh://root@hv1.example.test/system
reserve_memory = 4096
max_guests     = 20

[hv2]
libvirt_uri    = qemu+ssh://root@hv2.example.test/system

From then on every tool works out which machine it wants on its own, and you go on running them exactly as you did:

bin/provision mysite.test
bin/destroy   mysite.test
bin/snapshot  mysite.test
bin/restore   --latest mysite.test

bin/provision places the guest: whichever hypervisor already has it keeps it, otherwise the roomiest one that can actually hold it wins. Everything else finds it, by asking each hypervisor in turn whether it has a guest by that name. Nothing is written down, because a file recording where a guest lives goes stale the moment somebody migrates one by hand, and libvirt never does.

Placement compares what the guest asks for -- memory, cpus and size from its provision.conf -- against what each machine has left, and refuses if none of them can take it:

$ bin/provision hungry.test
Nowhere to put hungry.test: it wants 65536MB of memory, 16 CPUs and 500GB of disk,
and no hypervisor in /etc/trog-provisioner/hypervisors.conf can spare that.
  hv1: needs 65536MB of memory, 24576MB free (65536MB physical, 36864MB committed, 4096MB reserved)
  hv2: needs 500GB of disk, 210GB free in the pool after a 50GB reserve
  hv2: already has 20 guests, and max_guests is 20

Memory is counted as committed rather than used: a guest that was promised 8G is holding 8G against the machine even while it idles at 400M, and overcommitting memory is how you get the OOM killer picking one of your VMs. CPUs are the opposite -- overcommitting cores is normal -- so those are measured against cpu_overcommit times the physical count. Of the machines that fit, the one chosen is the one whose tightest resource is least tight afterwards, which is what stops one machine filling its disk while the fleet still has plenty of RAM.

A guest that genuinely has to live on a particular machine can say so in its own provision.conf:

hypervisor=hv1

It still has to fit; a pin to a machine with no room is an error rather than a quiet reassignment.

One machine, on the command line

Skip all of the above and say where to build:

bin/provision --hypervisor hv1 mysite.test
bin/destroy   --hypervisor hv1 mysite.test
bin/snapshot  --hypervisor hv1 mysite.test
bin/restore   --hypervisor hv1 --latest mysite.test

--hypervisor names a block of hypervisors.conf and skips the rest of it: no search, no capacity check, act there. A name the file does not have stops the command, which then says which names it has. It is a name and not a connection string because a hypervisor is libvirt or a cloud now, and a libvirt URI cannot say the second kind.

Without a hypervisors.conf, libvirt_uri in a guest's provision.conf still works the way it used to, and it stays a URI: a name means a block of a file that isn't there. It's deprecated -- it's a property of the machine, not the guest -- and it's ignored, with a warning, once a fleet exists. With no fleet and no libvirt_uri, you build on this machine, as before.

What that actually does

The URI is used for every libvirt call we make, which all go through Sys::Virt -- no virsh, so libvirt's own transports do the work. Beyond that, a fair amount of what this tool does is not libvirt at all -- discovering the bridge devices, dropping virtiofs-better in /usr/libexec, putting the domain directory somewhere the guest can fetch it from -- and all of it has to happen on the hypervisor. So for a remote URI we open one Net::OpenSSH::More connection there and do it over that, commands and file transfers alike.

The hypervisor and the guest are reached the same way, so both are subclasses of Trog::Machine, which holds the connection, the commands and the file operations. Trog::HV adds libvirt and the paths; Trog::Guest adds the waiting a freshly built VM needs; Trog::Local is us, which matters because a guest fetches its payload from here rather than from the hypervisor. Nothing in either goes over sftp: Net::SFTP::Foreign hangs rather than failing when the far side refuses a write, so files are poured down the standard input of a command instead, and sudo goes in front of the write rather than in front of a move afterwards.

That means:

1. A remote hypervisor has to be named with an ssh transport. qemu+ssh://user@host/system (or +libssh/+libssh2) tells us both how to talk to libvirt and how to get a shell. qemu+tcp:// and qemu+tls:// give us libvirt but no filesystem, so they're refused up front rather than failing halfway through a build.

2. You need passwordless SSH to the HV as the user the URI names. We use your ~/.ssh/config, so jump hosts and per-host identities work as you'd expect.

   Passwordless sudo there is strongly preferred but no longer required: sudo runs with -n, and if the far side asks for a password you get prompted once and it's remembered for the rest of the run. Run non-interactively -- from cron, or with stdin redirected -- and there's nobody to ask, so you get an error naming the user and the NOPASSWD line to add instead of a hang.

3. The guest needs a routable address. We normally find a new VM by its libvirt NAT lease (192.168.122.x), which is only reachable from the HV itself. When the HV is remote we SSH to the first entry in ips instead, so a remote build requires ips to be set in provision.conf. You'll get a clear error rather than a hang if you forget.

4. The domain directory is not copied to the HV at all. It used to be, the whole of it, because the guest fetched its payload from the hypervisor -- which meant the guest's private key was on the hypervisor too. The guest fetches from this machine now, so neither is true; the key does not leave here, and since it is kept in the secret store rather than in the domain directory it is not at rest here either. See perldoc Trog::Guest.

   Anything outside the domain directory that a guest expects to find on the HV -- dir= entries in mounts.txt point at hypervisor-side paths, for instance -- is yours to provision, and always was.

5. The HV still has to be set up as a hypervisor: the bridge devices, the qemu/kvm group membership. See UBUNTU DEPS above. --connect points at a hypervisor; it doesn't build one.

   It no longer has to run an apt mirror, or collect anybody's logs. Both used to be assumed of every hypervisor and compiled into every guest. A mirror is a guest now, built with the aptmirror recipe and named in _global's mirror; so is a cache for what recipes download, built with fetchcache and named in _global's cache; a log collector is a guest built with logcollector, and logshipper is what points a guest at one. See docs/CONFIGURATION.md.

   It does not have to be the machine you run this from either. The machine that runs the provisioner is itself a guest, built with the trogrunner recipe: perl, the CPAN dependencies, /etc/trog-provisioner, and a key a hypervisor will trust. perldoc Provisioner::Recipe::trogrunner.

6. Nothing holds a guest to a budget unless you give it one. There is no quota in libvirt -- no accounting, no limit -- and on the system URI every guest runs as libvirt-qemu whoever defined it, so there is no UID for a disk quota to attach to. reserve_memory, reserve_disk and max_guests in hypervisors.conf are this tool asking itself for permission, which is enough for a fleet you drive yourself and nothing at all against something driving virsh directly.

   What the kernel will enforce is a storage pool on a filesystem with a limit on it, and a systemd slice. Both are named in the hypervisor's block:

    pool_path = /srv/vm-disks/runner
    pool_name = runner_disks
    partition = /machine/runner
   

   Name pool_name as well as pool_path -- libvirt looks a pool up by name, so a path given beside the name of a pool that already exists elsewhere is ignored and every volume lands in the existing one. Then zfs create -o quota=500G (or any other filesystem that can carry one) under that path, and systemctl set-property machine-runner.slice CPUQuota=400% for the slice. bin/preflight says whether the pool you have is actually limited or is just the whole filesystem.

   Leave memory to reserve_memory: a MemoryMax on a slice full of VMs kills one rather than refusing the next.

   Neither stops a determined guest naming a different pool or partition; that takes libvirt's polkit access driver, which ships disabled. These bound the accident, not the adversary, and it is worth knowing which you have before telling anybody a runner is capped.

One repository, one run

bin/new_config and bin/provision used to live in separate repositories and be run one after the other: generate a domain's configuration, then build the VM. That held together while the generator could assume it was running on the hypervisor it was writing about -- it read the internal IP and the sshd port straight off the local machine. Once which hypervisor to use became a choice, both halves had to agree on the answer, and the only honest way for them to agree is to be one program.

So bin/provision generates the configuration first, and the generator is what picks the hypervisor: the recipe already says how much memory, disk and CPU the guest wants, which is everything placement needs. provision.conf never has to carry it. Every provision generates, for that reason: a configuration written for one hypervisor says the wrong thing about any other, down to the address the guest is given, and a build that skipped the generation could not tell. A domain you want built is a domain in recipes.d.

Configuration lives in /etc/trog-provisioner

Four files, none of them in the checkout -- hypervisors.conf, recipes.yaml, recipes.d/ and secrets.kdbx. What goes in each of them is docs/CONFIGURATION.md.

They describe an installation, not this software. They used to sit in the checkout, which meant .gitignore was the only thing between a password database and a public repository, and meant every command had to be run from one particular directory to find them. .gitignore still lists them, so an old working copy can't commit one by habit.

TROG_PROVISIONER_CONFIG points somewhere else -- another installation, or a temporary directory in a test with no business reading the real one. Every command also takes --recipes and --hvconf individually.

Spinning one up from the recipes

bin/new_guest writes a domain configuration from a list of recipes, so a scratch guest is one command rather than a file you copy from another guest and edit:

$ bin/new_guest nginx mariadb
Wrote /etc/trog-provisioner/recipes.d/f1698301-2db6-41ec-90fd-effec2f148d4.test.yaml
f1698301-2db6-41ec-90fd-effec2f148d4.test: 2 recipes (nginx, mariadb)

Fill these in before provisioning:
  mariadb.dumpfile
  mariadb.root_pw
  mariadb.version

With no --hostname it picks a UUID under .test, which is a name nobody else has and a TLD reserved for exactly this. --memory, --cpus and --size are what placement reads; the defaults are small on purpose.

Every recipe says what it takes in args(), an OpenAPIv3 schema, so what to write is derivable rather than guessed: fields the recipe requires get their default if it has one and CHANGEME if it does not, and fields it does not require are left out, so the recipe's own defaults keep applying instead of being frozen into the file the day it was generated. Anything _base already supplies is left to _base, which wins the merge anyway.

A CHANGEME that reaches bin/new_config stops it, naming the keys. Without that it would validate quite happily -- it is a string, and root_pw wants a string -- and you would get a database whose root password is CHANGEME.

Two of what bin/recipes lists are not things to put under a domain, and it does not offer them: ubuntu is the distro recipe, which says what a guest built on Ubuntu is -- its image, its packager, and the network configuration, cloud-init and setup script it first boots from -- and vm is the machine it runs on, which owns the libvirt XML and every disk knob. Both are configured out of a domain's _global, both are asked for by name, and bin/recipes vm prints what the disk knobs are.

bin/recipes is the other half of the same thing:

$ bin/recipes                      # every recipe, and what it is for
$ bin/recipes mariadb              # what mariadb takes, as JSON
$ bin/recipes --scaffold mariadb   # what new_guest would write for it
$ bin/recipes --json               # the listing, for something other than a human

Both are Provisioner::Cookbook with a command line attached. It does not live under lib/Provisioner/Recipe/, because everything there is discovered and loaded as a recipe.

Everything is spelled out

Domains used to be written two ways: short, with a tld from ipmap.cfg's global section appended, or fully qualified for anything under a different parent -- which the config called an "addon" and handled down a separate path. Both spellings arrived at the same place, and the separate path existed only to undo the appending.

There is one spelling now, in full, matching what recipes.d already named its files:

bin/new_config mysql.example.net
bin/provision  mysql.example.net

[addons] and tld are gone, and the aliases of a domain -- and the top-level key of each recipe -- are fully qualified. The one-time rename was bin/qualify_site_data, which is gone now that every known configuration is migrated. An installation still on the old format can take it from the history, with git show $(git log -1 --format=%H --diff-filter=D -- bin/qualify_site_data)^:bin/qualify_site_data.

One file describes an installation, not two

Who administers a guest, how it reaches the network and where its addresses come from lived in ipmap.cfg, in the format Config::Simple reads. What a guest is made of lived in recipes.yaml. Nothing distinguished the two lists: a recipe read admin_user out of one and its own settings out of the other, and each new setting was an argument about which file it belonged in.

They are one list now, the _global of _base in recipes.yaml, which a domain overrides in its own _global. Provisioner::Cookbook->global_schema declares it, so an installation missing a setting is told all of them at once rather than the first one a script happened to read. That is also the end of a failure this repeated: bin/new_guest built a configuration that never went through bin/new_config, so nothing refused the guest it wrote with no admin_user in it, and the recipes that own a file on the guest rendered one owned by nobody.

An installation that still has the older file moves it across with bin/ipmap_to_globals. The addresses do not move: they live in ips.db, and nothing has read them out of a file since bin/assign_ip took over handing them out.

Why there is no terraform any more

There was, and every one of these is a thing it did:

• It wanted a state file per hypervisor, because one shared between two would have it believe HV A's resources lived on HV B and destroy accordingly.
• Reaching a hypervisor that had been provisioned from itself, that state started empty -- so terraform believed nothing existed and set about recreating the pool, the base image and every domain, which libvirt refused one at a time.
• Adopting what already existed needed import blocks, plus ignore_changes for every attribute the provider wouldn't report back afterwards (target on the pool, create and backing_store on the volumes) -- without which it planned a replacement on every run, which prevent_destroy then refused.
• And libvirt_volume can't be imported at all in 0.9.1. It takes the id, looks for the volume, and answers Cannot import non-existent remote object for the volume key, its path and pool/name alike, on a volume it can otherwise see perfectly well.

The machinery holding all that at bay got larger than the thing it stood in for. bin/provision now renders libvirt XML from domain.xml.tmpl and hands it to Sys::Virt: define the pool if it's absent, fetch the base image if it's absent, make the guest's disk as an overlay on it, build the cloud-init seed with xorriso, define the domain and start it. There's no state file, because libvirt is the state.

The hypervisor needs xorriso (or genisoimage/mkisofs) for the seed ISO, and curl to fetch the base image. Nothing else.

Checking a hypervisor before you build on it

bin/preflight asks whether a provision is worth starting. Every check it makes is one that, if it fails, fails the whole run — so they happen up front, all of them run even after the first failure, and each one that fails says what to do:

$ bin/preflight
Hypervisor: qemu+ssh://someadmin@hv1.example.test/system

  ok   Reached someadmin@hv1.example.test as someadmin
FAILED No passwordless sudo for someadmin on hv1.example.test
  ok   Cloud-init seed builder: xorriso
  ok   libvirt answers, running 10.0.0
  ok   Sys::Virt 10.0.0 matches libvirt 10.0.0 on the hypervisor
  ok   Configuration to copy from: /etc/trog-provisioner

Exits 0 when a run is worth starting and 1 when it is not, so it works in a script. --hypervisor checks a particular hypervisor instead of whichever the fleet would pick.

Two of these are worth the trouble because of how they fail otherwise:

Passwordless sudo. Provisioning writes to the storage pool and defines domains, all through sudo. A password prompt in the middle of that has nowhere to be answered from, so the run hangs rather than failing, and you find out by waiting.

Sys::Virt against the hypervisor's libvirt. Sys::Virt is released in lockstep with libvirt and binds the API of the release it was built against. Talk to a daemon from a different one and the failure does not announce itself as a version problem — it is a constant that is not exported, or a call the far side does not implement, blamed on whatever was being attempted at the time. Compared on major.minor; a patch release apart is fine.

When a guest will not boot

bin/provision can only tell you a guest never asked for an address. A guest that never boots has no lease, no ssh and no logs to go and get, so there is nothing to read. bin/debug_boot is how you read it anyway.

Start with the console. It points the guest's serial port at a file on the hypervisor, restarts it, and hands back every line the kernel and systemd printed — as text, from the first line of firmware:

bin/debug_boot --console vm.example.test

A hang is the last line before the silence. --fetch re-reads that capture without another restart, and --shot grabs the screen as it is now.

With libguestfs on the hypervisor, you can go further without booting anything:

bin/debug_boot --cat vm.example.test /var/log/cloud-init.log
bin/debug_boot --ls  vm.example.test /etc/netplan
bin/debug_boot --single vm.example.test     # then --vnc for the tunnel

--single writes single onto the kernel command line in the guest's own grub.cfg and starts it, so it comes up in rescue mode by itself — no five second window to catch and no keystrokes to get right. The guest has to be off first, since it is a write to its disk. bin/preflight says whether libguestfs is there; without it, --hold turns on the firmware boot menu and --keys sends keystrokes, which is holding ESC by hand and about as reliable.

--console, --hold and --single all change something. --restore puts the serial port, the boot menu and the kernel command line all back.

RUNNING THIS WITHOUT A TERMINAL

Two passwords get asked for: the passphrase to secrets.kdbx, and the sudo password on the hypervisor if the login there has not been given passwordless sudo. Both are fine when a person is sitting in front of it and neither works at all when something else is driving -- tCMS's reprovision button, a cron, CI -- because there is no terminal to ask at. The run dies several minutes in, on a prompt nobody will ever see.

So hand them in first, on standard input, with --credentials:

printf 'keepass: %s\nsudo: %s\n\n' "$PASSPHRASE" "$SUDO" \
  | bin/provision --credentials vm.example.test

One name: value per line, read until a blank line. The flag is not optional sugar: nothing reads standard input unless you ask it to, because a caller that is not a terminal, meant to hand nothing over, and never closes its end would otherwise block here forever -- which is the same hang, moved earlier.

A name you leave out is one you get asked for, so handing in the sudo password and typing the passphrase works, if somebody is there to type it. The names are keepass and sudo, and one it does not know is an error rather than something to ignore -- a misspelled keypass that quietly meant "prompt for it" is exactly the hang this avoids. See perldoc Trog::Credentials.

Giving the login passwordless sudo on the hypervisor is still the better answer where you control it:

youruser ALL=(ALL) NOPASSWD: ALL

TPMs

A guest gets an emulated TPM if, and only if, the hypervisor has a real one and swtpm is installed to share it out. bin/preflight says when you have the first and not the second.

The condition is the point. An emulated TPM keeps its state in a file on the hypervisor, so a guest that seals a key to it has sealed that key to a file sitting beside its own disk image -- whoever takes the disk takes the TPM with it. That is worth having when the hypervisor's disk is protected by hardware of its own, and worth nothing when it is not, and a TPM that cannot keep a secret is worse than no TPM: something on the guest will use it and believe it.

So guests on a hypervisor with no TPM are built without one, and anything on them that wants to seal a secret does without and says so. systemd-creds falls back to its host key, and tPSGI's vault key with it.

UBUNTU DEPS

virt-manager bridge-utils

swtpm swtpm-tools as well, on a hypervisor with a TPM -- see above.

An apt mirror is no longer among them. A guest running the aptmirror recipe is one, and _global's mirror is what points the fleet at it -- so nothing is assumed of the hypervisor that nobody said. bin/preflight says when a fleet has no mirror configured, and what it costs.

IF YOU ENCOUNTER MYSTERIOUS 'cannot access image' issues

It's almost certainly that you need to add an override for those dirs in either an apparmor config for libvirt-qemu or selinux.

This and a profile for /usr/libexec/virtiofs-better -- which has none anywhere -- are the two apparmor items still left to an operator. They are the last of what bin/setup_provisioner is for; everything else that script used to claim is either a recipe now or the runner's own business.