Different runtimes / state of Bastille

[h-2]

First of all: this looks really amazing! I am currently using bastille to manage my jails, but there is quite a bit of overhead involved, as I manually create volumes for all my jails that I mount into the jail. And then there are upgrades and backups of course... So I really think this might simplify my setup!

I realise this project is quite new, and you probably have many other things to do, so please see this as a friendly user report, not like a demand to change anything 🙂

I am a bit confused about the different "runtimes" (podman vs appjail vs bastille). I think I understand why they exist, but it makes parsing the documentation a lot more difficult. For example, the documentation section has several pages (Permissions, ZFS storage, Networking) that seem to be exclusive to the podman backend. But as a reader, I need to figure this out myself. Maybe it would help to add "badges" or icons for each runtime, and clearly present those on every documentation page?

Anyway, which runtime should I use?

My first thought was: I should stick with bastille, because I know that. But the state of the bastille runtime is particularly unclear, the documentation does not mention it at all, but most images have a section for "bastille". bastille's own documentation is quite sparse on the topic. The version available in ports doesn't even have the bastille up command. Fleet documentation states It requires buildah and shares the host network stack (inherit). but bastille documentation itself says that vnet is supported.

So I thought, maybe I should use AppJail, because it is better supported and allows native-vnet (compared to podman) and most of my current jails have vnet. But then I looked at AppJail's documentation and thought "woah, seems a bit over-engineered". Also, the documentation states that they basically invented it, because OCI was not available on FreeBSD. Now OCI is available, so why should I still use AppJail? Furthermore, the configuration for AppJail seems to be a lot more complicated, typically split over 3-4 files (example), while podman and bastille each have a single much shorter file.

There are also questions that I have, that might help me decide, that would be nice to cover on some overview/comparison table. These include

• Firewall required: [I currently have vnet jails and regular ones created directly on the bridge; no PF)
• Configuration: one global file, one file per container, multiple files per container?
• Upgrades: automatic [when?]? On startup? Manual?
• Backups: sufficient to backup volumes? How to restore / roll-back after failed Upgrade?

[ahze]

Thanks, reports like this are very helpful!

Runtime badges in the docs: You're right. Permissions, ZFS, and Networking are written for podman, and nothing tells you that. Badges per page and a comparison table on the quick start are a good idea; I'll add them. We probably have some room to improve the docs in general as well.

Which runtime: Podman or AppJail. Both are supported and recommended, and both run the same OCI images, so you can switch later without changing images. CI tests every image on both before it's published. Podman feels like Docker: one compose.yaml per app, and most of the docs are written for it today. AppJail is closer to plain jails, so if you're used to managing jails it may feel more natural. Its config is spread over a few files because Director describes the jail itself as well as the container. You don't write those files by hand: they're generated with each image.

Bastille: Honest answer, it's experimental on our side. We just don't have the cycles at the moment to fully support Bastille, but we absolutely welcome patches and invite you to contribute if it's your runtime of choice. The Bastille section on each image page is generated, and we don't test it the way we test podman and AppJail. bastille up is newer than what's in ports, which is why you don't have it. The "shares the host network" note may well be out of date now that Bastille supports vnet for OCI jails. I'll check it and fix the docs. I wouldn't pick Bastille for this today unless you want to help us shake it out (which we welcome you to do!)

Your questions:

• Firewall: ith podman, pf is only needed for the default bridge mode with port mappings. Host networking needs no pf at all. Giving each container its own address on your existing bridge, like your vnet jails today, is something Fjord sets up for you (see below).
• Configuration: podman uses one compose.yaml per app or stack, plus an optional .env. AppJail uses a Director file plus a Makejail per app.
• Upgrades: Manual by default (pull, then recreate). Nothing updates on its own unless you set that up.
• Backups: Your data lives in the volumes you mount (we use /containers/<app>), so backing those up is enough; the image can always be pulled again. To roll back a bad upgrade, go back to the previous tag (or pin a digest) and restore the volume if the app migrated its data. On ZFS, a snapshot before upgrading makes that trivial.

Also have a look at Fjord, which is our answer to a lot of this. It's a web UI that installs the images with podman or AppJail and gives containers their own LAN address on your bridge. Before you update it tells you what changes; after, it checks the app stays up, and you can roll back. It's early and moving fast, but it's built for exactly the setup you're describing.