Inclusion of BastilleBSD

[tschettervictor]

Is there any chance of including BastilleBSD as a package?

[zvaultio]

Enabling other Jail managers are a very strong interest of ours, along with allowing user choice in a lot of other areas.
However there are complications, to this goal due to the way that TrueNAS is designed.

Having dug through the code for long enough we will be making a large post in a few days/weeks to explain our thoughts on the path forward and to get community input on the future direction of the project.

[tschettervictor]

Great!

[n-connect]

Good to hear!
Do you have a shortlist kind of approach for jail/bhyve mgmt? I'd like to raise CBSD as a possible alternative for iocage. Maybe the best is to move forward with a plug-in like approach with tested ones.

[tschettervictor]

Any progress on this yet?

Even just a "pkg install bastille" would do it for me, and having that change persist.

I'm eagerly awaiting migration to zVault, but this one thing is keeping me.

[JohnKaul]

I would imagine there would be a stronger argument for NOT adding a jail manager in an effort to reduce dependencies. The base system already has the ability to manage and create jails. If any (extra) manger would be installed I would wager Podman would be a safer bet.

[tschettervictor]

I think the idea of "plugins" where you can install custom pkgs would satisfy most users.

Iocage already has the built in GUI, which I would still be using if not for IX (not wanting to start anything here)

But whatever gets decided will tell me which path I will take.

[zvaultio]

Any progress on this yet?

Our overall design thoughts have been delayed to the effort that went into removing all the proprietary code that we had to remove.
We are working on that doc, but its not complete yet.

Even just a "pkg install bastille" would do it for me, and having that change persist.

The problem is that TrueNAS and thus by extension zVault, is an appliance. Even if we installed bastille in the image it's doubtful that it would be of much use. The middleware wipes all changes to the system configuration. Even if Bastille's configs could be kept entirely separate they would be wiped away with every update. We go into this problem in depth in the aforementioned doc that we're working on.

Suffice to say, TrueNAS was designed intentionally to limit a users options for what they are able to do on the system. Business wise this makes sense for an appliance developed by a company that has to provide support.
Changing this is possible, but would be a monumental task. And based on other comments we have received, would definitely upset some people (MSPs) who specifically want zVault to remain an appliance so they can charge their customers for support.
We're undecided on which way we want to go. We are continuing to investigate and will openly explain our findings to everyone.

I would imagine there would be a stronger argument for NOT adding a jail manager in an effort to reduce dependencies. The base system already has the ability to manage and create jails. If any (extra) manger would be installed I would wager Podman would be a safer be

People want jail managers because it makes managing a lot of jails easier. We understand this. Podman is an interesting option, but there are still some rough edges on FreeBSD. We are sure that they will be smoothed out in time though.

I think the idea of "plugins" where you can install custom pkgs would satisfy most users.

While this would be possible on a normal FreeBSD system, it is not really possible with the middleware that exists in TrueNAS (and as a result in zVault).

We go into much more detail about this in our design doc that we're working on but here's a quick breakdown.

              +-------------+
              |     UI      |
              +-------------+
                    ^   |
                    |   v
+---------+     +-------------+     +-----------+
|   OS    | --> |  Middleware | <--> | Database |
+---------+     +-------------+     +-----------+
                       |
                       v
              +-------------------+
              | Userland Utility  |
              +-------------------+

The OS loads the middleware. From that point on the Middleware is in charge. When you interact with the UI and ask it to do something, the UI sends that instruction to the middleware. The middleware then checks the configuration for that task in the database. The database returns the configuration, and then the middleware instructs the userland utility to do whatever.

This is why everyone has said not to make changes at the CLI in TrueNAS. Those changes do not get saved in the database and so are never applied.

Example. If you edit your SMB conf. When you go to restart SMB to apply your changes. The middleware checks the database for the SMB configuration. The middleware then writes that file to the OS and starts the Samba service. Any changes you made are overwritten by the middleware. If you try to change them manually and restart Samba, the middleware will once again check the database then apply the saved configuration to disk and then restart samba.

The middleware will only allow the system to perform actions and configurations stored in its database. The database will only save configurations that have been presented to it by the middleware. Which means the database can only save configurations, that the middleware has been explicitly written to allow. Along with this, the UI must be explicitly developed to present the user with only the fields that the middleware is coded to interpret.

The system is designed to limit all actions to a small subset that iXsystems decided were allowable.

Even if you were to install something custom (scripts or side loaded binaries), during an update, any configurations not saved into the database are overwritten and effectively deleted.

The only way to change this is to "un-appliance" the system. However at that point, we're not dealing with a storage appliance anymore, you're dealing with a customized version of FreeBSD with a bunch of decisions that no longer make any sense since its no no longer an appliance.

The UI wont work anymore since is explicitly written to function with the middleware. It is useless on its own.

"Un-appliancing" zVault, would effectively make it simply a worse version of FreeBSD, than just using FreeBSD natively.

We will go into much more detail and depth about this in our doc, but I wanted to give you a short summary of some of the details.

[tschettervictor]

My solution for now is to clone the Bastille repo to my pool, add a tunable "bastille_enable=YES" and a post-init script "make -C install /path/to/bastille" as well as "cp /my/config/file /usr/local/etc/bastille/bastille.conf".

This works, but would be better to have official support.

[JohnKaul]

The middleware wipes all changes to the system configuration. Even if Bastille's configs could be kept entirely separate they would be wiped away with every update. We go into this problem in depth in the aforementioned doc that we're working on. ...

There is a lot to unpack there (in that entire post actually) and I'm mostly waiting for your write up but my thoughts keeps coming back to this line of thinking: could it (a possible "solution") be as simple as modifying this line? My thought is that if a jail.conf file be moved to somewhere else it wont get wiped out. Possibly a dumb thought and I should probably wait for that doc to keep from muddying the waters but...

https://github.com/zvaultio/zvio-os/blob/9c45f235a1084258998f0482a3eb43fe1b7c2c21/usr.sbin/jail/jailp.h#L37

EDIT:
Please allow me to expand a bit (to give my suggestion more of an opportunity to not be immediately dismissed and save you some time unpacking it). Bullet point expansion:

1. iocage was a good candidate because jail.conf files were stored in a separate location from the base. -e.g. /mnt/dataset/jails/jailname/jail.conf and the ui could just call the middleware, and then the userland iocage tool.

2. If jail were allowed to search at a different location for conf files like: master conf /mnt/dataset/jail.conf which contained an .include statement, jails could be created at any location essentially.

3. I'm not actually proposing an actual change to that hardcoded value because jail already accepts -f flag. So, my thought is could that be used to solve the issue of a jail manager's storage location?

4. The UI would call middleware which calls the userland jail utility.

[zvaultio]

My solution for now is to clone the Bastille repo to my pool, add a tunable "bastille_enable=YES" and a post-init script "make -C install /path/to/bastille" as well as "cp /my/config/file /usr/local/etc/bastille/bastille.conf".

We could include Bastille on the installed image easy enough, but it's not really a good solution to the overall problem. And what we dont want to do is start including a bunch of random things in the image to fill every request for some way that someone wants to hack the system to do something else. To do it properly, we'd have to turn Jails into a service, where you could enable iocage or Bastille. However that would take significant effort to build out not only the UI pages but the middleware code for it to support either or.

4. The UI would call middleware which calls the userland jail utility.

Except your missing a few steps in that chain that would need to be handled. The UI calls the middleware which calls the database which returns a conf which the middleware writes to the disk overtop of whatever currently is sitting on disk... and then the middleware calls the userland utility to start the jail.

Conf files in the system get continually overwritten by the middleware based on what's stored in the database. You can't think of zvault (or for that matter TrueNAS CORE) like you would where the conf files sitting in /etc are the source of truth. In zvault and TrueNAS, the database is the source of truth, the file system where the conf is sitting is only a temporary spot to save information from the database so the userland application an do what the database has saved inside it.

[josephholsten]

With the existence of https://daemonless.io images, it seems obvious that podman/ocijail support is important. I still like BastilleBSD for my basic jail configs, but I'd absolutely bet on hobbyists jumping on daemonless images if they've used linuxserver.io images elsewhere.