Document and adjust Hoster to be executed in a Jail

[yaroslav-gwit]

Running Hoster in a Jail will bring a "true multi-tenancy" to life. You'll be able to simply create a new Jail, install Hoster in it, start the SSH service and pass the credentials down to the end user. Even if there are multiple users on the same host, they would not be able to know that someone else is using the same node and running even more VMs on it.

For this feature to be implemented, I'll have to create some static files in the Jail's FS to indicate resource constrains applied by rctl and link those with Hoster. Otherwise the end user might over-provision and crash their Jail.

I call this feature Hoster-As-A-Service 🔥 🤣🤣

Future refs:
churchers/vm-bhyve#267
https://github.com/lattera/articles/blob/master/freebsd/2018-10-27_jailed_bhyve/article.md
https://forums.freebsd.org/threads/bhyve-inside-jail.79935/
https://news.ycombinator.com/item?id=26742588

[yaroslav-gwit]

Here is the rough implementation plan.

Install process:

• Create a Hoster-in-a-Jail installation script (or an Ansible playbook), because the normal node-init.sh would not work

Related to jail_config.json

ZFS Mounts:

• Add a new jail configuration flag, that will allow mounting Hoster datasets within a Jail automatically

"mount_zfs_datasets": ["tank/vm-encrypted/hoster-jail/pool", "tank/client_pool_23452210878"]

It will execute these 2 commands on each Jail start:

zfs set jailed=on tank/vm-encrypted/hoster-jail/pool
zfs jail netbirdCheck tank/vm-encrypted/hoster-jail/pool

Add a new configuration flag jailed_hoster

This flag will make sure hoster binary gets the information normally not available within the Jail (e.g. CPU Info, etc) from a new location. It will also give me the ability to add some if statements when a VM or a Jail starts within the Jail, to apply a Jail-special set of parameters, etc.

Here is how it'll look like in the config file:

"jailed_hoster": true

Apply new Host sysctls

sysctl security.jail.vmm_allowed=1
sysctl security.jail.mount_zfs_allowed=1
sysctl security.jail.mount_devfs_allowed=1
sysctl security.jail.mount_tmpfs_allowed=1
sysctl security.jail.mount_procfs_allowed=1
sysctl security.jail.mount_zfs_allowed=1
sysctl -w security.bsd.unprivileged_proc_debug=1

Start devfs service on the Host

service devfs enable
service devfs start

Add the appropriate devfs permissions to /etc/devfs.rules

# Allow Hoster-in-a-Jail to use PF
[devfsrules_jail_unhide_pf=10]
add path 'pf*' unhide

TBD:

• Check why internal Jails don't start
• Check if self running in Jail, and apply a set of modified hoster init options
• Set the DNS server to 127.0.0.1 at the Jail boot time
• Generate the public network interface for the Jailed PF dynamically
• Make sure all these options are available in the jail_custom_parameters.conf:

zfs;
zfs.mount_snapshot;

allow.vmm;
allow.set_hostname = 0;
allow.raw_sockets;
allow.sysvipc;
allow.socket_af;
allow.quotas;
allow.raw_sockets;

allow.mount;
allow.mount.devfs;
allow.mount.tmpfs;
allow.mount.procfs;
allow.mount.zfs;

devfs_ruleset = 10; # our PF ruleset
enforce_statfs = 1;

mount.devfs;
mount.fdescfs;
mount.procfs;

[yaroslav-gwit]

This was a really nice idea initially, but I've hit way too many blockers along the way (at least for now?). Rctl doesn't work within a Jail, a bunch of hardware based checks related to monitoring too, and it will require too much effort to support all these exceptions/workarounds.

Closing for now. May come back to it later at some point, when there is more free time on my hands.