xigmajail (& xismabhyve) fails to install specified directory

[n-connect]

Hi @olevole ,

I've just seen the 2 XigmaNAS plugins and wanted to test the second part for my CBSD PR #767, with a virgin / embedded FB system.

I've followed the readmes, to the letter/screenshots.
It seems the plugin installs successfully, but the CBSD initialization fails for both plugins:

• on webgui level it does not take the 3rd level directory like /mnt/storage/jails or /mnt/storage/cbsd
• on SSH running the env=... initenv command drops a cdboot error -> some binaries are missing
• tried to give complettely bogus path /mmnt/storage/cbsdjb/, it also put the cbsd framework into the /mmnt/storage/ only.
• both plugins tried alone, as first install -> same results on init, either in webgui, or in SSH

Background info:

• XigmaNAS is "13.3.0.5 - Hesterion (revision 9979)"
• The whole XigmaNAS is ZFS. the data disk is mounted under /mnt (I do not see in webgui yet, how to get the additional ZFS pools directly under "/" it as in standard FB), so /mnt/data, or /mnt/storage depends on the instance.
• Checked the /var/log/cbsd-init.log & /var/log/cbsdd-init.log, the missing command not seen, just in stderr in console.

Is there's a limitation that CBSD initenv supports the 2nd level of dirs only? My other servers have them on the 2nd..

Anyway let me know if I'm missing something, happy to test. Thanks for the plugins!

SSH command:

xigmanas: ~# env workdir="/mnt/storage/cbsd" /usr/local/cbsd/sudoexec/initenv
-------[CBSD v.14.0.9a]-------
 This is install/upgrade scripts for CBSD.
 Don't forget to backup.
-----------------------------
Do you want prepare or upgrade hier environment for CBSD now?
[yes(1) or no(0)]
1
>>> Installing or upgrading
CBSD workdir first init: /mnt/storage
[Stage 1: account & dir hier]
 * Check hier and permission...
cmdboot: no such executable, CBSD dependency/requirement: column
initenv cmdboot error: missing necessary binaries
xigmanas: ~# env workdir="/mnt/storage/cbsd/" /usr/local/cbsd/sudoexec/initenv
-------[CBSD v.14.0.9a]-------
 This is install/upgrade scripts for CBSD.
 Don't forget to backup.
-----------------------------
Do you want prepare or upgrade hier environment for CBSD now?
[yes(1) or no(0)]
1
>>> Installing or upgrading
CBSD workdir first init: /mnt/storage
[Stage 1: account & dir hier]
 * Check hier and permission...
cmdboot: no such executable, CBSD dependency/requirement: column
initenv cmdboot error: missing necessary binaries

[olevole]

Hmm, for install CBSD on Xigma, standard initialization /of CBSD/ is not suitable because Xigma does not have some binaries that are required by the CBSD (yes, you have already noticed that).

To do this, the plugin downloads an archive with the missing files from the CBSD website: https://github.com/cbsd/xigmajail/blob/main/utils/cbsdjail_install.sh#L119

Another necessary thing is that you must specify the extra path where the cbsd will look for binary files: https://github.com/cbsd/xigmajail/blob/main/utils/init#L207

The CBSD has protection against the absence of necessary bin files, for this reason we call all external commands (which the cbsd uses) through macros ( _CMD, e.g.: date - DATE_CMD, cat - CAT_CMD ) that are initialized at the very beginning, before any scripts start running ( https://github.com/cbsd/cbsd/blob/develop/misc/cmdboot -> ~cbsd/cmd.subr )

So, just create /tmp/initenv.conf where you should specify the extra path to binary files ( e.g. /mnt/storage/XXX/bin )

post_cbsd_cmd_path="/mnt/storage/XXX/bin"

for example, take a look on my archive for Xigma 13:

oleg@host:/tmp$ wget -q https://www.bsdstore.ru/downloads/xigma/13/xigma-rootfs.tgz
oleg@host:/tmp$ tar xfz xigma-rootfs.tgz 
oleg@host:/tmp$ find xigma-rootfs/ -type f -print
xigma-rootfs/usr/sbin/etcupdate
xigma-rootfs/usr/sbin/valectl
xigma-rootfs/usr/sbin/ndp
xigma-rootfs/usr/sbin/setfib
xigma-rootfs/usr/sbin/makefs
xigma-rootfs/usr/sbin/idprio
xigma-rootfs/usr/bin/bsdcpio
xigma-rootfs/usr/bin/whoami
xigma-rootfs/usr/bin/column
xigma-rootfs/usr/bin/bsdcat
xigma-rootfs/usr/bin/strings
xigma-rootfs/usr/bin/mkimg
xigma-rootfs/usr/bin/make
xigma-rootfs/usr/bin/ulimit
xigma-rootfs/sbin/pfctl

If you copy xigma-rootfs to /mnt/storage, you can use /tmp/initenv.conf like this:

post_cbsd_cmd_path="/mnt/storage/bin/sbin:/mnt/storage/bin/usr/bin:/mnt/storage/bin/usr/sbin"

[n-connect]

Thanks for the details, I'll check the xigma-rootfs.
My guess is there is at least one additional binary missing.

(I've just started to put together a TrueNAS CORE build VM, to try to make a v14.x version (after checking the build process changes for v13.3 they have just released a beta, plus their clear intension to left the CORE/FB based Freenas successor dry out). Liked the various predefined jails, like NC (works), gitea(failed bc postgress11 out from pkg),gitlab(failed cause of v13.2 v13.3 mismatch). But jumped wagon after I've seen you made the 2 plugins here.)

So, let me know your Xigma install type, where the plugin install just worked, and the CBSD directory path (2nd or 3d level) for initenv. I just wanna recreate your way - to have a green/working path too.
The Xigma I've installed was via the "...RootOnZFS ...Expert" menu with default values (again TrueNAS CORE -> boot pool).

How do you generate the xigma-rootfs? I mean, what is you way determining and finding what binaries are necessary? And what could be still missing?
For example, a way to raise the log to debug level for initenv?

[n-connect]

Forgot about the number of directories in the path. Based on your answer there should not be a limitation about that, right? So if I give a /mnt/data/disk1/jails/ as CBSD workdir, should it work in general?

[olevole]

Let me check my instructions (and your information) again using the latest XigmaNAS release (came out a few days ago). I'll take a timeout for 2-3 days...

[n-connect]

Ok, thanks.

Some additional tests, data:

It seems the issue can be something about how deep is the target directory path for working dir (at least on XigmaNAS only(?).

---

Here comes into the picture XigmaNAS's own weird/buggy Disks/ZFS menu to create a new ZFS pool [for data disks], [while creating a vdev automatically] & set the mountpoint for that pool...

After several tries I've left the Disks/ZFS webui menu and created the data disk ZFS config via the usual 2 commands:

• zpool create -f -O compress=zstd data /dev/ada0p1 (-f to overwrite the previously created vdev over the buggy Disks/ZFS/Pool/Virtual Devises webui)
• zfs set mountpoint=/data data
• the created zpool and its mount works, but the Disks/ZFS/Pools not showing the vdev/pool etc. Only the "configuration" has correct data: but for zroot and data pools too. It must be some XigmaNAS bug. From now on Truenas CORE and its webui looks better for me, by the way.

With the above commands I have the similar zfs data disk setup with a /data, /work, /tank, whatever zfs mount under root directly.

---

Now, after we have a wokring ZFS data disk mounted as /data running the xigmajail setup and make the initenv with workdir `/data/cbsd˙ over extensions webui just works. The CBSD content goes under the speficied /data/cbsd/*

Created a quick jail, with all the default options but a 256Mbytes mem limit:

• created successfully, runs
• the webui based console works

However, right after the init & testjail runs, the CLI in SSH not working right after:

• SSH based cbsd jls command -> not found
• cbsd added via pkg, so it shows the proper files content under /usr/local/*
• env shows $PATH has /usr/local/bin/

Quick solution/fix:
EDIT: it does not need a reboot. An SSH logout/relogin will do the trick (this and the constant need to restart is very annoying in XigmaNAS, must be connected to their embedded way(?), even its running with ZFSonRoot) -> Please do make a change in the plugin codes with a hint of an SSH login after initenv done, OR SSH logout/relogin. I did created a test jail before a reboot and worked over webui, but was in SSH alrey started before plugin install. Running cbsd with full path in SSH works, confirming its "just" cli env issue this time, tested later with another pkg install directly in SSH. Command worked after lofout/relogin without ful path. Your thinking of cdboot/command substitue and the whole related works pays off for sure.

xigmanas: ~# /usr/local/bin/cbsd jls
JNAME  JID  IP4_ADDR  HOST_HOSTNAME    PATH                        STATUS
jail1  1    10.0.0.2  jail1.my.domain  /storage/cbsd//jails/jail1  On
xigmanas: ~# /usr/local/bin/cbsd expose
Expose for jail1:
3000 -> 22 (inaddr=0 -> 192.168.0.11 tcp)

---

Xigmabhyve installed too, giving the /data/cbsd/ over the webui as workdir, works. I need to add a public key. Can you give some link/hints what is accepted here? Newer used CBSD's API / bhyve before at all :)
Here I've found ssh pubkey calls: CBSD API SSI

Feature idea: an info,hint/sample command built in the plugin's UI fix the need at once. I guess this must be and SSH pubkey for the host system running CBSD with root level access? Will a sudo user's pubkey works? Lack of sudo user pubkey support would mean one must compromise the SSH access allowing direct root login for the host...

I'll bring and quote Xigmabhyve's stuff to its repo if next things come up.

---

All in all, it (the oriiginal issue of setting-up xigmajail) still can be CBSD specific or XigmaNAS related, I'm not sure.

[n-connect]

Ok, last one (came from wanting to create a test VM with bhyve in SSH):

To the binaries missing list/angle: runnig the *-tui commands they stuck into the initial help text ("-- dialog --msgbox Attention! This is automatic translation..."), nor Enter (OK button active), or Esc works, just repeats the errors below. Only killing the *-tui process works

cbsd jconstruct-tui
/usr/local/cbsd/subr/dialog.subr: get_construct_profile: not found
/usr/local/cbsd/subr/dialog.subr: f_dialog_title: not found
/usr/local/cbsd/subr/dialog.subr: f_dialog_backtitle: not found
/usr/local/cbsd/subr/dialog.subr: f_mustberoot_init: not found
last_cache_crc:  -> 18d613138333fb37649467feb15438b8
/usr/local/cbsd/subr/dialog.subr: f_dialog_default_fetch: not found
/usr/local/cbsd/subr/dialog.subr: curval: not found
/usr/local/cbsd/subr/dialog.subr: inc_menu_index: not found
/usr/local/cbsd/subr/dialog.subr: curval: not found
/usr/local/cbsd/subr/dialog.subr: inc_menu_index: not found
/usr/local/cbsd/subr/dialog.subr: curval: not found
/usr/local/cbsd/subr/dialog.subr: inc_menu_index: not found
/usr/local/cbsd/subr/dialog.subr: curval: not found

cbsd bconstruct-tui
/usr/local/cbsd/subr/dialog.subr: get_construct_profile: not found
/usr/local/cbsd/subr/dialog.subr: f_dialog_title: not found
/usr/local/cbsd/subr/dialog.subr: f_dialog_backtitle: not found
/usr/local/cbsd/subr/dialog.subr: f_mustberoot_init: not found
/usr/local/cbsd/subr/dialog.subr: f_dialog_default_fetch: not found
/usr/local/cbsd/subr/dialog.subr: curval: not found
/usr/local/cbsd/subr/dialog.subr: inc_menu_index: not found
/usr/local/cbsd/subr/dialog.subr: curval: not found
/usr/local/cbsd/subr/dialog.subr: inc_menu_index: not found
/usr/local/cbsd/subr/dialog.subr: curval: not found
/usr/local/cbsd/subr/dialog.subr: inc_menu_index: not found
/usr/local/cbsd/subr/dialog.subr: curval: not found
/usr/local/cbsd/subr/dialog.subr: inc_menu_index: not found
/usr/local/cbsd/subr/dialog.subr: curval: not found
/usr/local/cbsd/subr/dialog.subr: inc_menu_index: not found
/usr/local/cbsd/subr/dialog.subr: curval: not found
/usr/local/cbsd/subr/dialog.subr: inc_menu_index: not found

[n-connect]

@olevole

Could you check the differences with the "13.3.0.5" versus the version you've been created xigmajail/xigmabhyve originally?

If not yet, can we go through together what are the required binaries for CBSD to work in SSH/console in XigmaNAS? I'd like to close the host pf integration part fo CBSD. XigmaNAS' firmware style FreeBSD build is the best candidate testing it out as one of the hardest to make it work compared to "normal" FreeBSD ZFS root/boot pool installs.

[n-connect]

@olevole

I did a fresh XigmaNAS and Xigmajail install:

• the CBSD init step been done via sigmajail webUI, /data/cbsd/ has browsed out, which saved into the dialog as '/data/cbsd` correctly.
• a new jail was created, with all the default settings. After creation, if can be started
• than the jail settings has been modified -> start at boot
• after reboot the jail starts

All in all, it seems my previous XigmaNAS install, more precisely the data disk ZFS mount created the init <-> directory problems, I hope at least :)

The remaining problem is I can't run the *-tui commands, cause of missing binaries, only "kill" can close the cbsd jconstruct-tui command. Should I rename the issue for this problem?

Some screenshots:

• start:
  
• after pressing Enter on 'OK' there's this black screen with errors shortly, than comes back to the first screen.
  

[n-connect]

Hmm, for install CBSD on Xigma, standard initialization /of CBSD/ is not suitable because Xigma does not have some binaries that are required by the CBSD (yes, you have already noticed that).

To do this, the plugin downloads an archive with the missing files from the CBSD website: https://github.com/cbsd/xigmajail/blob/main/utils/cbsdjail_install.sh#L119

Another necessary thing is that you must specify the extra path where the cbsd will look for binary files: https://github.com/cbsd/xigmajail/blob/main/utils/init#L207

The CBSD has protection against the absence of necessary bin files, for this reason we call all external commands (which the cbsd uses) through macros ( _CMD, e.g.: date - DATE_CMD, cat - CAT_CMD ) that are initialized at the very beginning, before any scripts start running ( https://github.com/cbsd/cbsd/blob/develop/misc/cmdboot -> ~cbsd/cmd.subr )

So, just create /tmp/initenv.conf where you should specify the extra path to binary files ( e.g. /mnt/storage/XXX/bin )

post_cbsd_cmd_path="/mnt/storage/XXX/bin"

for example, take a look on my archive for Xigma 13:

oleg@host:/tmp$ wget -q https://www.bsdstore.ru/downloads/xigma/13/xigma-rootfs.tgz
oleg@host:/tmp$ tar xfz xigma-rootfs.tgz 
oleg@host:/tmp$ find xigma-rootfs/ -type f -print
xigma-rootfs/usr/sbin/etcupdate
xigma-rootfs/usr/sbin/valectl
xigma-rootfs/usr/sbin/ndp
xigma-rootfs/usr/sbin/setfib
xigma-rootfs/usr/sbin/makefs
xigma-rootfs/usr/sbin/idprio
xigma-rootfs/usr/bin/bsdcpio
xigma-rootfs/usr/bin/whoami
xigma-rootfs/usr/bin/column
xigma-rootfs/usr/bin/bsdcat
xigma-rootfs/usr/bin/strings
xigma-rootfs/usr/bin/mkimg
xigma-rootfs/usr/bin/make
xigma-rootfs/usr/bin/ulimit
xigma-rootfs/sbin/pfctl

If you copy xigma-rootfs to /mnt/storage, you can use /tmp/initenv.conf like this:

post_cbsd_cmd_path="/mnt/storage/bin/sbin:/mnt/storage/bin/usr/bin:/mnt/storage/bin/usr/sbin"

@olevole

I've tried the post_cbsd_cmd_path="/mnt/storage/XXX/bin" method, no success, the rootfs content still not found by which, nor cbsd *-tui worked. Then copied out all the binaries from the xigma-rootfs.tgz to their original place within the / structure . Hoped, it will make a difference, but no joy with cbsd *-tui commands.
After I've made a pkg upgrade on the cbsd package to 14.1.0, and tried the cbsd bconstruct-tui -> same result bunch of missing element, detailed in my previous comment.

Please help what could be the still missing parts, or not applied variabled causing these issues, I've tried iocage & bastilebsd (but first updated the connected xigmanas plugin to v13.3) on xigmanas they work well, but can't handle bhyve.

Is there a platform we can have a talk/live chat, to make things faster?

14.1.0 upgrade, this is the 3rd attempt, loging out and relogging to see if there's less error message. YES, here only the missing Xen cause errors :)

xigmanas: ~# env workdir=/data/cbsd cbsd initenv
-------[CBSD v.14.1.0]-------
 This is install/upgrade scripts for CBSD.
 Don't forget to backup.
-----------------------------
Do you want prepare or upgrade hier environment for CBSD now?
[yes(1) or no(0)]
1
>>> Installing or upgrading
[Stage 0: upgrading and migration data from 14.1.0 to 14.1.0]
pre-initenv-backup in progress ( can be disabled via ~cbsd/etc/initenv.conf ): initenv_backup_bases=3
  * pre-initenv-backup: [1/1] exprort/backup settings for: jail1 (prune: local.sqlite-20240616094018.bkp)
  * pre-initenv-backup: prune old backup dirs: /data/cbsd/var/db-20240616094018
[Stage 1: account & dir hier]
 * Check hier and permission...
    cmdboot: no such optional executable: xl, xen_feat disabled
[Stage 2: build tools]
chmod: cbsd:cbsd: No such file or directory
[Stage 3: local settings]
[Stage 4: update default skel resolv.conf]
[Stage 5: refreshing inventory]
[Stage 6: authentication keys]
[Stage 7: nodes]
[Stage 8: modules]
[Stage 9: cleanup]
 * Remove obsolete files...
[Stage X: upgrading]
>>> Done

---

Binary copy, no joy. cbsd *-tui commands still missing several things

xigmanas: ~#tree -a /mnt/ext/cbsd-jail/bin/
/mnt/ext/cbsd-jail/bin/
├── sbin
│   └── pfctl
└── usr
    ├── bin
    │   ├── bsdcat
    │   ├── bsdcpio
    │   ├── column
    │   ├── cpio -> bsdcpio
    │   ├── make
    │   ├── mkimg
    │   ├── strings
    │   ├── ulimit
    │   └── whoami
    └── sbin
        ├── etcupdate
        ├── idprio
        ├── makefs
        ├── ndp
        ├── setfib
        └── valectl

5 directories, 16 files
xigmanas: ~# cp -v /mnt/ext/cbsd-jail/bin/usr/bin/* /usr/bin/
/mnt/ext/cbsd-jail/bin/usr/bin/bsdcat -> /usr/bin/bsdcat
/mnt/ext/cbsd-jail/bin/usr/bin/bsdcpio -> /usr/bin/bsdcpio
/mnt/ext/cbsd-jail/bin/usr/bin/column -> /usr/bin/column
/mnt/ext/cbsd-jail/bin/usr/bin/cpio -> /usr/bin/cpio
/mnt/ext/cbsd-jail/bin/usr/bin/make -> /usr/bin/make
/mnt/ext/cbsd-jail/bin/usr/bin/mkimg -> /usr/bin/mkimg
/mnt/ext/cbsd-jail/bin/usr/bin/strings -> /usr/bin/strings
/mnt/ext/cbsd-jail/bin/usr/bin/ulimit -> /usr/bin/ulimit
/mnt/ext/cbsd-jail/bin/usr/bin/whoami -> /usr/bin/whoami
xigmanas: ~# cp -v /mnt/ext/cbsd-jail/bin/usr/sbin/* /usr/sbin/
/mnt/ext/cbsd-jail/bin/usr/sbin/etcupdate -> /usr/sbin/etcupdate
/mnt/ext/cbsd-jail/bin/usr/sbin/idprio -> /usr/sbin/idprio
/mnt/ext/cbsd-jail/bin/usr/sbin/makefs -> /usr/sbin/makefs
/mnt/ext/cbsd-jail/bin/usr/sbin/ndp -> /usr/sbin/ndp
/mnt/ext/cbsd-jail/bin/usr/sbin/setfib -> /usr/sbin/setfib
/mnt/ext/cbsd-jail/bin/usr/sbin/valectl -> /usr/sbin/valectl