Move cbsd working dir

[Tectu]

I have a host where the cbsd working directory is installed on the same ZFS pool as the host system (/usr/jails/).
Due to upcoming changes in the drive layout of this host I'd like to move the CBSD working directory from /usr/jails/ to a different location (a different ZFS pool).

How cumbersome is it to perform this operation? Is it even possible? If so, how?
Ideally I'd like to migrate the location of everything CBSD related (including the location of the jails & VMs) to a different location and manually update various configurations needed to get everything back to working.

I'd appreciate your input on this.

[olevole]

Theoretically, you can use the possibilities of ZFS for the transfer of snapshots via zfs send | zfs recv.

Places that will need to change (where the paths appear) is (when cbsdd process and all jails/vm stopped):

1. /etc/rc.conf and cbsd_workdir= path
2. If you use bhyve virtual machines, It will be necessary to fix the simlink for each virtual disk in all machines., They indicate to /dev/zvol/'volname'/'path'

see: ls -la ~cbsd/jails-data/<vmname>/ | grep dsk

It seems that everything is needed.

[Tectu]

Alright, that sounds like this is very doable. I was worried whether this will require manual changes in a trillion of files, database entries and so on - but this seems very reasonable.

I will give this a try in the next week or two and update this task with new information to help future endeavors.

[Tectu]

@olevole Regarding point 2 of your instructions: is this as simple as removing the existing symlink and using ln -s to create a new one or is more magic required?

[olevole]

@Tectu changing symlink (e.g.: ln -sf) must be enough because the database uses relative paths for disk information

[Tectu]

I feel a bit anxious about this operation but I am gonna give this a try this week... Can't really push it back any further xD

[olevole]

Reinsurance measures are not canceled - use ZVOL snapshots and backups to remote servers ;-)

[Tectu]

I feel a bit anxious about this operation but I am gonna give this a try this week... Can't really push it back any further xD

Look everybody! I managed to push it back by more than a year! :D

So, this host is currently not using any virtual machines. Therefore, I assume that point 2 of your initial response does not apply. There are only jails.

Here's the dataset layout of my host:

NAME                               USED  AVAIL     REFER  MOUNTPOINT
nvme                              1.92M   225G       96K  /nvme
nvme/cbsd                           96K   225G       96K  /nvme/cbsd
storage                           10.6T  6.79T      128K  /storage
storage/nextcloud                  133G  6.79T      132G  /storage/nextcloud
zroot                             68.9G   154G       88K  /zroot
zroot/ROOT                        68.8G   154G       88K  none
zroot/ROOT/20230102               10.0G   154G     7.20G  /
zroot/ROOT/20230112                  8K   154G     7.19G  /
zroot/ROOT/default                58.8G   154G     5.52G  /
zroot/ROOT/default/blog            473M   154G      220M  /usr/jails/jails-data/blog-data
zroot/ROOT/default/cppproperties   199M   154G      114M  /usr/jails/jails-data/cppproperties-data
zroot/ROOT/default/git            2.34G   154G     2.34G  /usr/jails/jails-data/git-data
zroot/ROOT/default/irc             965M   154G      574M  /usr/jails/jails-data/irc-data
zroot/ROOT/default/nextcloud      4.77G   154G     1.93G  /usr/jails/jails-data/nextcloud-data
zroot/tmp                          892K   154G      496K  /tmp
zroot/usr                          844K   154G       88K  /usr
zroot/usr/home                     468K   154G      220K  /usr/home
zroot/usr/ports                    144K   154G       88K  /usr/ports
zroot/usr/src                      144K   154G       88K  /usr/src
zroot/var                         7.26M   154G       88K  /var
zroot/var/audit                    144K   154G       88K  /var/audit
zroot/var/crash                    144K   154G       88K  /var/crash
zroot/var/log                     4.22M   154G     1.50M  /var/log
zroot/var/mail                    2.47M   154G     2.27M  /var/mail
zroot/var/tmp                      208K   154G       88K  /var/tmp

There are 5 jails under zroot/ROOT/default, namely blog, cppproperties, git, irc and nextcloud.
I'd like to move everything cbsd related to the nvme/cbsd dataset.

My planned procedure looks like this:

1. Stop all jails
2. Stop cbsdd
3. For each jail:
   a. Create a snapshot: zfs snapshot zroot/ROOT/default/jailname@migration
   b. Move snapshot: zfs send zroot/ROOT/default/jailname@migration | zfs receive nvme/cbsd/jailname
   c. Update the mountpoint of the newly create dataset to /nvme/cbsd/jails-data/jailname-data
4. Update cbsd_workdir in /etc/rc.conf from /usr/jails (current value) to /nvme/cbsd
5. Reboot
6. Cross fingers really, really hard
7. Check for success
8. For each jail, delete the old dataset: zfs destroy -r zroot/ROOT/default/jailname

@olevole Does this seem reasonable to you? Anything I overlooked? I would especially like to know whether the destination path of 3b is correct together with the new value of cbsd_workdir and the updating of the jail dataset mount point in step 3c.
Furthermore, I guess if I would feel confident I could do a zfs rename on each jail dataset rather than snapshot, send | recv then delete snapshot.

The host is currently running cbsd 13.1.20.

Thank you very much for your immense efforts in developing and supporting cbsd!

[olevole]

Hmm, path also saved in SQLite databases, e.g.:

cd ~cbsd/var/db
sqlite3 local.sqlite '.dump' > x.sql

it will probably also be necessary to replace the paths in x.sql dump and make an import to new CBSD workdir.

But.. why don't you want to use CBSD export / import? dataset not very big. In addition, you can adjust the level of compression (for example, turn it off to speed up copying, e.g.: cbsd jexport jname=myjail compress=0 - check cbsd jexport --help). E.g.:

service cbsdd stop

#!/bin/sh

myjails=$( cbsd jorder )

for i in ${myjails}; do
  echo "exporting $i:"
  cbsd jexport jname=${i}
done

3. Change workdir in /etc/rc.conf to NVME path

4. Just import images from old ZROOT:

cbsd jimport /usr/jails/export/<jail>.img  

You can import the smallest image first and immediately check before import other

[olevole]

hint for myself ;-) : definitely, we must develop a convenient mechanism for migrating environments from one pool to another, this becomes very relevant for the [multi/pool binded] cluster/installation

[Tectu]

I'm happy to report that this operation was a success!
I followed your last instructions involving cbsd jexport / cbsd jimport. Everything went smoothly.

hint for myself ;-) : definitely, we must develop a convenient mechanism for migrating environments from one pool to another, this becomes very relevant for the [multi/pool binded] cluster/installation

I agree. However, I'd also like to point out that documentation is a big aspect for a tool like this. Over the years, I've shown cbsd to many people. Sometimes it's a bit difficult to get people to a point where they would actually try it as they got confused by the existence of the "old" and "new" documentation sources at the same time.
Personally, I think that cbsd is a great tool and far ahead of the competition in many aspects - but documentation is seriously lacking.
Not complaining - just trying to share some insights :)

Thanks again for your awesome work & support!

[olevole]

Thank you for the feedback.

About docs:

I hope with the increase in the number of contributors, the documentation will change. And it is already beginning to change, for example, Michael Reim started a series of articles that will help to know the CBSD better: https://eerielinux.wordpress.com/2022/11/15/exploring-the-cbsd-virtual-environment-management-framework-part-0-virtualization-overview/ ( Also available in Gemini ).

As for official docs at https://www.bsdstore.ru/en/docs.html - i started the structure in such a way that you could glue together small articles and get a complete CBSD handbook ('single file' link) ( perhaps Michael will succeed and we will launch the book in print) ) . Later, an attempt was made to transfer part of the content to https://cbsd.io (domain is very expensive and we will abandon it). However, the work was not complete and over time there was a large out of sync.

At the moment, I would like to transfer all the documentation to the markdown format and host it on the GitHub as WIKI pages ( https://github.com/cbsd/cbsd/wiki ). But this is a big enough job. Maybe someday I can hire a technical writer for this job.