vmrc-instructions.txt

Copyright (c) 2012-2014 Michael Dexter <editor@callfortesting.org>
Version: v.0.6.1

Thank you for trying vmrc, the Virtual Machine rc script.

If you did NOT just download this, redownload it for the lastest version
as it is highly-experimental and changing rapidly. See the "TO DO" section
for known issues and feature ideas.

CHANGES from .0.5 to .0.6

* Introduces multiple VM NIC support. Simply add 1-7 in "vm_addtl_nics"
in the VM configuration file. This introduced a limit of 100 VM's but
that can be increased with a documented, one-character change.
* Adds updated pfSense
* Minor cosmetic cleanup

SIGNIFICANT CHANGES from .0.4 to .0.5

* The host_vmroot layout has changed.
* /usr/local/vm/ is now /usr/local/vmrc/vm/
* The "provision" option has been replaced by mkvm.sh which is based on a
library of functions and templates.
* Experimental preflight script and multiple block device support added.
* Experimental ZFS on root support is included but requires the included
zfs-boot components or a recent snapshot.

This has come a long way from a 2005 jail(8) provisioning and management script
to a 2012 FreeBSD 9.0 bhyve build script to a series of sequential FreeBSD 10.*
VM provisioning and management scripts.

Because bhyve is only the latest multiplicity solution on FreeBSD, this
rc script includes basic jail and qemu support and could include chroot, simh
and GXemul support. Of these, QEMU is intended to help prepare OSs like OpenBSD
that require VirtIO and console adjustments prior to booting under bhyve.


### CHEAT SHEET ###

Unpack vmrc.tar or retrieve the latest version from GitHub:

https://github.com/michaeldexter/vmrc


## FreeBSD VMs ##

Run 'sh install-vmrc.sh' or read it and perform the steps manually.

Run templates/install-templates.sh to install all templates into:

/usr/local/vmrc/templates

Verify your host_nic and optional host_zpool in /usr/local/etc/vm.conf

The mkvm.sh script can provision a Virtual Machine based on one of the
included templates in /usr/local/vmrc/templates

mkvm.sh can be used in a captive manner or by passing in a template name and a
VM name. i.e.: sh mkvm.sh t_freebsd10 freebsd which will create a VM named
freebsd followed by an auto-generated number such as 3: freebsd3

Note that all VMs are distinguished by unique trailing ID numbers:

OK: vm0, freebsd1, openbsd2

NOT OK: vm0, freebsd0 and openbsd0

Consider using a numeric word to describe an OS version such as "freebsd-ten3"

Once a VM is provisioned, run:

/usr/local/etc/rc.d/vm                (Show usage)
/usr/local/etc/rc.d/vm start          (Loads and boots all VMs)
/usr/local/etc/rc.d/vm start vm0      (Start a specific VM)

The vm rc script can also be called with the service(8) command:

service vm start vm0

Alternatively there is "ISO" mode:
/usr/local/etc/rc.d/vm iso vm0        (Fetches and boots and ISO image w/ grub)

"ISO" mode will prepare a disk image to install the OS to and includes fetch
support for FreeBSD, Ubuntu and CentOS.


## Linux VMs ##

Copy a template to a VM directory such as /usr/local/vmrc/vm/linux0/linux0.cfg
and use "ISO" mode:

/usr/local/etc/rc.d/vm iso linux0


### COMMAND USAGE ###

/usr/local/etc/rc.d/vm commands can take in zero or more VM arguments.

'start', 'stop' and 'status' will act on all VMs in /usr/local/vm/ or
multiple VMs i.e.:

/usr/local/etc/rc.d/vm start vm0 vm2 vm3
/usr/local/etc/rc.d/vm status
/usr/local/etc/rc.d/vm status vm3


### COMMANDS ###

Top level commands without a VM name (i.e. /usr/local/etc/rc.d/vm start)

start		Load and boot all VMs in /usr/local/vm/
stop		Ungracefully terminate all VMs in /usr/local/vm/
status		List running VMs

All VMs in /usr/local/vm/ will be launched on boot! (see "CONSOLES" below)

Issue a "reboot" within the VM to shut it down. Cleaner ACPI shutdown is
available after the 10.0-RELEASE and is included in vmrc.

VM-specific commands (i.e. /etc/rc.d/vm attach vm0)

start		Load and boot VMs
stop		kill and 'destroy' VMs
load		Load a VM with bhyveload(8) or grub-bhyve()
boot		Boot a VM with bhyve(8)
destroy		'destroy' a VM
restart		Run the stop and start functions on a VM
attach		Attach to a VM's nmdm serial port or tmux session
debug		List bhyvectl(8) --get-all output
iso		Format storage device, fetch an ISO image, boot to it
grub		Launch grub with the VM's disk and ISO images
mount		fsck(8) and mount(8) a FreeBSD VM disk image (UFS only)
umount		umount(8) a FreeBSD VM disk image
jail		Launch a FreeBSD VM's disk image with jail(8)
format		Create and format a FreeBSD VM disk image
delete		Delete a VM and its images and directory tree


### VM LAYOUT ###

/usr/local/vmrc/vm/distributions/ OS distribution sets and ISOs
/usr/local/vmrc/vm/templates/     VM templates
/usr/local/vmrc/vm/vm#/vm#.conf   VM config file
/usr/local/vmrc/vm/vm#/vm#.img    VM raw disk image (format or copy in)
/usr/local/vmrc/vm/vm#/vm#.iso    Link to bootable ISO image (fetch or copy in)
/usr/local/vmrc/vm/vm#/mnt/       VM mount point for raw disk image


### CONSOLES ###

The included templates direct each VM's output to an "nmdm" serial device.

Alternatively it can be directed to "stdio" or a tmux session,

"Detached" nmdm and tmux-detached "vm_console" types are available.
All other examples are set to "nmdm" and a bonmdm and tmux-detached consoles
can be attached to with using the "attach" command, i.e.:

/usr/local/etc/rc.d/vm attach vm0

You can detach from nmdm consoles with "~ CTRL-d" and from tmux sessions with
"CTRL-b d".

NOTE: nmdm consoles have the advantage of using the in-base cu(1) command but
had a bug in 10.0R that would cause the VM to boot only upon first attachment.
tmux may be a wiser choice in production environments.


### DEPENDENCIES ###

tmux functionality requires ports/sysutils/tmux
ISO and non-FreeBSD VM support requires ports/sysutils/grub2-bhyve 
QEMU functionality requires ports/emulators/qemu(-devel)

Note: Early versions of the grub2-bhyve port include a gcc runtime requirement
which is in fact only a build requirement. This has been fixed.

grub2-bhyve version .22 and later support OpenBSD bsd and bsd.mp kernels but
not the bsd.rd install kernel. Consider flashrd (documentation to come)


### MISC ###

Please send questions, bug fixes, love mail and hate mail to:
Michael Dexter <editor@callfortesting.org>

FreeBSD VMs have their disk images fsck'd before boot given that fsck at boot
time is not always comprehensive and may result in VM boot failure. Jailed VMs
with dirty file systems may cause kernel panics. (bug?)

FreeBSD guests can be provisioned with "iso" mode but it will not set the needed
console. (A "smart" console is undergoing testing that would eliminate the need
for any console changes). For now, just as with vmrun.sh, start a "manual
configuration" console and run:

cat >> /etc/ttys << EOF
console "/usr/libexec/getty std.9600"   vt100   on secure
EOF

GNU/Linux users will find the "grub" command very useful for locating disks and
grub configuration files.

A few GNU/Linux-related features did not make it into the FreeBSD 10.0 release:

>3000M RAM support
The -w flag for better CentOS and RHEL performance


### DEVELOPER NOTES ###

GitHub: https://github.com/michaeldexter/vmrc

While patches are welcome, please take the time to understand how this script
works before proposing solutions to problems that might be more complex than
they appear. Ask questions!

vmrc aims to have zero dependencies to operate. It optionally uses external
utilities like grub2-bhyve, tmux, qemu and the like to add functionality but
will hopefully never depend on them.

The vm rc script and mkvm.sh perform two key functions:

1. Programmatically construct virtual machine boot devices (mkvm.sh/"iso" mode)
2. Manage the lifecycle operation of the virtual machines it creates

A "Virtual Machine" is comprised of a directory containing a bootable disk
image, a configuration file derived from a template, a mnt/ directory for the
attachment and mounting of the disk image (FreeBSD only) and an optional
preflight script.

mkvm.sh is largely a simplified FreeBSD installer that is comprised of
functions.

The vm rc script largely creates bhyveload(8) and bhyve(8) command syntax for
loading and booting virtual machines based on the criteria in the configuration
files such as the VM RAM allocation, block device to boot, network interfaces
etc. The vm rc script will output the bhyveload and bhyve command strings that
it generates prior to executing them. If you copy these out, they should work
on their own along with the preflight script if included.

Each VM directory in /usr/local/vmrc/vm/ is fully self-contained and is
only referenced by the vm script looking for VMs in the vm/ directory and then
using the unique numeric VM ID number at the end of the virtual machine.

For example: vm0 has the arbitrary name of "vm" and the VM ID of "0". VMs can
share the same name (vm0, vm1, vm2) but must have a unique ID (vm0, freenas1,
openbsd3). VMs vm0 and linux0 would have conflicting VM IDs.

This naming is extended to the contents of the vm/ directory (vm/vm0/vm0.img)

The VM ID is used to generate unique, non-conflicting tap# network interfaces.

With the introduction of multiple VM virtual network devices, a new convention
was needed to keep VM tap devices unique while preserving their obvious
association with a given VM. The solution is to add a suffix to the tap device,
indicating the NIC ID (vm0's tap0 -> tap01 or vm3's tap3 ->tap31).

This revealed a problem: ifconfig will now allow for tap ID's to be
prefixed with "0" and thus vm0's tap03 would be invalid. To solve this, we use
a double digit VMID that is prefixed with the arbitrary "8", resulting in vm0's
fourth network interface being tap8033 (8 | VM ID | Network Device ID).
Network device "3" is the fourth device because the first one is "0".


### SPECIAL THANKS ###

Thank you Neel, Peter, John and Tycho for your hard work on bhyve.

Thank you Devin for greatly improving this script and Allan at ScaleEngine for
testing it and providing resources for it and bhyve's improvement.


### mkvm.sh TO DO ###

fix: Homogenize templates to master
fix: Make sure devices and zvols are not left attached after installation
fix: ${var_name:=variable} syntax if not set...
fix: Correct rc behavior... vm_enable="YES"
fix: Shorten FreeBSD boot timeout: autoboot_delay="3"
test: Check extraction destinations. zvols will not mount to directories
	with .snap in them etc.
ext bug: The PC-BSD automounter will delete a mount point you use. Fix coming.
	f_mount does create a directory but may not help all cases.
feature: Implement zfs "mount" and "umount"
feature: Implement make installworld installation
note: Expanding .tar is faster than .txz. Maybe institutionalize.
feature: Bring back 8.* dist set support. Maybe repackage to new .txz layout.
feature: ZFS clone-based storage.
feature: ZFS dataset for each /dev/vm/* for versioning.
feature: getopts? Use bsdconfig-style environment variables?
feature: Have an IP prefix/subnet in vm.conf? Override in mkvm.sh?
feature: Better preparation of VMs for jail use
        sendmail_enable="NO"
        inetd_flags="-wW -a 192.0.2.23"
        rpcbind_enable="NO
feature: beadm?
feature: panicmail, firstboot?


### vm TO DO ###

*NOT* PACKAGE THIS YET - IT IS NOT READY AND IS CHANGING RAPIDLY
note: Tested on 10.0-RELEASE, PC-BSD 10.0 and FreeBSD 11 snapshots
fix: QEMU support - not seeing disk images even if present
fix: vm_destroy after tmux session
news: bhyve automatic console detection is being tested, eliminating tty issues
feature: a "disable" option for VMs to prevent boot upon host boot
test: Try every incorrect sequence of commands and see if checks work
See "# fix: " throughout the script
concern: loaders can be captive and hang: detach loader steps, check flow
feature: disable vm option - move to ~/disabled
missing: mount zpool to ~/vm#/mnt
test: Why are ubuntu VMs read-only upon (first) boot?
test: TrueOS, Does android have VirtIO?
fix: Test: acpi kill in FreeBSD 11-current
linux needs acpi-utils, openbsd doesn't support power button?
	hw.allowpowerdown=1 ?
fix: Ubuntu often has a captive loader, halting host system boot
fix: Test: host_vmroot without trailing / (make idiot-proof)
fix: any way to avoid //'s in paths but safely not add trailing one in conf?
fix: CHECK FOR CONFLICTING VM ID NUMBERS BEFORE BOOT
	(Consider a PID or otherwise style of keeping VMs unique)
fix: Cosmetic: stalled return after cu session clean up (cleared with ENTER)
fix: Line wrapping!
fix: Formatting: Remaining spaces in place of tabs, for multiline info()?
fix: Formatting: Any way to elegantly wrap long URL's?
fix: Merge f_iso and vm_load/vm_boot
fix: Concern: could a VM booted at startup precede full networking?
fix: Homogenize info/error/debug messages, lose some
fix: Check for a valid tmux session before attempting to attach?
fix: Add zvol and raw device support to grub and load
fix: Destroy VM zvols in delete function
fix: Add root-on-ZFS configuration for when the zfs_boot change is made
fix: Add pty console attachment option (/dev/pts/<n>), try a physical port
fix: Why does second run of 'bhvye' on a loaded vm simply give a new line?
fix: Does grub boot need the CD ISO?
fix: Can grub NOT wipe out all preceding on screen (debug) text?
fix: f_vmmcheck: Is vmm going to be built in by default? (not yet!)
fix: f_vmmcheck: Note left-justified reverse indenting
fix: Remove info's and DEBUG's when satisfied?
fix: NEVER too many checks
fix: Push more checks to separate f_functions?
fyi: bhyve: not loaded first: vm_open: Invalid argument or error 33
fix: Elegant way to use official VM images? (qemu-img conversion works)
	qemu-img convert -O raw freebsd.qcow2 freebsd.raw (needs console!)
fix: idea: Dump the load and boot commands plus diagnostic info to a log file.
fix: idea: Add a 'top' command function 'top -uaP' ? How to show only bhyve?
fix: idea: Consider what other commands may want to act on multiple VMs
fix: idea: Cloud! (What was the question?) (ssh is your friend)
fix: idea: Popular jail functionality
fix: idea: Have a separate jail shell-only (no rc) command for maintenance?
fix: idea: Config file hooks for commands at various steps?
fix: idea: "Master" VM zvol cloning?
fix: idea: Add chroot, simh and GXemul support if they have serial output
fix: idea: freebsd-update on boot? (external problem?)
fix: idea: Probe disk layout and fs types? (vmrun.sh probe fails on zvols)
fix: idea: Maybe check for /mnt/.snap on new fs's as a test
fix: idea: Consider an "installworld" method as per the jail man page?
fix: idea: config file-> flags like vm_flags="em0 root ..."
fix: idea: Some flag to indicate "don't boot this VM" (move from dir)?
fix: idea: Add full ext2/3/4 support to the "mount" option (fuse ext4)
Note e2fsprogs and e2fsck /dev/md0s1 and mount -t ext2fs /dev/md0s1 /mnt
Linux ZFS may be very helpful as a LCD FS

fix: Note Neel's vmrun.sh loop:
while [ 1 ]; do
<destroy>
<load>
<boot>
	if [ $? -ne 0 ]; then
		break
	fi # exit if you get a non 0 return
done