Using minimega to run dozens of Harvey VMs

Although we have scripts in the Harvey repo to run a single VM, if you want to run a network of Harvey VMs the scripts may turn out to be insufficient. That's where minimega comes in.

Build minimega

minimega (minimega.org) is a tool for starting and managing lots of VMs. You'll need the latest version, fetched from https://github.com/sandia-minimega/minimega. You can follow the directions on the Installing Minimega article to build minimega from source.

You'll also need OpenVSwitch, QEMU, and dnsmasq installed. Their Debian/Ubuntu packages are 'qemu-kvm', 'dnsmasq', and 'openvswitch-switch'. After installing dnsmasq, you may need to stop the daemon, with the command 'service dnsmasq stop'.

Once you've built minimega, assuming you're at the top level of the minimega repository you can simply run sudo ./bin/minimega to start it. You'll get a command prompt.

Prepare Harvey

You'll need to have the Harvey source built. You'll also want to start ufs to serve files, so run ./util/ufs -root $HARVEY &

Run the script

Copy the following into a file and call it something like harvey.mm:

# host tap
tap create harvey-net ip 192.168.0.1/16
dnsmasq start 192.168.0.1 192.168.1.1 192.168.254.254
dnsmasq configure 0 ip 00:11:22:33:44:55 192.168.0.2

# set up the vm config
vm config memory 512
vm config kernel /home/john/harvey/sys/src/9/amd64/harvey.32bit
vm config net harvey-net,00:11:22:33:44:55
vm config serial 1
vm config qemu-override add "-cpu host" "-cpu Opteron_G1"
vm config append service=cpu nobootprompt=tcp maxcores=1024 fs=192.168.0.1 auth=192.168.0.1 nvram=/boot/nvram nvrlen=512 nvroff=0 acpiirq=1

# launch a single VM
vm launch kvm cpu
vm start all

# start the web interface
web

clear vm config net
vm config net harvey-net
vm config append service=terminal nobootprompt=tcp maxcores=1024 fs=192.168.0.1 auth=192.168.0.2 nvram=/boot/nvram nvrlen=512 nvroff=0 acpiirq=1 user=harvey mouseport=ps2 vgasize=1024x768x24 monitor=vesa
vm launch kvm terminal[1-10]
vm start all

You'll need to change the path on the "vm config kernel" line to point at your own harvey.32bit file.

Now, at the minimega command line, run read /path/to/harvey.mm

Point your web browser at localhost:9001 and you should see something like this:

You can see in the screenshot that we've started one CPU server and 10 terminals. If you click on "Connect", you'll be taken to the console for that VM.

Modifications

Note the following lines in the script:

dnsmasq configure 0 ip 00:11:22:33:44:55 192.168.0.2
vm config net harvey-net,00:11:22:33:44:55

This configured dnsmasq to offer a specific IP to a specific MAC address, then specified that the VM should have that specific MAC address. We then launched one CPU server and reset the MAC addresses for future VMs with the commands clear vm config net and vm config harvey-net

If you want to boot multiple CPU servers but don't really care about IP addresses, you could run a script like this:

tap create harvey-net ip 192.168.0.1/16
dnsmasq start 192.168.0.1 192.168.1.1 192.168.254.254

vm config memory 512
vm config kernel /home/john/harvey/sys/src/9/amd64/harvey.32bit
vm config net harvey-net
vm config serial 1
vm config qemu-override add "-cpu host" "-cpu Opteron_G1"
vm config append service=cpu nobootprompt=tcp maxcores=1024 fs=192.168.0.1 auth=192.168.0.1 nvram=/boot/nvram nvrlen=512 nvroff=0 acpiirq=1

vm launch kvm cpu[1-10]
vm start all

web

Or if you wanted 5 CPU servers, each with static IPs:

tap create harvey-net ip 192.168.0.1/16
dnsmasq start 192.168.0.1 192.168.1.1 192.168.254.254
dnsmasq configure 0 ip 00:00:00:00:00:01 192.168.0.2
dnsmasq configure 0 ip 00:00:00:00:00:02 192.168.0.3
dnsmasq configure 0 ip 00:00:00:00:00:03 192.168.0.4
dnsmasq configure 0 ip 00:00:00:00:00:04 192.168.0.5
dnsmasq configure 0 ip 00:00:00:00:00:05 192.168.0.6

vm config memory 512
vm config kernel /home/john/harvey/sys/src/9/amd64/harvey.32bit
vm config serial 1
vm config qemu-override add "-cpu host" "-cpu Opteron_G1"
vm config append service=cpu nobootprompt=tcp maxcores=1024 fs=192.168.0.1 auth=192.168.0.1 nvram=/boot/nvram nvrlen=512 nvroff=0 acpiirq=1

vm config net harvey-net,00:00:00:00:00:01
vm launch kvm cpu1
vm config net harvey-net,00:00:00:00:00:02
vm launch kvm cpu2
vm config net harvey-net,00:00:00:00:00:03
vm launch kvm cpu3
vm config net harvey-net,00:00:00:00:00:04
vm launch kvm cpu4
vm config net harvey-net,00:00:00:00:00:05
vm launch kvm cpu5
vm start all

web

See the minimega.org website for more information.