Running OSv on Firecracker

The easiest way to run OSv on Firecracker is to use a Python script firecracker.py. The firecracker.py automates the process of launching firecracker VMM executable in order to start OSv micro-VM. It does it by either submitting necessary REST api calls over UNIX domain socket to a running firecracker process or creating a configuration file and passing its path as an argument to execute firecracker (see --api_less option). For insight of why this script does what it does look at Firecracker quick start guide.

Step 0: Prerequisites

In order to run OSv on Firecracker you need bare-metal machine (AWS i3.metal or other Nitro-based EC2 instance should work as well) with Linux installed and KVM enabled and OSv build environment to build arbitrary images. Alternatively one can use Mikelangelo capstan to build images. In short, if one has already OSv development environment ready, firecracker should work out of the box. The firecracker.py will automatically download and install the latest version of the firecracker unless you point to an alternative location of firecracker executable through FIRECRACKER_PATH variable.

Step 1: Building and running

There is nothing special about building OSv images to run them on Firecracker. However please bear in mind that Firecracker boots OSv by directly executing build/release/loader.elf (which conforms to Linux vmlinux 64-bit ELF format) instead of usr.img used by QEMU and other hypervisors. The build/release/usr.raw file (the raw version of usr.img) can be used to mount a zfs or rofs disk with an application.

./scripts/build image=native-example #Add fs=rofs to build Read-Only image
./scripts/firecracker.py 

On average ROFS image can boot within 5-6ms whereas ZFS takes at least 50 ms to boot.

Please note that firecracker.py provides a subset of functionality similar to run.py with almost identical options:

./scripts/firecracker.py --help
usage: firecracker [-h] [-c VCPUS] [-m MEMSIZE] [-e CMD] [-i CMD] [-k CMD]
                   [-n] [-b BRIDGE] [-V] [-l] [-p PHYSICAL_NIC]

optional arguments:
  -h, --help            show this help message and exit
  -c VCPUS, --vcpus VCPUS
                        specify number of vcpus
  -m MEMSIZE, --memsize MEMSIZE
                        specify memory: ex. 1G, 2G, ...
  -e CMD, --execute CMD
                        overwrite command line
  -i CMD, --image CMD   path to disk image file. defaults to
                        ../build/release/usr.img
  -k CMD, --kernel CMD  path to kernel loader file. defaults to
                        ../build/release/loader-stripped.elf
  -n, --networking      needs root to setup tap networking first time
  -b BRIDGE, --bridge BRIDGE
                        bridge name for tap networking
  -V, --verbose         pass --verbose to OSv, to display more debugging
                        information on the console
  -l, --api_less        do NOT use socket-based API to configure and start OSv
                        on firecracker
  -p PHYSICAL_NIC, --physical_nic PHYSICAL_NIC
                        name of the physical NIC (wired or wireless) to
                        forward to if in natted mode

Networking

The difference is that unlike run.py, firecracker.py does not enable networking automatically. Instead, you have to explicitly pass -n option to make it automatically setup TAP device and pass IP address to OSv:

./scripts/firecracker.py -n
The tap interface fc_tap0 not found -> needs to set it up!
Setting up TAP device in natted mode!
[sudo] password for XXXXX: 
To make outgoing traffic work pass physical NIC name
OSv v0.54.0-25-g56baf71c
eth0: 172.16.0.2
Booted up in 8.35 ms
Cmdline: /go.so /httpserver.so  
Go version: go1.12.6

By default, firecracker.py creates TAP device in a NAT-ed mode and allows for incoming traffic from host at the IP address 172.16.0.2. To enable outgoing traffic from OSv guest and DNS resolution one needs to pass the name of the physical NIC (wireless or wired) using the --physical_nic option. If you want OSv to be available from outside the host then please use the --bridge option to pass the name of the bridge that could be created using .scripts/setup-external-bridge.sh. Firecracker.py delegates network setup to a number of shell scripts:

• ./scripts/setup_fc_networking.sh - the top-level script that orchestrates networking setup and delegates to other scripts; called from firecracker.py
• ./scripts/create_tap_device.sh - creates TAP device setup with NAT or bridge
• ./scripts/setup_dnsmasq.sh - sets up and starts dnsmasq service to provide DNS setup for outgoing traffic
• ./scripts/setup_iptables.sh - sets up IP tables to allow for outgoing traffic between TAP and selected physical NIC
• ./scripts/restore_iptables.sh - restores IP tables to the state before setup_iptables.sh called
• ./scripts/remove_dnsmasq.sh - destroys dnsmasq setup created by setup_dnsmasq.sh