Skip to content
A demo running 4000 Firecracker microVMs.
Shell Python
Branch: master
Clone or download
acatangiu update to using firecracker v0.17.0
Signed-off-by: Adrian Catangiu <acatan@amazon.com>
Latest commit ef07b2f Aug 12, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github style: apply markdown linter on all .md files Aug 7, 2019
scripts
.gitignore Adding .gitignore Aug 7, 2019
0.initial-setup.sh Changes required to run demo as unprivileged user Nov 27, 2018
9.cleanup.sh worthless change Nov 30, 2018
CODE_OF_CONDUCT.md style: apply markdown linter on all .md files Aug 7, 2019
CONTRIBUTING.md style: apply markdown linter on all .md files Aug 7, 2019
LICENSE license: add Apache 2.0 license Dec 3, 2018
README.md style: apply markdown linter on all .md files Aug 7, 2019
copy-root-fs.sh initial demo commit Nov 20, 2018
extract-times.sh only plot the microvm boot-time Nov 21, 2018
firecracker update to using firecracker v0.17.0 Aug 12, 2019
gnuplot.script only plot the microvm boot-time Nov 21, 2018
microvm-tiles.py Revert "tiles: add dark grey for nevery-active VMs" Nov 30, 2018
one-time-setup.sh
parallel-start-many.sh Changes required to run demo as unprivileged user Nov 27, 2018
run-multiple-seq.sh initial demo commit Nov 20, 2018
start-firecracker.sh
start-many.sh Changes required to run demo as unprivileged user Nov 27, 2018
vmlinux initial demo commit Nov 20, 2018
xenial.rootfs.ext4
xenial.rootfs.id_rsa add ssh key to connect to guest Nov 20, 2018

README.md

Firecracker-demo

Disclaimer

This demo showcases Firecracker's agility and high-density capabiliies. It's been run on an EC2 I3.metal host (the defaults start 4000 microVMs) with an Ubuntu and an Amazon Linux 2 host OS, from an Ubuntu client.

Deviations from this setup will probably lead to issues and/or sub-par performance. If you want to help us support the demo on more platforms...we take pull requests :)

Step-by-Step Instructions

Get this repo on an EC2 i3.metal instance. Open two terminals/ssh-connections to the instance.

Terminal window 1

will show a heatmap of network traffic done by each microVM.

python3 microvm-tiles.py

Terminal window 2

will control the rest of the demo.

Raise the maximum processes limit.

sudo tee -a >> /etc/security/limits.conf <<EOL
ec2-user soft nproc 16384
ec2-user hard nproc 16384
EOL

Reload the ssh session to have the new limit applied.

Install additional dependencies: python3 and iperf3.

Fix permissions on /dev/kvm and the ssh key:

sudo chmod 777 /dev/kvm
chmod 400 xenial.rootfs.id_rsa

Create 4000 TAPs, configure networking for them and start 4k iperf3 servers each bound to their respective TAP.

sudo ./0.initial-setup.sh 4000

Start 4000 Firecracker microVMs

Use 6 parallel threads to configure and start 4000 microVMs. Each thread will get an equal slice of the 4k total and sequentially configure and issue the start command for each microVM.

The script will report total duration as well as mutation rate.

# start a total of 4k uVMs from 6 parallel threads
./parallel-start-many.sh 0 4000 6
# ... wait for it ... should take around 60 seconds ... watch the heatmap

Each microVM has a workload (iperf client) and will run it in a loop with a random sleep between iterations.

Looking at the heatmap you should see six 'snakes' advancing which are the microVMs that have just been powered up and are doing their first iteration of the workload. Once that's done, the random sleep will lead to random lighting of the heatmap.

Pick a microVM and play with it

Pick a number 0 <= ID < 4000. For this example 42 was chosen.

ID="42"
# get the IP for that microVM
ifconfig fc-$ID-tap0 | grep "inet "
       inet 169.254.0.170  netmask 255.255.255.252  broadcast 0.0.0.0

# IP of microVM on other side is *one less*
ssh -i xenial.rootfs.id_rsa root@169.254.0.169

You're now inside the microVM. Do as you please.

Let's make it stand out in the heatmap.

# stop the workload service
localhost:~# rc-service demo-workload stop
 * Stopping demo-workload ...                                    [ ok ]
# manually run iperf with a higher bandwidth than the rest
localhost:~# iperf3 -c $(./gateway-ip.sh) -b 104857600
# check out the heatmap

This microVM should now shine brighter in the heatmap.

Demonstrate the network throughput of this microVM:

localhost:~# iperf3 -c $(./gateway-ip.sh)
Connecting to host 169.254.0.170, port 5201
[  5] local 169.254.0.169 port 53392 connected to 169.254.0.170 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.72 GBytes  14.8 Gbits/sec    0    952 KBytes
[  5]   1.00-2.00   sec  1.67 GBytes  14.4 Gbits/sec    0    952 KBytes
[  5]   2.00-3.00   sec  1.76 GBytes  15.1 Gbits/sec    0    952 KBytes
[  5]   3.00-4.00   sec  1.69 GBytes  14.5 Gbits/sec    0    952 KBytes
[  5]   4.00-5.00   sec  1.69 GBytes  14.5 Gbits/sec    0    952 KBytes
[  5]   5.00-6.00   sec  1.66 GBytes  14.3 Gbits/sec    0    952 KBytes
[  5]   6.00-7.00   sec  1.67 GBytes  14.4 Gbits/sec    0    952 KBytes
[  5]   7.00-8.00   sec  1.77 GBytes  15.2 Gbits/sec    0    952 KBytes
[  5]   8.00-9.00   sec  1.76 GBytes  15.1 Gbits/sec    0    952 KBytes
[  5]   9.00-10.00  sec  1.42 GBytes  12.2 Gbits/sec    0    952 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  16.8 GBytes  14.4 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  16.8 GBytes  14.4 Gbits/sec                  receiver

iperf Done.

Plot the 4000 Firecracker microVMs boot times

To plot the boot times, on your local machine or any non-headless setup:

scp -i <identity-key> ec2-user@<i3.metal-ip>:firecracker-demo/{data.log,gnuplot.script} .
gnuplot gnuplot.script
xdg-open boot-time.png  # on Ubuntu. For other distros just use your default .png viewer.
You can’t perform that action at this time.