Butterfly connects Virtual Machines and control their traffic flow
C++ Shell CMake Protocol Buffer
Latest commit b9c9444 Jan 18, 2017 @outscale-jju outscale-jju tests: tab war is over
Signed-off-by: Jerome Jutteau <jerome.jutteau@outscale.com>



Butterfly connects Virtual Machines (VM) and controls their traffic flow.

Each VM traffic is contained in a specific VXLAN network and traffic is filtered by (EC2/Openstack-like) security groups.

Security groups can be applied to any VM interface, and contain a list of simple network rules (dropping traffic by default).

Using Butterfly

Butterfly is a daemon you can control over a network API.

It is packaged with a client mainly allowing you to add/remove/list virtual network interfaces and security groups.

You can of course directly code your calls to Butterfly's API. API message transport is based on ZeroMQ and messages are encoded in Protobuf format. Check out protocol for more details.

Here is an example of Butterfly with 6 Virtual Machines isolated in 3 networks (VNI 42, 51 and 1337).

Butterfly execution

Butterfly binds a dedicated NIC to send/receive VXLAN packets and binds a socket (default: tcp) to listen to queries on its API. If you use a DPDK compatible card, you won't be able to access the API through it.

You can build this configuration in few lines of client calls:

butterfly nic add --ip --mac 52:54:00:12:34:01 --vni 42 --id vnic_1
butterfly nic add --ip --mac 52:54:00:12:34:01 --vni 51 --id vnic_2
butterfly nic add --ip --mac 52:54:00:12:34:02 --vni 51 --id vnic_3
butterfly nic add --ip --mac 52:54:00:12:34:03 --vni 51 --id vnic_4
butterfly nic add --ip --mac 52:54:00:12:34:01 --vni 1337 --id vnic_5
butterfly nic add --ip --mac 52:54:00:12:34:02 --vni 1337 --id vnic_6

Tip: if you want to see what the graph looks like: run butterfly status and copy past the dot diagram in webgraphviz.com

You can edit security groups whenever you want, and virtual nics filtering will be updated. Here, we simply create a new rule to open the http port for the whole world and ask some vnics to use this security group.

butterfly sg add sg-web
butterfly sg rule add sg-web --ip-proto tcp --port 80 --cidr
butterfly nic sg add vnic_1 sg-web
butterfly nic sg add vnic_2 sg-web

Note: Butterfly API uses idempotence, meaning that two calls should produce the same result.

Installing Butterfly

The easiest way to install Butterfly is to download and install a package from github releases. You can also build Butterfly yourself (as shown in the next section) and run make install .

Building Butterfly

This building procedure has been tested on a fresh Centos7.

First, install some dependencies (jemalloc needs manual installation:

$ sudo yum update -y
$ sudo yum install -y gcc-c++ glibc-devel glib2-devel libtool libpcap-devel automake kernel-headers make git cmake kernel-devel unzip zlib-devel wget libstdc++-static
$ wget http://cbs.centos.org/kojifiles/packages/jemalloc/3.6.0/8.el7.centos/x86_64/jemalloc-devel-3.6.0-8.el7.centos.x86_64.rpm
$ wget http://cbs.centos.org/kojifiles/packages/jemalloc/3.6.0/8.el7.centos/x86_64/jemalloc-3.6.0-8.el7.centos.x86_64.rpm
$ sudo rpm -i jemalloc-devel-3.6.0-8.el7.centos.x86_64.rpm jemalloc-3.6.0-8.el7.centos.x86_64.rpm

Build Butterfly:

$ git clone https://github.com/outscale/butterfly.git
$ mkdir butterfly/build
$ cd butterfly/build
$ cmake ..
$ make

Preparing Your Machine

Configure Huge Pages

Butterfly needs some huge pages (adjust to your needs):

  • Edit your /etc/sysctl.conf and add some huge pages:
  • Reload your sysctl configuration:
$ sudo sysctl -p /etc/sysctl.conf
  • Check that your huge pages are available:
$ cat /proc/meminfo | grep Huge
  • Mount your huge pages:
$ sudo mkdir -p /mnt/huge
$ sudo mount -t hugetlbfs nodev /mnt/huge
  • You may want to add this mount in your /etc/fstab:
hugetlbfs       /mnt/huge  hugetlbfs       rw,mode=0777        0 0

Prepare DPDK Compatible NIC

Before being able to bind your port, you will need to enable Intel VT-d in your BIOS and have IOMMU explicitly enabled in your kernel parameters. Check DPDK compatible NICs and how to bind NIC drivers. Packetgraph also has an example on how to bind DPDK NICs.

Additionally, you may also want to isolate a specific core for Butterfly, check isolcpus kernel parameters.

Running Butterfly Server

To get help, see: butterflyd --help

For example, if you have a DPDK compatible NIC, Butterfly will use the first available DPDK port. If no port is found, a (slow) tap interface is created.

$ sudo butterflyd -i -s /tmp

If you don't have a DPDK compatible card, you can also init a DPDK virtual device (which is much slower than a DPDK compatible hardware).

For example, we can ask Butterfly to listen to the existing eth0 interface:

$ sudo butterflyd -i -s /tmp --dpdk-args "-c1 -n1 --socket-mem 64 --vdev=eth_pcap0,iface=eth0"

Alternatively, you can ask Butterfly to read a configuration file at init:

$ sudo butterflyd -c /etc/butterfly/butterfly.conf


Why Another Virtual Switch?

Because we just want a fast vswitch answering our simple needs:

  • simple API: EC2/Openstack security groups style
  • Have some VXLAN for network isolation
  • Have some firewalling per Virtual Machine based on security groups
  • Use as little CPU as possible (and let Virtual Machines use all other cores)
  • Ease a (Cloud) orchestrator to control the whole thing through a simple API

What's Behind Butterfly?

Butterfly is based on:

  • Packetgraph: creates network graph
  • DPDK: fast access to NICs (in Packetgraph)
  • NPF: firewalling (in Packetgraph)
  • ZeroMQ: message transport
  • Protobuf: message encoding/versioning

How Fast?

Benchmarks setup:

  • Two physical machines directly connected
  • A third machine remote setup and launch benchmarks using ./benchmarks/run_benchmarks.sh


  • OS: Centos 7 (3.10.0-327.18.2.el7.x86_64)
  • NICs: Intel 82599ES 10-Gigabit SFI/SFP+ (DPDK compatible used with vfio-pci driver)
  • CPU: AMD Opteron(tm) Processor 3350 HE
  • All details for host 1 and host 2 (same)

Results (nov 10 2016):

                     |  VMs on same host  | VMs on remote host |
| Ping (min/average) |  0.067ms / 0.110ms |  0.157ms / 0.169ms |
| TCP                |    15.9 Gbits/s    |     3.0 Gbits/s    |
| UDP                |     1.4 Gbits/s    |     1.7 Gbits/s    |


  • We are working on accelerating inter-host performances
  • UDP is really bad at the moment, we are working on it
  • We can get even faster with zero copy in vhost-user (in future DPDK versions)
  • We can get faster by embedding a more recent libc (make package-fat)
  • If you try to run some benchmarks, you may want to configure your CPU throttling. On Centos7, check cpufreq governors page

Well, we have a good margin for improvements :)

How to Connect a Virtual Machine to Butterfly?

Butterfly does not launch your Virtual Machine for you, it just creates a special network interface (vhost-user) so you can connect your Virtual Machine to it. Vhost-user interfaces are Unix sockets allowing you to directly communicate with Virtual Machines in userland.

We tested Butterfly with QEMU >= 2.5 and added the following parameters to the machine's arguments (to adapt):

Some shared memory between the guest and Butterfly:

-object memory-backend-file,id=mem,size=124M,mem-path=/mnt/huge,share=on -numa node,memdev=mem -mem-prealloc

For each network interface:

-chardev socket,id=char0,path=/path/to/socket -netdev type=vhost-user,id=mynet0,chardev=char0,vhostforce -device virtio-net-pci,netdev=mynet0,gso=off

For more details, check vhost-user dpdk guide.

Do You Support Any Containers?

Not yet, Butterfly only supports vhost-user network interfaces.

Anyways, connecting a container should be possible too.

Does Butterfly Support IPv6?

  • Virtual Machine traffic can be in IPv6.
  • Outer network (VXLAN's side) is made in IPv4 for the moment.

What If My Virtual Machine Crashes/Reboots/Stops?

Vhost-user interface will still exist until it is removed from Butterfly.

You can just restart your VM, it will be reconnected to Butterfly and run as before.

This is possible because Butterfly acts as "server" in vhost-user communication.

What If Butterfly Crashes?

Too bad, you can just restart Butterfly but the VM won't reconnect, as vhost-user "server" is located on Butterfly's side.

Filing an issue is very valuable to the project. Please provide the following information:

  • The operating system with its version
  • The Butterfly version (butterfly --version)
  • The Butterfly logs (check syslogs)
  • Is your system under memory pressure?
  • What was Butterfly doing? (Heavy traffic? Doing nothing? How many VMs?)
  • The estimated Butterfly uptime until crash
  • Do you have a way to reproduce it?

It may be soon possible to choose which one is the vhost-user server between QEMU and Butterfly, comming soon in DPDK :)

What Ss Butterfly’s License?

Butterfly is licensed under GPLv3.

Is There Any Authentification on the API or Protection?

The network API is currently not protected at all.

For now, it's up to you to secure your administration network where the Butterfly API listens.

On Which Port Does Butterfly Listen?

By default, Butterfly listens on the tcp:// port, but it's up to you!

Butterfly uses ZeroMQ for message transport and allows you to bind in different ways. (like tcp, ipc, inproc, pgm, ...)

Questions? Problems? Contact Us!

Butterfly is an open-source project, feel free to chat with us on IRC, open a Github issue or propose a pull request.

server: irc.freenode.org

chan: #betterfly