Communication between microVMs

[Anjali05]

I have set an environment where I can run multiple microVMs at once, using the scripts provided in this repo. I was trying to do network bandwidth measurement between two VMs to check if they can communicate with each other. I am getting the IP using the gateway-ip.sh which is like one more than the IP I use to ssh. I have tried with both, the one I use to ssh and the one in the gateway-ip.sh but I always get iperf3: error - unable to connect to server: Connection refused. The goal is to have a client and server setting to run few benchmarks. Do I need to change any network settings in the microVMs to facilitate communication between them?
Thanks.

[acatangiu]

Hi. Yes, you need no create the necessary network plumbing on the host to connect the two or more VMs between themselves or the internet.

This is not Firecracker specific, you basically need to configure routing on your host to properly route the packets leaving the VMs.

See for example https://www.linux-kvm.org/page/Networking and https://werewblog.wordpress.com/2015/12/31/create-a-virtual-network-with-qemukvm/

[Anjali05]

@acatangiu Thanks, I will try to set it up.

[Anjali05]

@acatangiu I read the article and I think the easiest way to do this is connect the tap devices attached to each VM to a bridge.

These are the steps I followed:

• I added the bridge by modifying /etc/netor/interfaces file by adding this,

auto lo br0
iface lo inet loopback 

iface eno49 inet manual 

iface br0 inet dhcp 
	bridge_ports eno49

• These are the network interfaces available on my machine:

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 128.110.154.104  netmask 255.255.252.0  broadcast 128.110.155.255
        inet6 fe80::9af2:b3ff:fec8:2934  prefixlen 64  scopeid 0x20<link>
        ether 06:7e:58:f6:af:8e  txqueuelen 1000  (Ethernet)
        RX packets 71900  bytes 4248504 (4.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 58887  bytes 7449494 (7.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:74:69:ec:d4  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno49: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 128.110.154.104  netmask 255.255.252.0  broadcast 128.110.155.255
        ether 98:f2:b3:c8:29:34  txqueuelen 1000  (Ethernet)
        RX packets 420165  bytes 34413180 (34.4 MB)
        RX errors 0  dropped 10  overruns 0  frame 0
        TX packets 76507  bytes 9776448 (9.7 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 3210  bytes 168804 (168.8 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3210  bytes 168804 (168.8 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

• I create the tap device using this script:

#!/bin/bash -e

TOTAL_TAPS="${1:-0}"  #Default to 0
echo $TOTAL_TAPS
WIRELESS_DEVICE_NAME="br0"

#enable ipv4 forwarding
#sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"

# Load kernel module
sudo modprobe kvm_intel

# Configure packet forwarding
sudo sysctl -w net.ipv4.conf.all.forwarding=1

# Avoid "neighbour: arp_cache: neighbor table overflow!"
sudo sysctl -w net.ipv4.neigh.default.gc_thresh1=1024
sudo sysctl -w net.ipv4.neigh.default.gc_thresh2=2048
sudo sysctl -w net.ipv4.neigh.default.gc_thresh3=4096


MASK_LONG="255.255.255.252"
MASK_SHORT="/30"


#setup all the tap devices, equal to the number of threads
for (( i=0; i<$TOTAL_TAPS; i++ ))
do
   TAP_DEV="tap"$i


   FC_IP="$(printf '169.254.%s.%s' $(((4 * $i + 1) / 256)) $(((4 * $i + 1) % 256)))"


   TAP_IP="$(printf '169.254.%s.%s' $(((4 * $i + 2) / 256)) $(((4 * $i + 2) % 256)))"
   FC_MAC="$(printf '02:FC:00:00:%02X:%02X' $(($i / 256)) $(($i % 256)))"

   echo "$FC_IP"
   echo "$TAP_IP"
   echo "$FC_MAC"

   sudo ip link del "$TAP_DEV" 2> /dev/null || true
   sudo ip tuntap add dev "$TAP_DEV" mode tap

   sudo sysctl -w net.ipv4.conf.${TAP_DEV}.proxy_arp=1 > /dev/null
   sudo sysctl -w net.ipv6.conf.${TAP_DEV}.disable_ipv6=1 > /dev/null

   sudo ip addr add "${TAP_IP}${MASK_SHORT}" dev "$TAP_DEV"
   sudo ip link set dev "$TAP_DEV" up

   sudo iptables -t nat -A POSTROUTING -o $WIRELESS_DEVICE_NAME -j MASQUERADE
   sudo iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
   sudo iptables -A FORWARD -i $TAP_DEV -o $WIRELESS_DEVICE_NAME -j ACCEPT
   echo "$TAP_DEV"
done

• The tap device configuration looks like this after running the scripts:

tap0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 169.254.0.2  netmask 255.255.255.252  broadcast 0.0.0.0
        ether 66:bc:c8:50:1f:cf  txqueuelen 1000  (Ethernet)
        RX packets 41  bytes 2290 (2.2 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 321  bytes 21062 (21.0 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 169.254.0.6  netmask 255.255.255.252  broadcast 0.0.0.0
        ether 06:7e:58:f6:af:8e  txqueuelen 1000  (Ethernet)
        RX packets 28  bytes 1716 (1.7 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 310  bytes 20628 (20.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

• After this I ran this command to make connect the tap to the bridge:

ip link set tap{id} master br0

• I boot the VMs using this interface:

curl_put '/network-interfaces/br0' <<EOF
{
  "iface_id": "br0",
  "guest_mac": "$FC_MAC",
  "host_dev_name": "$TAP_DEV"
}
EOF

• When I try to ssh into the VM I am not able to:

ssh -i xenial.rootfs.id_rsa root@169.254.0.1
ssh: connect to host 169.254.0.1 port 22: No route to host

• When I try to boot without running the command ip link set tap{id} master br0, I am able to ssh but there is no communication between the VMs[But ping google.com works].

I am not sure what I am missing here. I would really appreciate some help on this.

Thanks

[pojntfx]

@Anjali05 I've been trying to get communication to work between two VMs for the last few days as well - setting up communication from guest to host and host to guest works this way, but communication between microVMs is not possible. Ethernet broadcasts (i.e. with etherecho) work between them, but that's it. I would really appreciate an example on how to set up this I assume very common usecase.

[nik736]

@pojntfx Same for me. Did you find any solution? :) Thanks

[hyperreality]

I managed to get this working and blogged about it here:

https://devopschops.com/blog/communicating-between-firecracker-microvms-using-bridges/