Skip to content

HOWTO: Bridged Qemu Networking

Jump to bottom
Nathaniel Wesley Filardo edited this page Sep 8, 2021 · 1 revision

Introduction

By default, cheribuild runs qemu with its so-called "user-mode networking" stack (i.e., SLiRP). This allows the emulated (CHERI) guest to make outbound TCP connections, additional command line options allow for bridging host TCP ports to in-guest listeners, and cheribuild even has qemu run smbd for the guest, but it's still a pretty restrictive networking setup. As such, we've had some requests for a bridged setup, where the emulated guest is on equal network footing with the machine running it. This not something cheribuild can completely automate away (one might speculate as to how firmly UNIX network stacks remain stuck in 1990, but that's another matter) and so will require some work if you wish to adopt this kind of setup, but we can hope to make the process as painless as possible.

Overview

In broad strokes, we're going to:

  1. Create a network bridge device on the host
  2. Move the host's current network interface under that bridge
  3. Move the host's network setup to configure the bridge instead
  4. Create a persistent TAP device under the bridge for qemu's use
  5. Run qemu with modified network flags

Steps 1 - 4

FreeBSD Hosts

Please note that this is only very lightly tested at the moment; some experimentation is doubtless required.

  1. Steps 1 through 3 can all be accomplished by changing /etc/rc.conf and rebooting. Use ifconfig to determine which network interface is the default (on many VM images it's em0 or hn0). Assuming that's em0 (and that you're using DHCP to get your network configuration) add to rc.conf these lines (or adjust appropriately):

    cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0 SYNCDHCP"
ifconfig_em0="up"

    I suggest you verify that networking continues to function on the host before proceeding. ifconfig bridge0 should show a sensible address, at least.

  2. Creating a persistent TAP device for user use requires several steps...

    1. setting some sysctls. Add to /etc/sysctl.conf these lines:

      net.link.tap_user_open=1
net.link.tap.up_on_open=1

    2. add a devd rule to /etc/devd.conf to control ownership of the TAP device. Replace user with your username on the host. If you routinely run everything as root (which we cannot recommend), you can skip this step.

      attach 100 {
  match "system" "ETHERNET";
  match "type" "IFATTACH";
  match "subsystem" "tap0";
  action "chown user /dev/tap0";
};

    3. run ifconfig per boot; add to /etc/rc.local (or create it with just these lines):

      ifconfig tap0 create
ifconfig bridge0 addm tap0

    Again I suggest rebooting to verify that everything is still functional and iterating until that's so. (ls -l /dev/tap* should show a /dev/tap0 owned by your user.)

Now skip to Step 5: Running qemu guests with TAP networking

Generic Linux Hosts

It's possible to dynamically reconfigure Linux's networking stack on the fly to have the requisite bridge device by running a series of commands as root. (Though note that this will likely play poorly with anything other than manually configured or static networking setups. That is, if you're using DHCP, perhaps change the configuration of the mechanism that starts DHCP.) You'll need ip (possibly packaged as iproute2), brctl (bridge-utils), and tunctl (uml-utilities) for this option.

Note that if your "generic Linux host" is a WSL2 VM, these instructions will allow your Linux host and qemu guest to communicate if you additionally manually assign an in-subnet address in the guest, but the guest will not be able to reach the WSL default gateway for reasons not yet understood (but likely something to do with ethernet addresses).

  1. Create the bridge:

    brctl addbr br0
ip link set dev br0 up

  2. Move the existing interface (presumed to be eth0) underneath the bridge. We're going to extract the current addresses and remove them from the raw device, too:

    ADDR4=$(ip -4 -o addr show dev eth0 | awk '{ print $4 }')
ADDR6=$(ip -6 -o addr show dev eth0 | awk '{ print $4 }')
GW4=$(ip -4 -o route show default | awk '{ print $3 }')

ip -4 route del default dev eth0
ip -4 addr del dev eth0 ${ADDR4}
ip -6 addr del dev eth0 ${ADDR6}

brctl addif br0 eth0

  3. Put the addresses back on the bridge:

    ip -4 addr add dev br0 ${ADDR4}
ip -6 addr add dev br0 ${ADDR6}
ip -4 route add default via ${GW4} dev br0

  4. Create the TUN device and join it to the bridge. Replace user with your username on the host.

    tunctl -u user -t tap0
brctl addif br0 tap0
ip link set dev tap0 up

Step 5: Running qemu guests with TAP networking

Now that the host OS is ready, we can run the guest with networking via this new TAP device. The simplest way to do this, so far, is to add a second network device to the qemu guests spawned by cheribuild, by using the --run/extra-options setting (if you're already using this, e.g., to pass -snapshot as well, then merely add the quoted bit here).

.../cheribuild.py run-riscv64-purecap --run/extra-options="-netdev tap,id=tapnet,ifname=tap0,script=no,downscript=no -device virtio-net-pci,netdev=tapnet,mac=52:54:AA:BB:CC:DD"

The guest should automatically bring up DHCP on vtnet0 and vtnet1 both (that is, on the qemu SLiRP network and the TAP network). If your host network is not configured by DHCP, it may be necessary to edit /etc/rc.conf in the guest disk image (perhaps by using the extra-files feature of cheribuild).

Clone this wiki locally