Etalon - a Reconfigurable Datacenter Network (RDCN) emulator
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin updating IP gen in gen-switch Feb 28, 2018
click-etalon @ 2345533
etc should fix a bug Feb 28, 2018
experiments typo Apr 2, 2018
flowgrind-etalon @ 396f332
libVT @ 583125f
reTCP reTCP documentation Feb 24, 2018
LICENSE Merge pull request #1 from mukerjee/david_edits Mar 9, 2018


Etalon is a reconfigurable datacenter network (RDCN) emulator designed to run on public testbeds. We define an RDCN to be a datacenter network designed around adding and removing circuits in realtime to add bandwidth to portions of the network on demand. This has been seen in many research papers with a variety of technologies: e.g., 60GHz wireless (Flyways), optical circuit switching (c-Through, Helios, Mordia, etc.), or free-space optics (FireFly, ProjecToR). Some of these proposals incorporate a packet switch, which we model as well.

With Etalon, we wish to provide a platform where repeatable end-to-end experiments on RDCN can be emulated at scale. We have a paper focused on end-to-end challenges on RDCNs in submission. We recommend emailing us (mukerjee at cs cmu edu) for a draft to better understand the context.


Etalon emulates a datacenter by running multiple virtual hosts ("vhosts"; Docker containers) on different physical machines. Physical machines represent datacenter "racks" in Etalon. vhosts on the same physical machine connect to the physical NIC using a virtualized switch (macvlan via docker), using tc to limit vhost link bandwidth (similar to a host to ToR link in a real datacenter).

A separate physical machine emulates the reconfigurable switch in Etalon. It does this using a software switch (Click). Within the software switch, we emulate ToR VOQs, a circuit switch, and a packet switch. All parameters for these elements are adjustable (e.g., VOQ length, circuit switch reconfiguration penalty, link bandwidths/delays, etc.).

Each "rack" is connected to the software switch via a physical switch in the testbed (e.g., a 40Gbps switch in CloudLab's Apt cluster). We also assume a second (control) network connects each "rack" and the software switch for convenience.

Finally, to allow our system to scale beyond a single 40Gbps link, we use time dilation (libVT) to emulate many 40Gbps links simultaneously.

See our paper for more information.


We assume experiments are going to be run on CloudLab (Apt cluster) on r320 nodes. Some things may need to be adjusted if not, as we have not tested Etalon on other testbeds. Please send us pull requests if you find/solve issues on other testbeds.

Initial Setup

  1. git clone this repo on your local machine.

  2. Create an experiment on a testbed (e.g., use the “Etalon” profile on CloudLab to launch an experiment).

  3. Nicely ask the network administrator to set your network to MTU 9126 and turn on Ethernet flow control. (For CloudLab Apt machines, ask to enable Ethernet. Ask if other ethernet experiments are running on the same switch. If so, ask to be in your own VLAN.)

  4. Update the host information in etalon/etc/handles on your local machine.

  5. Enable ssh agent forwarding on your local machine (e.g., run ssh-add on your local machine).

  6. If not using the pre-made Etalon profile on Cloudlab, log into each node and install the reTCP kernel using etalon/bin/ (The cloudlab profile has this already installed on the nodes). This script assumes there is an etalon repo in your home directory.

  7. Log into each node (after doing step 4, you can use the handles to log in, e.g., go to etalon/bin and type ./s host1, ./s switch, etc.):

    • On the switch machine, run etalon/bin/ (either clone the repo, or if on cloudlab, just run /local/etalon-master/bin/

    • On the other machines, run etalon/bin/ (either clone the repo, or if on cloudlab, just run /local/etalon-master/bin/

      • This may take about a half an hour (or longer) depending on internet connectivity.
  8. The machines will reboot after the previous step is done. At this point there should be no need to log into the individual nodes, only the switch.

Running An Experiment

After finishing the initial setup, creating an experiment, running it, and collecting the results are all handled by our experiment framework, run on the software switch machine.

The basic workflow for running an experiment on Etalon is:

  1. On the software switch machine, launch /etalon/bin/

  2. In another window on the software switch machine, launch an experiment file script (e.g., cd /etalon/experiments/buffers; ./

The experiment file script will build a docker image (on the software switch machine), push the image to all of the "rack" machines, launch a "rack" of vhosts, run the experiment (e.g., running 16 flows for 40 seconds), and then collect the results (e.g., logs), and tar them. These logs can be processed by graphing scripts which we provide (see Graphing the Results below).

See experiments/buffers/ and experiments/buffers/ for an example of an experiment file script. Generally, an experiment file script will call initializeExperiment(image), loop through some switch configurations (e.g., changing ToR VOQ buffer sizes) while running some benchmark on each configuration (e.g., flowgrind or dfsio), and then call finishExperiment(). The resulting tarball will always be in the directory where the script was launched from, and be named based on the timestamp at launch. finishExperiment() prints out this timestamp to make it easy to distinguish multiple resulting tar files.

Graphing the Results

After running an experiment, there is a tar file with the results on the software switch machine. We assume that most users will want the option to archive the raw result data on their local machine, so we assume graphing scripts will be run on the local machine, not on the software switch machine.

To graph results:

  1. Copy the results tar file to your local machine. The file in etalon/bin can simplify this. Assuming your etalon/etc/handles file is setup correctly, will try to scp a results file from the folders experiments/buffers/, experiments/adu, or experiment/hdfs on the software switch machine to the local machine, given some timestamp. e.g., ./ 1519494787 will look in those directories for a tar file with 1519494787 in its name.

  2. Untar the results file, into a new results folder (macOS' built in unarchiver will do this automatically, on Linux you'll need to create a folder first and put the tar in that folder and unarchive).

  3. Call a graphing script with the results folder as an argument , e.g., cd experiments/buffers/ ; ./ ../../bin/1519494787-buffers. This expects a folder named graphs to exist in the current folder.

After running this you should have some graphs in a subfolder called graphs. We include all graphing scripts used to create the graphs in our paper. If you wish to create other graphs, experiments/ will likely be useful.

Directory Overview

Read the in each subfolder for more information.

  • bin: various runables (e.g., performance tuning scripts, click script generation, installation scripts).

  • click-etalon: our software switch emulation; a modified version of Click.

  • etc: various configurations scripts.

  • experiments: experiment framework and various example experiment file scripts.

  • flowgrind-etalon: our flow generator; a modified version of flowgrind.

  • libADU: an interposition library that tells the software switch how much data is waiting in the endhost stack.

  • libVT: a virtual time interposition library ("time dilation") that we use to scale the number of links.

  • reHDFS: our modified HDFS write replica placement algorithm for reconfigurable datacenter networks.

  • reTCP: our TCP congestion control variant for reconfigurable datacenter networks.

  • rpycd: a daemon that runs on each "rack" that allows the software switch machine to setup experiments on each "rack".

  • vhost: code and supporting files for building the vhost image (docker image).

Questions, Comments, Feedback

We'd be happy to hear if you're using our emulator in your research. Let us know if you have any questions, comments, or feedback by sending an email to mukerjee at cs cmu edu.