Reproduce testbed experiments of MQ-ECN paper
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.


Software Requirements

To reproduce Figure 7 and 8 of MQ-ECN paper, you need following software:

In addition, you also need to install Linux 3.18x kernel which supports DCTCP. We used 3.18.11 kernel in our experiments. We also provide a patch for Linux 3.18.11 kernel which allows users to adjust TCP RTOmin using sysctl.

Testbed Setup

In the testbed, 9 servers are connected to a 9-port server-emulated switch. One server acts as the receiver while the rest 8 servers act as the sender. The IP address of the receiver is The IP addresses of senders are,, ... The server-emulated switch has 9 NICs whose IP addresses are,, ... To avoid large segments, please disable offloading techniques (e.g., GSO, TSO and GRO) on the server-emulated switch.

To enable packet forwarding on the server-emulated switch:

$ sysctl -w net.ipv4.ip_forward=1

To ensure servers on different subnets (e.g., and can access each other, you need to add some routing entries on servers. For example, at the receiver (, I add a new routing entry as follows:

$ route add -net gw

To enable DCTCP on servers:

$ sysctl -w net.ipv4.tcp_ecn=1
$ sysctl -w net.ipv4.tcp_congestion_control=dctcp

If you have applied our patch, you can adjust TCP RTOmin (in milliseconds) as follows:

$ sysctl -w net.ipv4.tcp_rto_min=10


MQ-ECN Software

To install MQ-ECN qdisc kernel module, please follow the guidance. By default:

  • It performs Deficit Weighted Round Robin (DWRR). All the queues have the same quantum (1.5KB).
  • All the queues belonging to a switch port share the per-port buffer space in a first-in-first-serve bias.
  • Packets with DSCP value i are classified to queue i of the scheduler.

To set per-port buffer size to 96KB:

$ sysctl -w dwrr.shared_buffer_bytes=96000

To enable per-queue ECN/RED with the standard threshold (32KB):

$ sysctl -w dwrr.ecn_scheme=1
$ sysctl -w dwrr.queue_thresh_bytes_0=32000
$ sysctl -w dwrr.queue_thresh_bytes_1=32000
$ sysctl -w dwrr.queue_thresh_bytes_2=32000
$ sysctl -w dwrr.queue_thresh_bytes_3=32000

To enable per-queue ECN/RED with the minimum threshold (8KB):

$ sysctl -w dwrr.ecn_scheme=1
$ sysctl -w dwrr.queue_thresh_bytes_0=8000
$ sysctl -w dwrr.queue_thresh_bytes_1=8000
$ sysctl -w dwrr.queue_thresh_bytes_2=8000
$ sysctl -w dwrr.queue_thresh_bytes_3=8000

To enable MQ-ECN (the standard marking threshold is 32KB):

$ sysctl -w dwrr.port_thresh_bytes=32000
$ sysctl -w dwrr.ecn_scheme=3

For more parameter settings, please see params.h.

Traffic Generator

To install the traffic generator, please follow the guidance. After installation, you should start server on senders ( to

Running Testbed Experiments

To generate realistic data center traffic, you need to move the files in conf folder to conf folder of Traffic Generator. There are 3 files in conf folder.

  • DCTCP_CDF.txt gives the flow size distribution of the web search workload.
  • websearch_config.txt is a client configuration file to generate balanced traffic. In balanced traffic pattern, a flow is randomly mapped to four service queues with equal probabilities.
  • websearch_unbalance_config.txt is a client configuration file to generate unbalanced traffic. In unbalanced traffic pattern, a flow is randomly mapped to four service queues with probabilities of 10%, 20%, 30% and 40%.

For example, to generate balanced traffic with the average throughput of 900Mbps, please input the following command in the top Traffic Generator folder:

$ bin/client -b 900 -c conf/websearch_config.txt -n 5000 -l fct.txt -s 100 -r src/script/

In above command, 5000 is the total number of flows (requests). fct.txt is the file to store flow completion time (FCT) results. 100 is the seed to generate random numbers. I used 100 as the seed for both Figure 7 and 8. src/script/ is the script file to parse FCT results.


If you have any question about MQ-ECN testbed experiments, please contact Wei Bai.