MQ-ECN Software Prototype
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

#1 What is MQ-ECN MQ-ECN is a new ECN marking scheme to enable ECN for multi-service multi-queue production data center networks. MQ-ECN can achieve high throughput, low latency and weighted fair sharing simultaneously.

For more details about MQ-ECN, please refer to our NSDI 2016 paper.

#2 How to use ##2.1 Compiling MQ-ECN software prototype is implemented as a Linux queuing discipline (qdisc) kernel module, running on multi-NIC servers to emulate switch hardware behaviors. So you need the kernel headers to compile it. You can find available headers on your system in /lib/modules. To install the kernel headers, you just need to use the following command:

$ apt-get install linux-headers-$(uname -r)

Then you can compile MQ-ECN kernel module:

$ cd sch_dwrr
$ make

This will produce a kernel module called sch_dwrr.ko. I have tested it with Linux kernel 3.18.11. MQ-ECN kernel module is built on the top of Token Bucket Filter (tbf) and Deficit Round Robin (drr) scheduler in Linux kernel.

##2.2 Installing MQ-ECN replaces token bucket rate limiter module. Hence, you need to remove sch_tbf before installing MQ-ECN. To install MQ-ECN on a device (e.g., eth1):

$ rmmod sch_tbf
$ insmod sch_dwrr.ko
$ tc qdisc add dev eth1 root tbf rate 995mbit limit 1000k burst 1000k mtu 66000 peakrate 1000mbit

To remove MQ-ECN on a device (e.g., eth1):

$ tc qdisc del dev eth1 root
$ rmmod sch_dwrr

In above example, we install MQ-ECN on eth1. The shaping rate is 995Mbps (line rate is 1000Mbps). To accurately reflect switch buffer occupancy, we usually trade a little bandwidth.

##2.3 Note To better emulate real switch hardware behaviors, we should avoid large segments on server-emulated software switches. Hence, we need to disable related offloading techniques on all involved NICs. For example, to disable offloading on eth0:

$ ethtool -K eth0 tso off
$ ethtool -K eth0 gso off
$ ethtool -K eth0 gro off

##2.4 Configuring Except for shaping rate, all the parameters of MQ-ECN are configured through sysctl interfaces. Here, I only show several important parameters. For the rest, see params.h and params.c for more details.

  • ECN marking scheme:
    $ sysctl dwrr.ecn_scheme
    To enable per-queue ECN marking:
    $ sysctl -w dwrr.ecn_scheme=1
    To enable per-port ECN marking:
    $ sysctl -w dwrr.ecn_scheme=2
    To enable MQ-ECN:
    $ sysctl -w dwrr.ecn_scheme=3
  • Per-port ECN marking threshold (bytes):
    $ sysctl dwrr.port_thresh_bytes
  • Per-queue ECN marking threshold (bytes) (i is queue index):
    $ sysctl dwrr.queue_thresh_bytes_i
  • Per-port shared buffer size (bytes):
    $ sysctl dwrr.shared_buffer_bytes

##2.5 WRR By default, MQ-ECN kernel module performs Deficit Weighted Round Robin (DWRR) scheduling algorithm. You can also enable Weighted Round Robin (WRR) as follows:

$ sysctl -w dwrr.enable_wrr=1