ODL stress test performance report v1.3

OpenDaylight Performance Stress Test Report v1.3: Berrylium vs Boron

Download below, the latest NSTAT Performance Stress Tests Report in pdf form.

[02/07/2017]: Performance Stress Tests Report v1.3: "Beryllium Vs Boron" (pdf)

Executive Summary

In this report we present comparative stress test results performed on OpenDaylight Beryllium SR0 controller against OpenDaylight Boron SR0. For these tests the NSTAT (Network Stress–Test Automation Toolkit) [1] testing platform and its external testing components (Multinet [2] , OFTraf [3] , MT–Cbench [4] , nstat-nb-generator [5] ) have been used. The test cases presented in this report are identical to those presented in our previous performance report v1.2, so the interested reader is referred to [6] for comprehensive descriptions. As opposed to v1.2, in this report all test components have been executed within Docker containers [7] instead of KVM instances.

NSTAT Toolkit

The NSTAT toolkit follows a modular and distributed architecture. With the term modular we mean that the core application works as an orchestrator that coordinates the testing process. It controls the lifecycle of different testing components and also the coordination and lifecycle management of the testing subject, the OpenDaylight controller. With the term distributed, we mean that each component, controlled by NSTAT, can either run on the same node or on different nodes, which can be either physical or virtual machines. In the latest implementation of NSTAT we introduced the use of Docker containers [7] .

With the use of containers1 we can isolate the separate components and their use of resources. In older versions of NSTAT we were using CPU affinity [10] to achieve this resource isolation. In Fig.1 NSTAT toolkit architecture is depicted

The provisioning of docker containers along with their interconnection, is achieved with the use of 1) the docker–compose provisioning tool and 2) the pre–built docker images which are present at docker hub [8] . All containers were running on the same physical server.

Experimental setup

Details of the experimental setup are presented in Table 1.

Switch scalability stress tests

In switch scalability tests we test controller towards different scales of OpenFlow switches networks. In order to create these networks we use either MT–Cbench [4] or Multinet [2] . MT–Cbench generates OpenFlow traffic emulating a “fake” Open- Flow v1.0 switch topology. Multinet utilizes Mininet [10] and OpenVSwitch v2.3.0 [ [11] ](http://openvswitch.org/.” http://openvswitch.org/) to emulate distributed OpenFlow v1.3 switch topologies.

In our stress tests we have experimented with topology switches operating in two modes, idle and active mode: switches in idle mode do not initiate any traffic to the controller, but rather respond to messages sent by it. Switches in active mode consistently initiate traffic to the controller, in the form of PACKET_IN messages. In most stress tests, MT–Cbench and Multinet switches operate both in active and idle modes.

For more details regarding the tests setup, the reade should refer to the NSTAT: OpenDaylight Performance Stress Test Report v1.2, [6] .

Idle Multinet switches

Test configuration

topology types: ”Linear”, ”Disconnected”
topology size per worker node: 100, 200, 300, 400, 500, 600
number of worker nodes: 16
group size: 1, 5
group delay: 5000ms
persistence: enabled

In this case, the total topology size is equal to 1600, 3200, 4800, 6400, 8000, 9600.

Results

Results from this series of tests are presented in Fig.3(a), 3(b) for a ”Linear” topology and Fig.4(a), 4(b) for a ”Disconnected” topology respectively.

Idle MT-Cbench switches

Test configuration

controller: ”RPC” mode
generator: MT–Cbench, latency mode
number of MT–Cbench threads: 1, 2, 4, 8, 16, 20, 30, 40, 50, 60, 70 , 80, 90, 100 threads.
topology size per MT–Cbench thread: 50 switches
group delay: 500, 1000, 2000, 4000, 8000, 16000.
persistence: enabled

In this case switch topology size is equal to: 50, 100, 200, 300, 400, 800, 1000, 1500 2000, 2500, 3000, 3500, 4000, 4500, 5000.

Results

Results of this test are presented in Fig.5(a), 5(b), 6(a), 6(b).

Active Multinet switches

Test configuration

controller: with l2switch plugin installed, configured to respond with mac–to–mac FLOW_MODs to PACKET_IN messages with ARP payload 12
topology size per worker node: 12, 25, 50, 100, 200, 300
number of worker nodes: 16
group size: 1
group delay: 3000ms
topology type: ”Linear”
hosts per switch: 2
traffic generation interval: 60000ms
PACKET_IN transmission delay: 500ms
persistence: disabled

Switch topology size scales as follows: 192, 400, 800, 1600, 3200, 4800.

Results

Results of this test are presented in Fig.8.

Active MT-Cbench switches

Test configuration, "RPC" mode

controller: ”RPC” mode
generator: MT–Cbench, latency mode
number of MT–Cbench threads: 1, 2, 4, 8, 16, 20, 40, 60, 80, 100 threads,
topology size per MT–Cbench thread: 50 switches
group delay: 15s
traffic generation interval: 20s
persistence: enabled

In this case, the total topology size is equal to 50, 100, 200, 400, 800, 1000, 2000, 3000, 4000, 5000.

Test configuration, "DataStore" mode

controller: ”DataStore” mode
generator: MT–Cbench, latency mode,
number of MT–Cbench threads: 1, 2, 4, 8, 16, 20, 40, 60, 80, 100 threads,
topology size per MT–Cbench thread: 50 switches
group delay: 15s
traffic generation interval: 20s
persistence: enabled

In this case, the total topology size is equal to 50, 100, 200, 300, 400, 800, 1000, 2000, 3000, 4000, 5000.

Results

Results for test configurations defined in sections 3.4.1, 3.4.2 are presented in Figs.9(a), 9(b) respectively.

Stability tests

Stability tests explore how controller throughput behaves in a large time window with a fixed topology connected to it, Fig.10(a), 10(b). The goal is to detect performance fluctuations over time.

The controller accepts a standard rate of incoming traffic and its response throughput is being sampled periodically. NSTAT reports these samples in a time series.

Idle Multinet switches

The purpose of this test is to investigate the stability of the controller to serve standard traffic requirements of a large scale Multinet topology of idle switches.

During main test execution Multinet switches respond to ECHO and MULTIPART messages sent by the controller at regular intervals. These types of messages dominate the total traffic volume during execution.

NSTAT uses the oftraf [3] to measure the outgoing traffic of the controller. The metrics presented for this case are the OpenFlow packets and bytes collected by NSTAT every second, Fig.15.

In order to push the controller performance to its limits, the controller is executed on the bare metal and Multinet on a set of interconnected virtual machines.

Test configuration

topology size per worker node: 200
number of worker nodes: 16
group size: 1
group delay: 2000ms
topology type: "Linear"
hosts per switch: 1
period between samples: 10s
number of samples: 4320
persistence: enabled
total running time: 12h

In this case switch topology size is equal to 3200.

Results

The results of this test are presented in Fig.15

Active MT–Cbench switches

Test configuration, ”DataStore” mode

controller: ”DataStore” mode
generator: MT–Cbench, latency mode
number of MT-Cbench threads: 10
topology size per MT–Cbench thread: 50 switches
group delay: 8s
number of samples: 4320
period between samples: 10s
persistence: enabled
total running time: 12h

In this case, the total topology size is equal to 500 switches.

Results

The results of this test are presented in Fig.11, 12.

Test configuration, ”RPC” mode

controller: ”RPC” mode
generator: MT–Cbench, latency mode
number of MT–Cbench threads: 10
topology size per MT–Cbench thread: 50 switches
group delay: 8s
number of samples: 4320,
period between samples: 10s
persistence: enabled
total running time: 12h

In this case, the total topology size is equal to 500 switches.

Results

The results of this test are presented in Fig.13, 14.

Flow scalability tests

With flow scalability stress tests we try to investigate both capacity and timing aspects of flows installation via the controller NB (RESTCONF) interface, Fig.16.

This test uses the NorthBound flow generator [5] to create flows in a scalable and configurable manner (number of flows, delay between flows, number of flow writer threads). The flows are being written to the controller configuration data store via its NB interface, and then forwarded to an underlying Multinet topology as flow modifications, where they are distributed into switches in a balanced fashion.

The test verifies the success or failure of flow operations via the controller’s operational data store. An experiment is considered successful when all flows have been installed on switches and have been reflected in the operational data store of the controller. If not all of them have become visible in the datastore within a certain deadline after the last update, the experiment is considered failed.

Intuitively, this test emulates a scenario where multiple NB applications, each controlling a subset of the topology, send simultaneously flows to their switches via the controller’s NB interface at a configurable rate.

The metrics measured in this test are:

Add controller time (tadd): the time needed for all requests to be sent and their response to be received (as in [9] ).
Add controller rate (radd): radd = N / tadd, where N the aggregate number of flows to be installed by worker threads.
End–to–end installation time (te2e): the time from the first flow installation request until all flows have been installed and become visible in the operational data store.
End-to-end installation rate (re2e): re2e = N / te2e

In this test, Multinet switches operate in idle mode, without initiating any traffic apart from the MULTIPART and ECHO messages with which they reply to controller’s requests at regular intervals.