Controller stability test with idle Multinet switches

Test description

The purpose of this test is to measure the stability of controller to identify and sustain a topology and its traffic load demands, of idle OpenFlow switches, over a period of time. With the term idle we mean that the switches do not send any PacketIN OpenFlow messages, that will trigger the controller to respond with FlowMod messages. The only traffic that exists is the standard traffic for handshaking, keep alive messages, topology discovery and statistics messages.

We want to see how efficiently the controller performs under situations where switch topology scales to thousands of idle switches. For that reason, the topology emulator we use for this test is multinet, which allows creation of largescale distributed topologies. In addition we need a tool that will measure the traffic generated from controller towards to the switches. To perform this measurement we use oftraf network monitoring tool.

Usage

A switch scalability test with Multinet topology emulator, can be started by specifying the following options in NSTAT command line:

--test=sb_idle_stability
--sb-generator-base-dir=<Multinet dir>

Under the stress_test/sample_test_confs/<controller_name>/ directory, the JSON files ending in _sb_idle_stability_multinet can be handled as template configuration files for this kind of test scenario. You can specify them to the --json-config option to run a sample test. For larger-scale stress tests, have a look at the corresponding files under the stress_test/stress_test_confs/<controller_name>/ directory.

Nodes deployment

For this test, the minimum number of nodes needed, are 3.

NSTAT node
controller node
SouthBound emulator node (Multinet). In this case we may have more than one nodes and we use one as a master node, which is the reference node for the communication between NSTAT node and the rest of the nodes of Multinet topology cluster, the worker nodes

In order to deploy these nodes, based on docker containers, we have two options

download the prebuilt environment from DockerHub
build your own container locally using the provided Dockerfiles for proxy and no-proxy environments, under the path deploy/docker

In both cases, docker has to be installed and any user that will manipulate docker containers, must be added to the docker group. To deploy the required nodes, see installation wiki.

After deployment of docker nodes, update the NSTAT repository using the following steps

open a new terminal and execute the command

docker ps -a

the output of the above command will be similar to the following

 CONTAINER ID       IMAGE                                  COMMAND               CREATED              STATUS              PORTS    NAMES

 4c05473bb7c8       intracom/nstat-sdn-controllers:proxy   "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp   controller
 72e4572878e2       intracom/multinet:proxy                "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp   mn-01
 13d191e6ef75       intracom/multinet:proxy                "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp   mn-02
 60db64735a26       intracom/nstat:proxy                   "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp   nstat

get the container names of all docker containers you created

for each docker name execute the following command

   WAIT_UNTIL_RETRY=2
   docker exec -i $container_name /bin/bash -c "rm -rf /opt/nstat; \
      cd /opt; \
      until git clone https://github.com/intracom-telecom-sdn/nstat.git -b master; do \
          echo 'Fail git clone NSTAT. Sleep for $WAIT_UNTIL_RETRY and retry'; \
      done; \
      if [[ $container_name =~ mn ]]; then \
          until service openvswitch-switch start; do \
              echo 'Fail starting openvswitch service. Sleep for $WAIT_UNTIL_RETRY and retry'; \
              sleep $WAIT_UNTIL_RETRY; \
          done \
      fi"

where you should replace the $container_name with the container name of the corresponded docker node, acquired from previous step. As we can observe all multinet nodes have the mn prefix in their names. In this case we have 2 multinet nodes.

Download the prebuild environment

Edit test configuration file

The IP addresses of all deployed VMs and the credentials to open SSH connections, must be configured in the json configuration file of the sample test we want to run. This action must be done in nstat_node.

Run the command
```
docker ps -a
```
to get container names of
- NSTAT node
- Controller node
- SouthBound emulator nodes (all multinet nodes Master and worker nodes)

Get IP Addresses of all nodes

docker exec -i $container_name /bin/bash -c "ifconfig"

SSH into nstat_node
```
ssh root@<NSTAT_node_ip>
```
the password to connect is root123.
Edit json file /opt/nstat/stress_test/sample_test_confs/boron/boron_sb_active_scalability_multinet.json and change the following lines changing IP addresses and SSH credentials:
```
"nstat_node_ip":"<NSTAT_node_ip>",
"nstat_node_ssh_port":"22",
"nstat_node_username":"root",
"nstat_node_password":"root123",

"controller_node_ip":"<Controller_node_ip>",
"controller_node_ssh_port":"22",
"controller_node_username":"root",
"controller_node_password":"root123",

"sb_emulator_name":"MULTINET",
"sb_emulator_node_ip":"<Multinet_master_node_ip>",
"sb_emulator_node_ssh_port":22,
"sb_emulator_node_username":"root",
"sb_emulator_node_password":"root123",
```
For the sb_emulator_node_ip we can select one ip address from the SouthBound emulator nodes. The Master can also participate in the list of the worker nodes multinet_worker_ip_list and have a dual role. The only requirement for this setup is to place the master process to listen on a different port for REST requests than the worker process running on the same node with the master. The REST port of the master process is defined from topology_rest_server_port key of the configuration file. For example if we have the following configuration
```
"multinet_worker_ip_list":["<Worker_1_ip>", "<Worker_2_ip>", ... ],
"multinet_worker_port_list":[<REST_port_worker_1>, <REST_port_worker_2>, ... ],
```
and we assume that sb_emulator_node_ip is equal to the <Worker_1_ip> then topology_rest_server_port should not have the same value with the first element in the list of multinet_worker_port_list

In json file of the test, we must configure the values of the IP address list and the port list of REST interface of Multinet workers. Multinet workers addresses, are the addresses of SouthBound emulator nodes. There must be one-to-one relation between the multinet_worker_ip_list and the multinet_worker_port_list.
```
"multinet_worker_ip_list":["<Worker_1_ip>", "<Worker_2_ip>", ... ],
"multinet_worker_port_list":[3333, 3333, ... ],
```
In case of starting more than one worker processes on the same node, we separate them by changing the port number. For example if we want to start 2 worker processes on localhost we would do the following configuration
```
"multinet_worker_ip_list":["127.0.0.1", "127.0.0.1", ... ],
"multinet_worker_port_list":[3333, 3334, ... ],
```

Run the test

In order to run the test

Open a new terminal and execute the following command

docker exec -i nstat /bin/bash -c "export PYTHONPATH=/opt/nstat; source /opt/venv_nstat/bin/activate; \
python3.4 /opt/nstat/stress_test/nstat.py \
  --test=sb_idle_stability \
  --ctrl-base-dir=/opt/nstat/controllers/odl_boron_pb/ \
  --sb-emulator-base-dir=/opt/nstat/emulators/multinet/ \
  --json-config=/opt/nstat/stress_test/sample_test_confs/boron/boron_sb_idle_stability_multinet.json \
  --json-output=/opt/nstat/results.json \
  --html-report=/opt/nstat/report.html \
  --output-dir=/opt/nstat/results_boron_sb_idle_stability_multinet/"

Inspect results

Once test execution is over, inspect the results under

/opt/nstat/results_boron_sb_idle_stability_multinet

Configuration keys

The configuration keys that must be specified in the JSON configuration file are:

config key	type	description
`nstat_node_ip`	string	IP Address of the NSTAT node
`nstat_node_ssh_port`	string	the ssh port of the NSTAT node
`nstat_node_username`	string	username for ssh login in the NSTAT node
`nstat_node_password`	string	password for ssh login in the NSTAT node
`controller_name`	string	name of the used controller. This value is used in Controller Factory method to return the appropriate controller object. For this test it should be `ODL`
`controller_node_ip`	string	IP Address of the Controller node
`controller_node_ssh_port`	string	The ssh port of the Controller node
`controller_node_username`	string	Username for ssh login in the Controller node
`controller_node_password`	string	Password for ssh login in the Controller node
`sb_emulator_name`	string	The name of SouthBound emulator. This value is used in Generator Factory method to return the appropriate SouthBound emulator object. For this test it should be `MULTINET`
`sb_emulator_node_ip`	string	IP Address of the Multinet node. Based on the multinet documentation, this is the IP address of the Multinet master node. This configuration key currently is not in use and is reserved for future releases of NSTAT.
`sb_emulator_node_ssh_port`	string	The ssh port of the Multinet node
`sb_emulator_node_username`	string	username for ssh login in the Multinet node
`sb_emulator_node_password`	string	password for ssh login in the Multinet node
`sb_emulator_clean_handler`	string	executable that cleans up locally Multinet files cloned from Multinet repository. The root path of this executable, is defined by `--sb-generator-base-dir` command line parameter
`sb_emulator_build_handler`	string	executable that clones locally Multinet files from Multinet repository. The root path of this executable, is defined by `--sb-generator-base-dir` command line parameter
`controller_build_handler`	string	executable for building controller (relative to `--ctrl-base-dir` command line parameter)
`controller_clean_handler`	string	executable for cleaning up controller directory (relative to `--ctrl-base-dir` command line parameter)
`controller_get_handler`	string	executable for downloading the controller prebuild version from its repository and extracts it.
`controller_start_handler`	string	executable for starting controller (relative to `--ctrl-base-dir` command line parameter)
`controller_stop_handler`	string	executable for stopping controller (relative to `--ctrl-base-dir` command line parameter)
`controller_status_handler`	string	executable for querying controller status (relative to `--ctrl-base-dir` command line parameter)
`controller_statistics_handler`	string	executable for changing the period that the controller collects topology statistics (relative to `--ctrl-base-dir` command line parameter)
`controller_persistent_handler`	string	disables persistence of controller. This can be acchieved by adding the attribute `persistent=false` in file `<controller_base_dir>/etc/org.opendaylight.controller.cluster.datastore.cfg`
`controller_oper_hosts_handler`	string	makes a RESTCALL to the NorthBound interface of the controller in order to get the number of hosts from the operational datastore
`controller_oper_links_handler`	string	makes a RESTCALL to the NorthBound interface of the controller in order to get the number of links from the operational datastore
`controller_oper_switches_handler`	string	makes a RESTCALL to the NorthBound interface of the controller in order to get the number of switches from the operational datastore
`controller_oper_flows_handler`	string	makes a RESTCALL to the NorthBound interface of the controller in order to get the number of flows from the operational datastore
`controller_flowmods_conf_handler`	string	configures the controller plugins to respond with flow modifications on any PacketIN message with ARP payload
`controller_logs_dir`	string	controllers logs directory (relative to `--ctrl-base-dir` command line parameter)
`controller_port`	number	controller port number where OF switches should connect
`controller_restconf_port`	number	controller RESTCONF port number
`controller_restconf_user`	string	controller RESTCONF user name
`controller_restconf_password`	string	controller RESTCONF password
`controller_statistics_period_ms`	array of numbers	controller different statistics period values (in (ms))
`topology_rest_server_boot`	string	executable that boots up all REST servers on Multinet master and workers nodes. The root path of this executable, is defined by `--sb-generator-base-dir` command line parameter
`topology_rest_server_stop`	string	executable that stop's all REST servers initiated by Multinet on master and workers nodes. The root path of this executable, is defined by `--sb-generator-base-dir` command line parameter
`topology_server_rest_port`	number	the port that Multinet server will listen to
`topology_init_handler`	string	executable that initializes a Multinet topology. The root path of this executable, is defined by `--sb-generator-base-dir` command line parameter
`topology_start_switches_handler`	string	executable that stops a Multinet topology. The root path of this executable, is defined by `--sb-generator-base-dir` command line parameter
`topology_stop_switches_handler`	string	executable that stops a Multinet topology. The root path of this executable, is defined by `--sb-generator-base-dir` command line parameter
`topology_get_switches_handler`	string	executable that retrieves the number of booted switches in a Multinet topology. The root path of this executable, is defined by `--sb-generator-base-dir` command line parameter
`multinet_topo_size`	array of numbers	number of Multinet switches per Worker. Total switches is equal to topology_size * number_of_workers
`multinet_topo_type`	array of strings	type of Multinet topology {RingTopo, LinearTopo, DisconnectedTopo}
`multinet_topo_hosts_per_switch`	array of numbers	number of Multinet hosts per switch
`multinet_topo_group_size`	array of numbers	size of a group of switches
`multinet_topo_group_delay_ms`	array of numbers	delay between different switches groups (in milliseconds)
`multinet_switch_type`	string	the type of software switch, that will be used from our Multinet topology
`multinet_worker_ip_list`	array of strings	a list of all IP addresses of worker nodes
`multinet_worker_port_list`	array of numbers	a list of port numbers of all REST servers on worker nodes
`multinet_traffic_gen_duration_ms`	number	the duration in miliseconds, during which traffic will be generated from topology switches to the controller. This traffic is PacketIN OpenFlow messages which contain ARP payload and they trigger controller to respond with FlowMod OpenFlow messages
`multinet_interpacket_delay_ms`	number	the delay between transmited PacketIN OpenFlow messages
`oftraf_rest_server_port`	number	the port number on which oftraf tool will listen to monitor traffic of OpenFlow messages
`oftraf_test_interval_ms`	number	the interval of time in milliseconds between samples requests from oftraf
`number_of_samples`	number	total tests samples. These are samples gathered from oftraf requests
`java_opts`	array of strings	java options parameters (we use this usually to define java VM memory and garbage collector configuration)
`plots`	array of plot objects	configurations for plots to be produced after the test

The array-valued configuration keys shown in bold are the test dimensions of the test scenario. The stress test will be repeated over all possible combinations of their values.

The most important configuration key, is the number_of_samples which defines how many times the SouthBound interface of the controller will ba sampled by oftraf. Also, another important configuration key is oftraf_test_interval_ms. The combination of these two configuration keys gives the total duration of the stability test (total test duration = oftraf_test_interval_ms * number_of_samples). In this test there are no dimensions.

With the parameters of topology_group_size and topology_group_delay_ms we can gradually start a topology of switches and not start them all in once (actually this is the case where topology_group_delay_ms=0)

Plot configuration

See the plotting page.

Result keys

The result keys produced by this kind of test scenario and which can be used subsequently to generate custom plots, are the following:

Result key	type	description
`global_sample_id`	number	unique (serial) ID for this sample
`timestamp`	number	unique timestamp for this sample
`date`	string	date this sample was taken
`of_out_packets_per_sec`	number	the outgoing OpenFlow packets per second, from controller to switches
`of_out_bytes_per_sec`	number	the outgoing bytes per second of OpenFlow packets, from controller to switches
`of_in_packets_per_sec`	number	the incomming OpenFlow packets per second, from controller to switches
`of_in_bytes_per_sec`	number	the incomming bytes per second of OpenFlow packets, from switches to controller
`tcp_of_out_packets_per_sec`	number	the outgoing tcp packets with OpenFlow payload per second, from controller to switches
`tcp_of_out_bytes_per_sec`	number	the outgoing bytes of tcp packets with OpenFlow payload per second, from controller to switches
`tcp_of_in_packets_per_sec`	number	the incomming tcp packets with OpenFlow payload per second, from switches to controller
`tcp_of_in_bytes_per_sec`	number	the incomming bytes of tcp packets with OpenFlow payload per second, from switches to controller
`traffic_generation_duration_ms`	number	the duration in miliseconds, during which traffic will be generated from topology switches to the controller. This traffic is PacketIN OpenFlow messages which contain ARP payload and they trigger controller to respond with FlowMod OpenFlow messages
`multinet_size`	number	number of Multinet switches connected to the controller
`multinet_worker_topo_size`	number	number of switches created within a single worker node
`multinet_workers`	number	total number of worker nodes, deployed by Multinet
`multinet_topology_type`	string	Multinet network topology type {ring, linear, disconnected, mesh}
`multinet_hosts_per_switch`	number	number of Multinet hosts per switch
`multinet_group_size`	number	size of a group of switches
`multinet_group_delay_ms`	number	delay between different switches groups (in (ms))
`controller_node_ip`	string	controller IP address where OF switches were connected
`controller_port`	number	controller port number where OF switches should connect
`controller_java_xopts`	array of strings	controller Java optimization flags (`-X`)
`one_minute_load`	number	one-minute average system load
`five_minute_load`	number	five-minute average system load
`fifteen_minute_load`	number	fifteen-minute average system load
`used_memory_bytes`	number	system used memory in bytes
`total_memory_bytes`	number	system total memory in bytes
`controller_cpu_shares`	number	the percentage of CPU resources of the physical machine, allocated to the controller process
`controller_cpu_system_time`	number	CPU system time for controller
`controller_cpu_user_time`	number	CPU user time for controller
`controller_num_threads`	number	number of controller threads measured when this sample was taken
`controller_num_fds`	number	number of open file descriptors measured when this sample was taken
`controller_statistics_period_ms`	number	the interval (in (ms)) of the statistics period of the controller

The result keys in bold (of_out_packets_per_sec, of_out_bytes_per_sec, of_in_packets_per_sec, of_in_bytes_per_sec, tcp_of_out_packets_per_sec, tcp_of_out_bytes_per_sec, tcp_of_in_packets_per_sec, tcp_of_in_bytes_per_sec) are the main performance metrics, produced by this test scenario, showing the performance of controller to sustain regular communication with switches through time over large topologies.

Sample experimental results

Sample experimental results for this test are shown below

Intro

Stress Tests

Emulators

Monitoring tools

OpenFlow monitoring tools
- oftraf

Design

Releases

ODL stress tests performance reports

ODL stress test performance report v1.3

Sample Performance Results

Provide feedback

Saved searches

Use saved searches to filter your results more quickly