-
Notifications
You must be signed in to change notification settings - Fork 42
Forwarding performance evaluation
A network is as fast as the number of packets its routers can forward per second. This section describes how to evaluate Linux XIA's forwarding performance against Linux IP.
Project Net-eval provides three tools for the evaluation: pw, rk, pc.
- pw (Packet Writer) creates packets to random destinations sampling the destinations with a Zipf's distribution, and sends them out as fast as possible.
- rk (Router Keeper) populates a router's routing table, and, if needed, keeps randomly updating the routing table at a given rate.
- pc (Packet Counter) counts the number of packets sent out by a router.
Net-eval provides two important files:
- 'seeds' is a large collection of high-entropy, 32-bit numbers collected from /dev/random. pw and rk use this file to seed their distribution sampling.
- 'prefix' is a list of IPv4 CIDR obtained from Route Views on March 17th, 2013 at 2pm. pw and rk use this file to generate packets and populate/update the routing table.
The setup described on this page assumes that there is a unique link between the router and each of its ports. If this assumption is not true, for example, the router has only a network card that is connected to its "ports" through a hub/switch, one has to disable ICMP redirects in the kernel; see here for an example.
Given that source address of the packets pw generates is not in the routing table of the router, and, even if it were there, rk would randomly pick a forward link for it, one has to disable reverse-path filtering of the kernel on the router. A short description of this feature is provided here. On an Ubuntu box, this is accomplished by commenting the lines below in file '/etc/sysctl.d/10-network-security.conf':
# Turn on Source Address Verification in all interfaces to # prevent some spoofing attacks. #net.ipv4.conf.default.rp_filter=1 #net.ipv4.conf.all.rp_filter=1
Linux IP has packet forwarding disabled by default. More information about this is available here. In order to enable packet forwarding on an Ubuntu box, one has to uncomment the following line in file '/etc/sysctl.conf':
# Uncomment the next line to enable packet forwarding for IPv4 net.ipv4.ip_forward=1
The first program that has to run on the router is pc. It is important to run pc first because not only does pc take the measurements, but it also drops the packets after the router forwards them. If those packets were not dropped, they would go back to the ports and influence the measurements given that the ports would have to process them. Moreover, those packets back to the ports may generate messages on dmesg(1) for a reason or another, and that would make it harder to diagnose a problem.
The following command starts pc as a daemon that adds ebtables(8) rules to filter the packets destined to interfaces veth.test1, veth.test2, veth.test3, and veth.test4 as well as saves all measurements in file "exp-num-port/ip/4/1". The option --parents makes sure that if any folder listed in the file's name does not exist, pc will create it. The name of the sampling file should follow the format experiment/stack/column/run in order to allow the generation of graphics with the R scripts provided.
./pc --daemon --add-rules --parents --file=exp-num-port/ip/4/run1 \ veth.test1 veth.test2 veth.test3 veth.test4
The following is an example of how to call rk for IP.
./rk --run=1 br0 192.168.0.2 br1 192.168.1.2 br2 192.168.2.2 br3 192.168.3.2
The sequence of interfaces and gateways must match the ports.
Make sure that XIA is loaded.
modprobe xia modprobe xia_ppal_ad modprobe xia_ppal_hid
Other principals are not necessary for this evaluation. pc and rk are called as follows:
./pc --stack=xia --daemon --add-rules --parents --file=exp-num-port/xia/4/run1 \ veth.test1 veth.test2 veth.test3 veth.test4 ./rk --run=1 --stack=xia \ br0 75ad3b8cc86c9356224b83f18bdb4465aac0f6d8 \ br1 3b479ba966e9553c11d453002f87076edbcfe3a8 \ br2 a37eb6f7e5baeb18f9dd7da32aac10fb2de19c5d \ br3 0df1eb7defcfac2b98a25b47d42686c6ea9628ff
In XIA, the IDs after the interfaces represent HIDs. Interfaces are not, in fact, necessary for XIA since HIDs are not associated to interfaces, but hosts; When --stack=xia, the code only tests interfaces' existence. This behavior was maintained this way to simplify reusing RK for IP and XIA, but it may change in the future.
The following are examples of how to call pw on IP ports:
./pw --pkt-len=64 --ifname=eth0 --dmac=46:ab:eb:43:b5:a1 --zipf=1.0 --nnodes=5 --run=1 --node-id=1 ./pw --pkt-len=64 --ifname=eth0 --dmac=02:7c:b3:2d:ff:97 --zipf=1.0 --nnodes=5 --run=1 --node-id=2 ./pw --pkt-len=64 --ifname=eth0 --dmac=9e:50:3c:49:07:23 --zipf=1.0 --nnodes=5 --run=1 --node-id=3 ./pw --pkt-len=64 --ifname=eth0 --dmac=aa:81:ec:ae:4b:8c --zipf=1.0 --nnodes=5 --run=1 --node-id=4
Notice that each command is meant to be called on a single port. The option --dmac must be the MAC address of the interface of the router that that port sees.
Below are the same commands for calling pw on ports running XIA:
./pw --stack=xia --daddr-type=fb3 --pkt-len=256 --ifname=eth0 --dmac=46:ab:eb:43:b5:a1 \ --zipf=1.0 --nnodes=5 --run=1 --node-id=1 ./pw --stack=xia --daddr-type=fb3 --pkt-len=256 --ifname=eth0 --dmac=02:7c:b3:2d:ff:97 \ --zipf=1.0 --nnodes=5 --run=1 --node-id=2 ./pw --stack=xia --daddr-type=fb3 --pkt-len=256 --ifname=eth0 --dmac=9e:50:3c:49:07:23 \ --zipf=1.0 --nnodes=5 --run=1 --node-id=3 ./pw --stack=xia --daddr-type=fb3 --pkt-len=256 --ifname=eth0 --dmac=aa:81:ec:ae:4b:8c \ --zipf=1.0 --nnodes=5 --run=1 --node-id=4
Using the series of commands in the previous sections, one can manually evaluate the forwarding performances of the IP and XIA network stacks. However, to set-up and run experiments on a larger scale, one can also use a series of Ruby scripts available here: Net-eval.
To obtain and run these scripts, execute the following commands:
git clone https://github.com/cjdoucette/net-eval.git cd net-eval/exp
Before the scripts can be used, the basic LXC package, Ruby 1.9.1, and tools for network bridges must be obtained:
# apt-get install lxc ruby1.9.1 bridge-utils
Additonally, the Ruby library ("gem") netaddr must be installed to run IP experiments:
# gem install netaddr
The first step in executing multiple experiments is to set up some temporary files that hold all of the information required to run the tests. To do this, net-eval-setup should be used. When run with the "--help" option, one can see all of the options possible for net-eval-setup:
Usage: ruby1.9.1 ./net-eval-setup.rb NUM_TRIALS EXP_TIME [OPTIONS]
-e, --enable Enable boot script testing
-d, --disable Disable boot script testing and exit
-a, --addr Test changing address type
-p, --ports Test changing number of ports
-s, --size Test changing packet size
-u, --updates Test changing routing table update rate
-z, --zipf Test changing the zipf distribution
-i, --ip Perform experiments on IP stack
-x, --xia Perform experiments on XIA stack
Where "NUM_TRIALS" is the number of trials that the user wants to run for each experiment and where EXP_TIME is the length of each run of each experiment, measured in seconds. The available options are explained below.
The first options in the list have to do with enabling the experiments to be run as a boot script. If "--enable" is specified when net-eval-setup is run, then a runlevel 2 boot script is installed which invokes the experiments upon reboot. Since each experiment ends with an automatic reboot, if the boot script is enabled the user can set-up a series of experiments and let the machine run through them without any user interaction.
If "--disable" is specified, then the boot script is removed (if it is present) and the script immediately exits.
One can run one or more types of experiments which test various parameters using the following options:
-a, --addr Test changing address type
-p, --ports Test changing number of ports
-s, --size Test changing packet size
-u, --updates Test changing routing table update rate
-z, --zipf Test changing the zipf distribution
Inside net-eval-setup there are default ranges for these tests; for example, the "--ports" option will create a series of tests using any number of ports from 1 through 16. These ranges can be changed manually in net-eval-setup or in the manner described in the miscellaneous section.
When experimenting with the packet size, if a packet size greater than the MTUs of the virtual devices (typically 1500B) is chosen, the script will automatically change the MTUs to be the size of the packets in the test.
To set-up an experiment, net-eval-setup needs to know which stack to run the experiments on. To run the experiments on an IP stack, use "--ip". To run the experiments on an XIA stack, use "--xia". To set-up the experiments to run on both stacks, specify both options.
If the user wants some experiments to be run on different stacks, the experiments can be run separately or in the manner described in the miscellaneous section.
To run all experiments on both stacks, while enabling boot script testing and changing the IP parameters to the appropriate values, execute the following command:
sudo ruby1.9.1 ./net-eval-setup.rb 10 60 --xia --ip --enable \ --addr --ports --size --updates --zipf
Or, equivalently:
sudo ruby1.9.1 ./net-eval-setup.rb 10 60 -xieapsuz
These will set up experiments that each have 10 trials, where each trial will run for 60 seconds.
Once the experiments have been set-up by net-eval-setup, a few temporary files will be created in the tmp directory which keep track of the progress of the experiments. Then, net-eval-run can be used to actually execute those experiments. When run with the "--help" option, one can see all of the options possible for net-eval-run:
Usage: ruby1.9.1 ./net-eval-run.rb [--start=START] [--end=END]
-s, --start=START Begin at test START
-e, --end=END End at test END
The experiments that were set-up with net-eval-setup are placed into tmp/net-eval-tests. Each experiment is given an integer identifier; that identifier can be passed to net-eval-run to indicate which experiments should be run.
When net-eval-run is invoked, it will create the containers necessary to run the experiments and then will perform the tests. If a boot script is used, upon reboot net-eval-run will be invoked by that script.
It is normal to see error messages such as init.lxc: initutils.c: mount_fs: 36 failed to mount /dev/shm : No such file or directory or sudo: unable to resolve host.
To first execute the experiment 6 in tmp/net-eval-tests and to end at experiment 13, use:
sudo ruby1.9.1 ./net-eval-run.rb --start=6 --end=13
To run all tests in tmp/net-eval-tests, use:
sudo ruby1.9.1 ./net-eval-run.rb
To customize the experiments, users can add, delete, or edit the data in both tmp/net-eval-tests and tmp/net-eval-status.
Each experiment in tmp/net-eval-tests is of the form:
<router's pc command> <router's rk command> <stack> <destination address type> <updates> <packet size> <zipf parameter> <number of ports> <trial> <name of experiment>
And the data in tmp/net-eval-status is of the form:
<next experiment to be executed> <last experiment to be executed> <identifier of last experiment in tmp/net-eval-tests> <path to the experiment scripts> <length of each experiment (in seconds)>
These files can be edited to fit a user's custom needs.
To create the packet generators on the ports for these experiments, the net-eval-run scripts sets up Linux containers (lxc) using the xlxc-create Ruby script as a wrapper for the lxc-create script. To destroy these containers, net-eval-run uses the xlxc-destroy Ruby script as a wrapper for the lxc-destroy script.
The experiments are run using the lxc-execute command to invoke pw in the containers.
All grants that have generously supported the development of Linux XIA are listed on our Funding page.