Bandwidth in ECN tutorial

[SanaaComp]

I'm trying to play around with the bandwidth between S1 and S2 in ECN. I changed the bandwidth to different values but according to iperf s1 s2 the bandwidth is about 20 Gbits/sec no matter of my value.

How can I change the bw?
Note: I'm new to P4.

[jafingerhut]

S1 and S2 are simple_switch processes? If so, achieving 20 gigabits/sec sounds impossibly high, and I would question how you reached that conclusion.

One possible way that might help to reduce the maximum rate is the command set_queue_rate in the simple_switch_CLI program, which can connect to at most one simple_switch or simple_switch_grpc process at a time (or at least that is the only way I have ever used it). Using the command help set_queue_rate shows the following help message:

Set rate of one / all egress queue(s): set_queue_rate <rate_pps> [<egress_port>]

I have never tested this technique myself to see whether it has the intended effect.

I am sure there are other ways.

[SanaaComp]

mininet> iperf s1 s2
*** Iperf: testing TCP bandwidth between s1 and s2 
*** Results: ['21.9 Gbits/sec', '21.9 Gbits/sec']

Using set_queue_rate did nothing.

[jafingerhut]

I tried the iperf command within mininet on my system and also see answers in units of Gbits/sec. I really sincerely doubt that these measurements are accurate. I do not know how the iperf within mininet works, but this throughput is so large, and the CPU utilization on my system so low, that it seems impossible that the reported throughput value could be correct. I would not believe it until someone provides extremely strong evidence that it is correct.

[jafingerhut]

Here is strong evidence that whatever the mininet iperf command is doing, it is not sending packets that get processed by the simple_switch processes that execute your P4 code:

In a separate terminal, look in the logs directory, which at least for the exercises/basic tutorial exercise I tried with, contains files like s1.log s2.log etc., one for each simple_switch process, with names that correspond to the names of the switches in the mininet topology. The contents of those files show detailed tracing information about how the P4 program processed every packet received by the switch that the file corresponds to.

When I run h1 ping h2, I see new trace lines added to the file s1 (again, my example is for the exercises/basic exercise -- your topology of hosts and switches may differ, but some file should be appended to when a switch processes packets).

When I run iperf s1 s2, none of those files are updated. So no packets are being processed by any of the simple_switch processes.

For the few seconds that iperf s1 s2 is running, I type this command in a separate terminal window:

ps axguwww | grep perf

the output shows these processes:

$ ps axguww | grep iperf
andy      6823  0.0  0.1  28428  4052 pts/11   S+   12:01   0:00 man iperf
root      7049 53.2  0.0 313884  3012 pts/7    Sl+  12:20   0:02 iperf -p 5001 -s
root      7056 86.7  0.0 166420  1948 pts/6    Sl+  12:20   0:03 iperf -p 5001 -t 5 -c 127.0.0.1
andy      7061  0.0  0.0  21532  1096 pts/10   S+   12:20   0:00 grep --color=auto iperf

The one with the command line iperf -p 5001 -t 5 -c 127.0.0.1 has the option -c 127.0.0.1. 127.0.0.1 is the IPv4 loopback address, meaning "the IPv4 address assigned to me, whatever the system is on which the address is resolved". I strongly suspect that these iperf commands are sending packets to each other not through the mininet emulated network, but directly through the host operating system's loopback interface, which is simply copying the packet from one process to another through the OS kernel.

[jafingerhut]

I think you should be using the mininet iperf command between hosts, not switches, e.g.

iperf h1 h2

When I try that on the exercises/basic tutorial, I see many packets being traced in the s1.log file for switch s1, which in that exercises's topology, is the only switch on the path from host h1 to h2.

When I did that, I saw performance results of about 19 Mbits/sec, which I can much more readily believe than anything in units of Gbits/sec.

[SanaaComp]

Thanks for clarifying.

[SanaaComp]

I re-opened this issue.
I created a simple network with three hosts and two switches. After running iperf, the performance results of about 700 Mbits/sec. But the bw in exercises/basic between h1 and h2 is about 10 Mbits/sec.

Is it because of the P4?

[jafingerhut]

Short answer: yes.

Long answer:

Yes, but the precise performance you see right now is not the fastest that the simple_switch process can do. It has multiple options it can be compiled with, and multiple choices for options on its command line when it is started, that can affect its performance significantly.

Several ways are described here: https://github.com/p4lang/behavioral-model/blob/master/docs/performance.md

Another way is to run simple_switch with neither the --log-console nor --log-file command line options, which is happening by default when doing make run in one of the tutorials exercises. Enabling those options is what produces detailed trace logs of processing packets, which can make debugging an incorrect P4 program much more straightforward, but also slows down packet processing.

[SanaaComp]

How can I run exercises/ecn without logs?

[jafingerhut]

How much Python do you know, or are willing to learn? It doesn't take very much Python knowledge, and willingness to dig in and add debug print statements, or whatever your favorite learning-how-an-existing-program-works techniques, to figure out where in the Python code included in the tutorials repository that it starts the simple_switch process. You can start by grep'ing all of the files for occurrences of simple_switch, then focus on the occurrences in Python source files. You would need to change it (or find out that perhaps there is already an existing option in the Python code) to start the simple_switch processes without the --log-file some-file-name-here option.

[jafingerhut]

Some more details on how you can go about this.

I know that when running make run in one of the tutorials exercises directories, it runs one simple_switch or simple_switch_grpc process for each switch in the network being emulated. So I go to the exercises/ecn directory, run make run, and after it starts up, switch to a different terminal window and type a command that shows all processes running on the system, with their full command line options. There are multiple such commands, but one is ps axguwww. That shows all processes, so I narrow it down to only the ones with simple_switch somewhere in the line of output using grep, e.g.:

ps axguwww | grep simple_switch

Here are 3 of the lines of output I see when I run that on my system:

root     32633  0.5  0.9 1419860 40028 pts/6   Sl+  11:32   0:03 simple_switch_grpc -i 1@s1-eth1 -i 2@s1-eth2 -i 3@s1-eth3 -i 4@s1-eth4 --pcap /home/andy/tutorials/master/exercises/ecn/pcaps --nanolog ipc:///tmp/bm-0-log.ipc --device-id 0 build/ecn.json --log-console --thrift-port 9090 -- --grpc-server-addr 0.0.0.0:50051
root     32650  0.7  0.9 1419860 40296 pts/7   Sl+  11:32   0:03 simple_switch_grpc -i 1@s2-eth1 -i 2@s2-eth2 -i 3@s2-eth3 -i 4@s2-eth4 --pcap /home/andy/tutorials/master/exercises/ecn/pcaps --nanolog ipc:///tmp/bm-1-log.ipc --device-id 1 build/ecn.json --log-console --thrift-port 9091 -- --grpc-server-addr 0.0.0.0:50052
root     32675  0.1  0.9 1350480 36816 pts/8   Sl+  11:32   0:00 simple_switch_grpc -i 1@s3-eth1 -i 2@s3-eth2 -i 3@s3-eth3 --pcap /home/andy/tutorials/master/exercises/ecn/pcaps --nanolog ipc:///tmp/bm-2-log.ipc --device-id 2 build/ecn.json --log-console --thrift-port 9092 -- --grpc-server-addr 0.0.0.0:50053

3 simple_switch_grpc processes, one for each of the 3 switches in the exercises/ecn network topology.

The --log-console option is one that causes simple_switch_grpc to print out detailed trace information while processing each packet. You want to no longer use that option.

Let us look at other command line options that might be slowing things down, while we are at it.

--pcap /home/andy/tutorials/master/exercises/ecn/pcaps causes simple_switch_grpc to write every packet received or sent on every port to a file. You want to no longer use those command line options, either.

I doubt you need the options --nanolog ipc:///tmp/bm-1-log.ipc except for extra debug kinds of tracing. You can try removing that, and if anything goes wrong, put them back.

Command line options you need to keep, and why:

-i 1@s3-eth1 -i 2@s3-eth2 -i 3@s3-eth3 are necessary for defining what ports exist on the software switch. You need those to be there, or your simple_switch_grpc switches won't have any ports to send and receive packets on.

--device-id 1 build/ecn.json. --device-id gives each of the switches a unique id, I think so that the P4Runtime API controller software can send commands to one of them vs. another. In any case, it should not be causing packets to be processed more slowly, that I know of, so keep it. build/ecn.json is the compiled version of your P4 program, output by the p4c compiler, read by simple_switch_grpc. You need that.

--thrift-port 9092 -- --grpc-server-addr 0.0.0.0:50053. --thrift-port 9092 tells the process what TCP port to listen on for incoming controller connections using the Thrift API. Keep that. Similarly for --grpc-server-addr, except that is for listening for incoming controller connections using the P4Runtime API. Keep that, too.

[jafingerhut]

Note that the bandwidth is limited to what is specified in the topology.json file, too. If you run a command like top while running your iperf tess, and see that the CPU usage of the simple_switch_grpc processes never goes above a few percent of one CPU core, as I see with the default contents of the topology.json file, the limiting factor is not simple_switch_grpc speed, but the link rate specified in the topology.json file.

[jnfoster]

You can add other arguments to Bmv2 on this line of the Makefile for the ecn exercise.