Bidirect mode implementation

[LikHait]

Aquantia actively uses iperf3 for our AQC 10G products verification and testing.
Must have testing in our case is bidirect traffic runs. We suffer from iperf3 missing this feature builtin.

Here is our proposal on implementation of --bidir (bidirectional) option for iperf.

We've tried to minimize the impact and reuse existing iperf3 apis an maximum.

Short overview of the solution:

• iperf_test: int sender is replaced by enum iperf_mode mode with 3 states:

enum iperf_mode {
	SENDER = 1,
	RECEIVER = 0,
	BIDIRECTIONAL = -1
};

• iperf_stream: add new flag "sender"

• Now streams are created depending on mode:

  • stream.sender == 1 if test->mode is SENDER
  • stream.sender == 0 if test->mode is RECEIVER
  • if test->mode is BIDIRECTIONAL we double number of streams, half has sender == 1 and half sender == 0. In future it'll be possible to create asymmetric bidir flows.

• iperf_new_stream (struct iperf_test * test, int s) changed to
  iperf_new_stream (struct iperf_test * test, int s, int sender)

• Summary information output is changed for each type of iperf_stream (senders and receivers)

---

Examples

iperf3 -c localhost --bidir -P 4

Interval output:

[  5][TX-C]   9.00-10.00  sec   584 MBytes  4.90 Gbits/sec    0   1.87 MBytes       
[  7][TX-C]   9.00-10.00  sec   584 MBytes  4.90 Gbits/sec    0   1.87 MBytes       
[  9][TX-C]   9.00-10.00  sec   584 MBytes  4.90 Gbits/sec    0   1.87 MBytes       
[ 11][TX-C]   9.00-10.00  sec   582 MBytes  4.89 Gbits/sec    0   2.12 MBytes       
[SUM][TX-C]   9.00-10.00  sec  2.28 GBytes  19.6 Gbits/sec    0             
[ 13][RX-C]   9.00-10.00  sec   453 MBytes  3.80 Gbits/sec                  
[ 15][RX-C]   9.00-10.00  sec   453 MBytes  3.80 Gbits/sec                  
[ 17][RX-C]   9.00-10.00  sec   453 MBytes  3.80 Gbits/sec                  
[ 19][RX-C]   9.00-10.00  sec   453 MBytes  3.80 Gbits/sec                  
[SUM][RX-C]   9.00-10.00  sec  1.77 GBytes  15.2 Gbits/sec     

Sum output:

[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-10.00  sec  5.83 GBytes  5.01 Gbits/sec    0             sender
[  5][TX-C]   0.00-10.02  sec  5.83 GBytes  5.00 Gbits/sec                  receiver
[  7][TX-C]   0.00-10.00  sec  5.83 GBytes  5.01 Gbits/sec    0             sender
[  7][TX-C]   0.00-10.02  sec  5.83 GBytes  5.00 Gbits/sec                  receiver
[  9][TX-C]   0.00-10.00  sec  5.83 GBytes  5.01 Gbits/sec    0             sender
[  9][TX-C]   0.00-10.02  sec  5.83 GBytes  5.00 Gbits/sec                  receiver
[ 11][TX-C]   0.00-10.00  sec  5.83 GBytes  5.01 Gbits/sec    0             sender
[ 11][TX-C]   0.00-10.02  sec  5.83 GBytes  4.99 Gbits/sec                  receiver
[SUM][TX-C]   0.00-10.00  sec  23.3 GBytes  20.0 Gbits/sec    0             sender
[SUM][TX-C]   0.00-10.02  sec  23.3 GBytes  20.0 Gbits/sec                  receiver
[ 13][RX-C]   0.00-10.00  sec  4.62 GBytes  3.96 Gbits/sec    1             sender
[ 13][RX-C]   0.00-10.02  sec  4.61 GBytes  3.95 Gbits/sec                  receiver
[ 15][RX-C]   0.00-10.00  sec  4.62 GBytes  3.96 Gbits/sec    1             sender
[ 15][RX-C]   0.00-10.02  sec  4.61 GBytes  3.95 Gbits/sec                  receiver
[ 17][RX-C]   0.00-10.00  sec  4.62 GBytes  3.96 Gbits/sec    1             sender
[ 17][RX-C]   0.00-10.02  sec  4.61 GBytes  3.95 Gbits/sec                  receiver
[ 19][RX-C]   0.00-10.00  sec  4.62 GBytes  3.96 Gbits/sec    1             sender
[ 19][RX-C]   0.00-10.02  sec  4.61 GBytes  3.95 Gbits/sec                  receiver
[SUM][RX-C]   0.00-10.00  sec  18.5 GBytes  15.9 Gbits/sec    4             sender
[SUM][RX-C]   0.00-10.02  sec  18.4 GBytes  15.8 Gbits/sec                  receiver

---

Comparison with legacy runs

We did a verification on our Aquantia AQC107 10G NICs.
We compared legacy usecase (using two servers for bidirect) with --bidir.

Two servers run:

iperf3 -c server -R -P 4 -t 100

[ ID] Interval           Transfer     Bitrate         Retr
[SUM]   0.00-100.03 sec   109 GBytes  9.38 Gbits/sec    0             sender
[SUM]   0.00-100.00 sec   109 GBytes  9.38 Gbits/sec                  receiver

iperf3 -c 169.254.177.210 -p 5202 -P 4 -t 100

[ ID] Interval           Transfer     Bitrate         Retr
[SUM]   0.00-100.00 sec   109 GBytes  9.39 Gbits/sec    0             sender
[SUM]   0.00-100.03 sec   109 GBytes  9.39 Gbits/sec                  receiver

New, bidir run

iperf3 -c 169.254.177.210 --bidir -P 4 -t 100

[ ID][Role] Interval           Transfer     Bitrate         Retr
[SUM][TX-C]   0.00-100.00 sec   109 GBytes  9.37 Gbits/sec    0             sender
[SUM][TX-C]   0.00-100.03 sec   109 GBytes  9.37 Gbits/sec                  receiver
[SUM][RX-C]   0.00-100.00 sec   109 GBytes  9.39 Gbits/sec    1             sender
[SUM][RX-C]   0.00-100.03 sec   109 GBytes  9.39 Gbits/sec                  receiver

We've added an extra column in output, Role: [TX-C]. This helps to distinguish results data.
Same data is available in updated json output.
CC @cail

[cail]

It seems to work in the very limited test cases

Let us know the failed usecases. We've tried to design this as much user-friendly as possible. Ultimately, user should be able to do --bidir on any of his/her existing configs.

[bmah888]

It seems to work in the very limited test cases

Let us know the failed usecases. We've tried to design this as much user-friendly as possible.
Ultimately, user should be able to do --bidir on any of his/her existing configs.

You might have missed the rest of sentence...I wrote "It seems to work in the very limited test cases I have tried so far" (emphasis mine). I meant that I had tested only a few cases and it worked in all of those.

[laoshaw]

Hope this can be added soon, bidir testing is essential for so many use cases.

[donotevercry]

What's the latest status? Is the merge conflict resolved? We'd like to help test it out.

[bmah888]

I think you should be able to test it out now if you check out the Aquantia/iperf respository.

@LikHait : Do you think you can try to resolve the merge conflicts in src/iperf_api.c? I think if we're going to get any significant amount of testing on this, I probably just need to merge this PR to our mainline (master) and hope more people try it out. Thanks.

If it takes a little time for this change to settle down, this might result in me doing some 3.6.x releases off a new release branch created from the 3.6 tag. I guess I'm OK with that.

[LikHait]

I will try to do it in two weeks.

[LikHait]

@bmah888: Could you help with t_timer? Why is it fail?
I have pass on local mashine:

Making check in src
make[1]: Entering directory '/home/ej/git/iperf/src'
make check-TESTS
make[2]: Entering directory '/home/ej/git/iperf/src'
make[3]: Entering directory '/home/ej/git/iperf/src'
PASS: t_timer
PASS: t_units
PASS: t_uuid
PASS: t_api
============================================================================
Testsuite summary for iperf 3.6+
============================================================================
TOTAL: 4
PASS: 4
SKIP: 0
XFAIL: 0
FAIL: 0
XPASS: 0
ERROR: 0
============================================================================
make[3]: Leaving directory '/home/ej/git/iperf/src'

[bmah888]

@LikHait : Thanks for fixing the merge conflict. I'll look at why tests are failing, might be a little later this week (I hope).

[bmah888]

I'm not able to reproduce that test failure either. I've tried on a CentOS 7 VM with both GCC (whatever CentOS 7 comes with) and with clang-5.0 (what Travis CI was using when it failed). t_timer just worked for me. I don't understand this. I'm leaning towards just merging this PR.

[bmah888]

Merged. Let's see how well this fares on the mainline! Thanks for submitting an innovative implementation of this feature.

[zzzuzik]

@LikHait @bmah888 thank you this

[blkeller]

@LikHait @bmah888 This new feature works great! I just used it to test a new fiber optic cable plant with two machines pushing line rate through a 40Gbps link in both directions at the same time. I never was able to get iperf2 to go full speed bidirectionally -- this new version of iperf3 was the only tool I could get working at line rate in both directions simultaneously to validate the link. Thank you!

[notthetup]

Thanks for the great work @LikHait and @bmah888.

I'm seeing this while trying out the new --bidir flag. Any ideas?

$ iperf3 -c 192.168.1.215 --bidir
Connecting to host 192.168.1.215, port 5201
[  5] local 192.168.1.42 port 38555 connected to 192.168.1.215 port 5201
iperf3: error - unable to initialize stream: Transport endpoint is not connected

[O-ring]

Hello everyone,

would be useful to add to -Y flag to get the same behavior as --bidir

Thanks

[cail]

I'm seeing this while trying out the new --bidir flag. Any ideas?
[ 5] local 192.168.1.42 port 38555 connected to 192.168.1.215 port 5201
iperf3: error - unable to initialize stream: Transport endpoint is not connected

Do you use updated iperf build on both sides?
Try also checking firewall or disable it.

[Ketanbankar]

unable to create new stream : Permission denied on my phone
can some some one help on this issue ?
using iPref3.6 library on my android phone.
Magic Ipref works successfully but my application gives permission denied error: unable to create new Stream.
Help!!!!!

[LikHait]

unable to create new stream : Permission denied on my phone
can some some one help on this issue ?
using iPref3.6 library on my android phone.
Magic Ipref works successfully but my application gives permission denied error: unable to create new Stream.
Help!!!!!

iperf3.6 doesn't support bidir feature. It's merged to master later

[itchenfei]

Thanks for submitting this feature