seastar tcpserver - goclient txtx test on DPDK is underperfoming x2

[slivne]

http://jenkins.cloudius-systems.com:8080/job/seastar-private-tcpserver-perf/8/PerfPublisher/

increasing the size of user_queue_space x2 does not help (user_queue_space = {212992 * 2}) - the associated results are from a build using asias/txtx branch in seastar-dev that has this change.

On Thor-Loki:

Goserver	tcp-server stack-native-dpdk171	tcpserver stack-native-dpdk18	tcpserver stack-posix
8.915581406	17.087416845	17.867953145	8.914382154

values are seconds to complete the tests - lower is better

To reproduce you can tun the tests in the following manner:

• install + configure dpdk
• start tcpserver
• clone tests.git
• cd tests
• scripts/tester.py run --config_param sut.ip: --config_param tester.ip:x apps/tcp/tests/test_1_goclient/tester
• check the output at apps/tcp/tests/test_1_goclient/tester/out

or

• install + configure dpdk
• start tcpserver
• clone tests.git
• cd tests/apps/tcp/tests/test_1_goclient/tester
• go run client.go -h

Please note you can use client.go -conn 1 option to limit the number of connections

[asias]

I bisected using vhost backend. And found "72324f0 net: implement bulk sending interface for virtio " is the first bad commit. With this commit, the txtx test can not complete (the tcp_server sending 100MiB data to client in each connection and there are 100 concurrent connections).

$ go run client.go -host 192.168.10.101 -conn 100 -test txtx

Never complete.

I also tested the commit just before the series "[PATCHv2 00/10] network rx path inversion":

3f483b2 future: add missing std::forward() in do_until_continued()  

I see no issue with vhost backend with tcp_server test. This suggests the tcp layer looks ok. We have a issue introduced in the "[PATCHv2 00/10] network rx path inversion"

Result for  commit 3f483b2:

vhost  tso off 1vcpu
[asias@thor goclient]$ go run client.go -host 192.168.10.101 -conn 100 -test all
========== ping ============
Server:  192.168.10.101:10000
Connections:  100
Total PingPong:  1000000
Total Time(Secs):  2.671862031
Requests/Sec:  374270.8225191288
========== txtx ============
Server:  192.168.10.101:10000
Connections:  100
Bytes Received(MiB):  10000
Total Time(Secs):  23.143342741
Bandwidth(Gbits/Sec):  3.624631106179408
========== rxrx ============
Server:  192.168.10.101:10000
Connections:  100
Bytes Sent(MiB):  10000
Total Time(Secs):  16.169089728
Bandwidth(Gbits/Sec):  5.188052105044266


vhost tso on 1vcpu
[asias@thor goclient]$ go run client.go -host 192.168.10.101 -conn 100 -test all
========== ping ============
Server:  192.168.10.101:10000
Connections:  100
Total PingPong:  1000000
Total Time(Secs):  2.491742781
Requests/Sec:  401325.5331269283
========== txtx ============
Server:  192.168.10.101:10000
Connections:  100
Bytes Received(MiB):  10000
Total Time(Secs):  8.875455087
Bandwidth(Gbits/Sec):  9.451468029269742
========== rxrx ============
Server:  192.168.10.101:10000
Connections:  100
Bytes Sent(MiB):  10000
Total Time(Secs):  8.836905488
Bandwidth(Gbits/Sec):  9.492698559910185

--->

$ git bisect good d72de7e
$ git bisect bad e9e1ed7
$
72324f0 is the first bad commit
commit 72324f0
Author: Gleb Natapov gleb@cloudius-systems.com
Date: Mon Jan 5 11:09:04 2015 +0200

net: implement bulk sending interface for virtio

:040000 040000 225321091be21cf3af6a071e7df4c366983922c9 b94ef6caad215f9e36118c2395bc80e8fe87534a M net

[asias]

Added more test with the good commit (3f483b2 future: add missing std::forward() in do_until_continued()) using dpdk.

txtx test with dpdk is not completely broken but it is much slower than virtio (4Gbps vs 9Gbps)! rxrx test with dpdk is broken. ping test with dpdk is ok.

3f483b2 future: add missing std::forward() in do_until_continued() 

Result for commit 3f483b2:
dpdk 1cpu 


[asias@thor goclient]$ go run client.go -host 192.168.10.101 -conn 100 -test ping
========== ping ============
Server:  192.168.10.101:10000
Connections:  100
Total PingPong:  1000000
Total Time(Secs):  2.435628829
Requests/Sec:  410571.58960077324

[asias@thor goclient]$ go run client.go -host 192.168.10.101 -conn 100 -test txtx
========== txtx ============
Server:  192.168.10.101:10000
Connections:  100
Bytes Received(MiB):  10000
Total Time(Secs):  19.584052577
Bandwidth(Gbits/Sec):  4.283387193236905


[asias@thor goclient]$ go run client.go -host 192.168.10.101 -conn 100 -test rxrx
========== rxrx ============
Server:  192.168.10.101:10000
Connections:  100
Never completes!!!

[slivne]

@asias the latest results from the nightly perf tests indicate that we have closed the gap
can you have a look

http://jenkins.cloudius-systems.com:8080/job/seastar-tcpserver-perf/38/

if all is ok can we close this issue ?

[asias]

@slivne let's close this issue. TSO support (1934160) in dpdk is supposed to close the x2 gap.