post some benchmarks

[rade]

We should post some performance benchmarks, comparing communication over the weave network with alternative ways containers on multiple hosts can be made to talk to each other, e.g. over exposed ports that have been made accessible by the respective firewalls.

[adieu]

I did some performance tests with two 512M digitalocean droplets in the same data center. The test tool is qperf as suggested by this rudder post.
Here is the result.

Test 1:

Use weave expose 10.0.1.1/24 on host1 and weave expose 10.0.1.2/24 on host2. Then test host to host speed.

Direct connection without weave:

tcp_bw:
    bw  =  117 MB/sec
udp_lat:
    latency  =  104 us

With weave:

tcp_bw:
    bw  =  6.55 KB/sec
udp_lat:
    latency  =  394 us

Test2:

Use weave expose 10.0.1.1/24 on host1 and weave run 10.0.1.100/24 on host2. Then test container to host speed.

tcp_bw:
    bw  =  32.8 KB/sec
udp_lat:
    latency  =  420 us

Test3:

Use weave expose 10.0.1.1/24 on host1, docker run ubuntu on host1 and weave run 10.0.1.100/24 ubuntu on host2. Then test container to container speed through host routing.

tcp_bw:
    bw  =  190 KB/sec
udp_lat:
    latency  =  395 us

Test4:

Use weave run 10.0.1.101/24 ubuntu on host1 and weave run 10.0.1.100/24 ubuntu on host2. Then test container to container speed without host routing.

tcp_bw:
    bw  =  17 MB/sec
udp_lat:
    latency  =  412 us

Test5:

The same setup with test4 but use -passwd in weave launch to enable encryption.

tcp_bw:
    bw  =  15.3 MB/sec
udp_lat:
    latency  =  479 us

From the tests result we could see we could only get a usable speed from two weave containers. We should avoid to weave expose for now.

I didn't dig further for the poor performance with weave expose. Maybe there're some tweaks to boost the speed. I'll leave this to core weave developers.

Thank you for make this wonderful software.

[rade]

So the performance issue is fixed, I think. @adieu, would be good if you were able to confirm this. Please grab the latest 'weave' script. The new code kicks in when the 'weave' bridge gets created, so if you already have that bridge, e.g. from a previous 'weave launch' then blow it away with

ip link del dev weave

first before launching weave. Also, you really do need ethtool for the fix to work, so if you haven't got that installed yet then please do so first.

[adieu]

I can confirm the performance issue have gone away with the latest commit. Thanks.

[inercia]

Does qperf change things like the source and destination ports? If it doesn't, Weave will be in a very favorable case (ie, all the hash table lookups will be in cache): maybe it could be a good idea to have some benchmarks by doing some port scans with a tool like nmap...

[rade]

Does qperf change things like the source and destination ports?

Don't think so.

it could be a good idea to have some benchmarks by doing some port scans with a tool like nmap.

Perhaps, but I see that very much as a follow-on task. The objective here should be the post the simplest possible benchmarks that useful. We can always improve them / add more later.

[inercia]

Yes, of course, this should be the first step in benchmarking Weave. A more formal approach could be developed in the future... ;)

[rade]

Hilarious that there's an IETF RFC for benchmarking :)

[tristanz]

Are there any new qperf numbers? It would be great to post basic qperf numbers like flannel.

[neurogenesis]

http://www.generictestdomain.net/docker/weave/networking/stupidity/2015/04/05/weave-is-kinda-slow/

[bboreham]

http://blog.weave.works/2015/06/12/weave-fast-datapath/

[neurogenesis]

@bboreham, wow. great to see the performance improvements!

[bboreham]

Closing with http://blog.weave.works/2015/11/13/weave-docker-networking-performance-fast-data-path/