Network performance deterioration with every next kernel upgrade

[romanshein]

Just my 2cc:

I've ran a series of tests to measure network upload performance between 2 VMs

1. Linux guest pushing: dd if=/dev/zero bs=1024K count=5120 | nc -v 192.168.1.2 2000
   to
2. Windows guest (192.168.1.2) with netcat listening at 2000

Both VMs are sitting on the same host connected over vSwitch.
All linux guests were created equal (static RAM - 1024MB, 2 vCPU, with GUI)

I've noticed that the upload speed is deteriorating significantly with every newer kernel.
Older kernel results:
CentOS 6.5 (kernel 2.6) - 100 MB/s
Debian 7.3 (kernel 3.2) - 190 MB/s

Newer kernels demonstrate worse performance:
Fedora 20 (kernel 3.11) - 65 MB/s
Ubuntu 14 (w kernel 3.13) - 40 MB/s
Debian 7.3 (upgraded to kernel 3.13) - 90 MB/s

[kattisrinivasan]

I don’t think we have run this test. Given the focus we have had on performance, I am curious to see where we regressed. Both RHEL7 and sles 11 sp3 have more or less latest upstream code. Vivek could you have somebody run these tests on RHEL or sles 11 sp3 guests. If you are testing with upstream bits, why is LIS 3.5 in the subject?

From: romanshein [mailto:notifications@github.com]
Sent: Thursday, February 06, 2014 3:08 AM
To: LIS/LIS3.5
Subject: [LIS3.5] Network performance deterioration with every next kernel upgrade (#5)

Just my 2cc:

I've ran a series of tests to measure network upload performance between 2 VMs

1. Linux guest pushing: dd if=/dev/zero bs=1024K count=5120 | nc -v 192.168.1.2 2000
   to
2. Windows guest (192.168.1.2) with netcat listening at 2000

Both VMs are sitting on the same host connected over vSwitch.
All linux guests were created equal (static RAM - 1024MB, 2 vCPU, with GUI)

I've noticed that the upload speed is deteriorating significantly with every newer kernel.
Older kernel results:
CentOS 6.5 (kernel 2.6) - 100 MB/s
Debian 7.3 (kernel 3.2) - 190 MB/s

Newer kernels demonstrate worse performance:
Fedora 20 (kernel 3.11) - 65 MB/s
Ubuntu 14 (w kernel 3.13) - 40 MB/s
Debian 7.3 (upgraded to kernel 3.13) - 90 MB/s

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5.

[abgupta]

Hi Roman,

Sorry for the delay on this. We did some testing and it appears that the issue is with running netcat on Windows. If you run netcat on bare metal Windows then you are likely to see much lower throughput vs Linux.

We belief that iperf is the right tool for this scenario and are happy to debug any iperf issues that you may see. Please let me know your thoughts and experiences on this.

Thanks,
Abhishek

[romanshein]

Hi Abhishek

Thanx for the feedback.
I agree that window's netcat may skew results, but it should skew them evenly. It doesn't explain great variation between distros.

BR, Roman.

[abgupta]

Hi Roman, Not sure what the root cause may be. Is it possible for you to try using iperf and see if the skew is there in that case? If not then may be there is some variation in the Linux network stack that is causing the interop with netcat to break. Please try once with iperf and let us know what you see.