Incorrect and impossible upload result using TCP #1069
Comments
|
@olekstomek, it seem as if TCP in your machines have unlimited (or huge) window size. From the information you sent it is clear that the transferred packets are buffered in the internal machine memory, so if there is no sending limitation the transfer rate is the rate of adding the data to memory. I don't know I this may happen, as maximum window size on windows seem to be 1GB. The following may help to ckeck the configuration of the window-size: Description of Windows TCP features. |
|
@davidBar-On Good and interesting point of view. I checked registry on my OS. There are no option for TcpWindowSize or something similar. Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces
Current global TCP settings in my OS. Speedtest.net CLI works fine on the same machine: |
|
@olekstomek, I don't really have any clue about what happens. Few things to consider/try:
|
Yes, but it was the newest version from here. I checked iPerf 3.1.3 on my Lenovo computer G50-80 with 12GB RAM, HDD and Windows 10 Home and it's OK: Additionally, I tried version 3.9 from this post (Ok, I'm a bit lazy and I didn't compile the version on my computer from source but I believe this version is fine)
With Default:
iPerf 2.0.9 (6 jun 2016 - 1.7 MiB for Windows Vista 64bits to Windows 10 64bits) When I check the information in the console on the laptop where the server is running, these are much smaller and real transfers at the level of several dozen Mbps (I don't include, so as not to impede the readability of the results from the client laptop). So you can see that the problem still exist. |
olekstomek
changed the title
|
@olekstomek, until we will understand the root cause of the problem it will not be possible to determine whether the problem is in the application. In any case you provided good data so I hope it will be possible to determine the root cause. Some observations from the data you sent:
Based on this, my current guess is that the problem is related to the network buffers which are the input to the TCP stack. It may be that somehow the network buffers size was set to a very big size - beyond the memory available, although I don't know how this can happen. Another option is that some kind of NIC Offload processing is defined, but that should not use the internal memory. In any case, following are some things that can be done to further evaluate the issue:
|
Yes you are right. I made such a thesis because, for example, the speedtest.net CLI works properly or other speed tests on web pages eg.: this or this (maybe the web cannot be compared with an application running on an OS without a browser).
Aside from the main thread, I thought the operating system would dump data from RAM to SSD, but I'm guessing 2GB/s data packets are big and it happens too fast. Going back to the main thread, yes, that's why I limited the test time with
As I checked before speedtest.net CLI works OK, I checked that they use TCP for the tests also.
This word "mainly" probably doesn't mean always but I haven't found the option to set the TCP/UDP options in the CLI and the GUI of the application. But I checked by Wireshark and protokol is TCP.
but not always - it's mobile connection in LTE:
Ethernet adapter: WiFi adapter:
On Ethernet adapter: As for the other settings, I've looked at them but it's hard for me to see their effect on the network performance. I came up with the idea that I would compare these settings with my Lenovo laptop on which the results are correct but on Windows Home I do not have the "Advanced" tab, so I will not check it. On WiFi adapter I don't have these options:
I did this option and re-tested on the local network but it is unchanged. As I mentioned in the first post, the problem occurs on two computers (one was a client and the other was a server, and then vice versa, and ~5Gbps vs ~18Gbps transfers may result from RAM - DDR3 vs DDR4). |
The explanation about how Speedtest works over TCP say that
As I don't see any problem with the different parameters settings, one more test that may help is to test the effect of changing the Transmit Buffers size. If the problem is related to the network buffers size, that will affect the test. Can you change the Transmit Buffers size to a small number, e.g. 10, and test the effect? I don't know what is "too small" so if nothing works with 10 try other value, e.g. 50. (It could also help trying some different values, but as I think reboot is required after the change, that may take too much time.) |
I haven't read it before. Ok, so the algorithm is different and we cannot compare the performance and behavior of these applications on the same operating system.
Additionally on one of my computer with Windows Enterprise I did network reset.
I will do it but I have to create a local network by cable first (because Transmit Buffers size is not in the WiFi adapter but is in Ethernet). I tried to do a test to an external server but without success. |
|
@olekstomek, I still don't have a clue what may the the problem ...
Please also try the test before changing the Transmit Buffers size, in case the issue is related to the WiFi adapter. (By the way, did you make sure the latest WiFi driver is installed?) |
Unfortunatelly nothing new using ethernet: The valid range of transmit buffers is from 80 to 2048 in increments of 8. Transmit buffers set to 80: Transmit buffers set to 1024: Transmit buffers set to 2048:
On one of the computers with the problem, I am using It is very difficult to guess what could be causing these false results. |
Just to make sure, did you check the log at the server side to see what is the actual throughput? I would expect that it will be higher over Ethernet.
I agree ... Following are some more actions that can be taken:
If nothing of this will help, then I think the next step (if you will be willing to spend the effort) is to write small client/server programs to see whether the way iperf3 is working with the socket is problematic on these computers (e.g. the use of |
Yes, I observed on server side actual throughput and it was about 100Mbps (LAN in my router which I used for tests has ports 100Mbps).
All tests here I did on ethernet. Usig UDP: I used also: and memory is the same (that's why I changed the name of the topic and made it clear that it was about TCP).
We can check, but I need more details. |
As you are using the 100Mbps interface, the |
Take a look: |
|
@olekstomek, I don't have suggestions for further testing, so the next step may be to see if changing the way iperf3 send data solve the issue. If you know how to build iperf3 under windows this is great. Otherwise we will have to find small server/client code that can be used for that. What I think should be checked first is if there is a problem in the way iperf3 sends data. For TCP this is done in iperf_tcp_send(): r = Nwrite(sp->socket, sp->buffer, sp->settings->blksize, Ptcp);At least under Linux, |
I didn't know but I did it (btw, I think issues #1067 and #1073 can be closed, the documentation for me was sufficient and simple - the configuration of the environment was the most difficult one, but this is a separate problem, the instruction tells how to compile the code on a ready environment). I downloaded Version of iperf3 on server: I changed this line in Line 91 in bd14377
It compiled. I run and: I changed this line in Line 91 in bd14377
and I got: I changed this line in Line 91 in bd14377
and I got: After every changing I did |
This should be very helpful!. As using
The debug code to add after the result = select(test->max_fd + 1, &read_set, &write_set, NULL, timeout);
if (test->debug) {
iperf_printf(test, "SELECT RESULT=%d, CNTL-FD=%d, READ-FDs-SET=", result, test->ctrl_sck);
if (result > 0) {
int i;
for (i = 0; i < test->max_fd + 1; i++) {
if (FD_ISSET(i, &read_set))
iperf_printf(test, "%d;", i);
}
}
iperf_printf(test, ", WRITE-FDs-SET=");
if (result > 0) {
int i;
for (i = 0; i < test->max_fd + 1; i++) {
if (FD_ISSET(i, &write_set))
iperf_printf(test, "%d;", i);
}
}
iperf_printf(test, "\n");
}The output should be something like: |
|
@davidBar-On
and more output with SELECT_RESULT:
and more output with SELECT_RESULT: As I mentioned earlier, I did tests to localhost and I noticed that the RAM is not growing. |
Test should be done to another machine. The reason is that the interface between two processes on the same machine is very fast, basically just writing to and reading from the machines internal memory, which based on the previous runs is some Gbps. This is probably why memory usage in not growing - every packet sent by the client is immediately consumed by the server and there is no need for network buffering. When sending to another machine, the bandwidth is much lower (25Mbps over WiFi and 100Mbps over Ethernet), and therefore network buffering is needed (writing Gbps for sending but sending 100Mbps from it). Can you run the tests again with sending to another machine? Note that I asked the run with |
|
@davidBar-On |
|
@olekstomek, unfortunately I don't see in the latest output anything that can help with understanding the problem. It just confirms that there is a problem related to the TCP buffer / cache. In normal situation once the transmit network buffers are full,
If I understand correctly, the problem happens only on one of these computers. Which of them show the problem when the client runs on it? Also, please send the output of the following shell command on both computers: |
The problem occurs on both computers. The first computer was the client and server and the second computer was also the client and server. Note that the upload results were ~ 5Gbps at one time and 18Gbps at one time - this was the result of the computer I use as a client. In the case of ~18Gbps, it was on a computer Dell, CPU Intel i5-10310U @ 1.70GHz 16GB RAM DDR4, SSD disk, in the case of ~5Gbps it was on computer Dell CPU Intel i5-5300U @ 2.30GHz 16GB RAM DDR3, SSD disk.
For Dell CPU Intel i5-10310U result is here: #1069 (comment) It's exactly the same (I copied the result of GET-NetTCPSetting and used ctrl + f in this thread) So it seems that it remains to wait, maybe someone else will come with a similar problem or we accidentally understand the problem on the occasion of another problem. |
I see that I start repeating myself ... The last test I thought of that you may want to try is disable the auto tuning of the window size, in case the TCP stack does not handle this parameter correctly. Under shell run the following commend and then run iperf3:
I agree (except maybe for the above test). You may also try to find a Windows forum and ask what may be the reason that when sending over TCP unlimited amount of memory is allocated for the TCP buffers or network transmit buffers. In any case, thanks for willing to put the effort to evaluate the issue. |
I did it and problem is the same. Additionally I restarted my OS and still the same. I also did a test on one of the computers where the upload is incorrect - I ran Ubuntu 20.04 from pendrive and installed iPerf 3.1.3 and the results are correct, below 100Mbps (but I noticed that the server had a different value and the transfer value was different on the Linux client in in each line, e.g. in the first line, I see that the client is sending data at a speed of e.g. 95Mbps and the server on Windows shows at this point that the transfer is e.g. 93.5Mbps)... But it's not unreal 5Gbps... I found some information about this here. The problem does not only occur with iPerf but also, for example, with a desktop application from here (result of upload significantly increase to several Gbps). Additionally, I could suspect that the computer is in the domain and has some special network policies imposed from above. Still, that doesn't mean I should be seeing incorrect results. Anyway, local tests I performed tests between computers in a local network without internet access.
I started a quick research on how to manipulate the speedtest results or cheat the test results - it's quite difficult because mostly ISP threads where they can prioritize traffic. But I found out that the problem does not only occur with Windows, but something similar also with Linux. The case is on Docker nerdalert/iperf3#2 but if I understand the problem is in image (image which works correct vs image which works incorrect).
I would also like to thank you for your time and suggestions. |
Context
Version of iperf3:
iPerf 3.1.3 (8 jun 2016 - 1.3 MiB for Windows Vista 64bits to Windows 10 64bits)
Hardware:
Dell, CPU Intel i5-10310U @ 1.70GHz 16GB RAM, SSD disk and
Dell CPU Intel i5-5300U @ 2.30GHz 16GB RAM, SSD disk
Operating system (and distribution, if any):
Windows 10 1909 Enterprise x64
Bug Report
Expected Behavior
Correct speed measurement
Actual Behavior
Incorrect speed measurement
Look at this:
It's client. 5 Gbits/sec is impossible.
It's server in the same time.
I have now changed that the former client is now the server and the server is now the client. This is a screen from the client. Over 12 Gbit/s. Take a look at RAM.
These tests were performed on a local network on two of my computers. Ports on the network card are 1Gbps so as you can see these results are not correct.
I saw a similar problem when I was measuring to an external server outside my local network.
Could the problem be with some configuration of my operating system? Of course I don't use VPN, proxy etc., I closed other apps.
In the task manager, this speed is not visible in the network card graph and values.
The text was updated successfully, but these errors were encountered: