extremely slow upload speed, using mod_sftp #1288
Comments
|
Can you provide the output from using the Linux CLI, e.g. what does |
Castaglia
changed the title
|
Hello, I have attached a log from running sftp -vvv agains the proFTPD server. I have only inserted the log AFTER the connection has been established, on the actual upload. The speed gets up to 600kb/s today and then starts decreasing. |
|
Hello, |
|
Unfortunately, the logs showing before the upload starts are also useful (including |
|
Hello, |
|
Hello, |
|
Hello, Sadly, I am not in a position to try and upgrade proFTPD from 1.3.5e to a newer version but I have red that there might be some mod_sftp improvements in 1.3.6. |
|
There have been some other improvements for SFTP uploads in the 1.3.9 development cycle; in particular, we now avoid an expensive (and unnecessary) memory copy for each SFTP I know you mentioned you may not be able to update ProFTPD yet, but I wanted you to know that I've not forgotten this issue. |
|
Hi @Castaglia, Upload command: Download command: Unfortunately, it seems that ProFTPD is approximately 4 to 5 times slower than OpenSSH. It looks to me that different channel packet sizes ( @Castaglia Have you got an idea what is going wrong here? I have attached the four output logs. I hope, that it helps to find the issue. put-proftpd.log |
|
ver:1.3.8 出现相同的问题 |
|
I did some performance tests on a share mounted at Write test: Read test: These are the captured transfer logs: |
|
@cfiehe @Castaglia mod_sftp can auto adjust buffer size？ openssh log===> debug2: channel 0: rcvd adjust 98421 |
|
@a270443177 Thanks for the hint. I have modified the test case and used added the options So far, I have not found a way to increase buffer size in case of SSHFS. |
|
@cfiehe FYI, I've now implemented the "limits@openssh.com" SFTP extension in With this, you should no longer need any client-side |
|
@Castaglia openssh-server-7.4p1-21.el7.x86_64 sftp -vvvv display this: same speed |
|
That is awesome. Thank you very much @Castaglia. I did some testing. The handshake succeeds and I can login, but the upload fails with |
Fascinating. Can you check the server-side logging, to see |
To answer my own question, it looks like this is coming from OpenSSH itself: which, in turn, means this is essentially an OpenSSH bug. The I guess that means, in the mean time, I might be forced to make the |
|
Ah-ha! Looks like this OpenSSH behavior (of not respecting its own limits) did get fixed, but in a later version than that you appear to be using @cfiehe see openssh/openssh-portable@48bf234 Looks like the above fix shipped in OpenSSH 9.2/9.2p1. Which means that the |
|
Ah, ok. Those if conditions are not nice, but actually the only chance to work around the issue. Fortunately, it has been fixed on client side. We use Ubuntu 22.04 that comes with an |
…se the `limits@openssh.com` extension, yet cannot properly handle the longer mod_sftp lengths.
…se the `limits@openssh.com` extension, yet cannot properly handle the longer mod_sftp lengths.
…se the `limits@openssh.com` extension, yet cannot properly handle the longer mod_sftp lengths.
|
Hi @Castaglia. Pretty cool. I did a quick test with the sftp client and build ProFTPD based on your workaround Now, we achieve the upload and download results from here #1288 (comment). I have tested it with again with the OpenSSH client Unfortunately, there remains a gap in speed in comparison with the OpenSSH server of factor 2. |
|
There seems to be some kind of adaption in the SFTP client when talking with the OpenSSH server. The magic seems to happen here https://github.com/openssh/openssh-portable/blob/00e63688920905e326d8667cb47f17a156b6dc8f/channels.c#L3660, but I am not sure. I have created new logs: |
|
Maybe the trigger is |
…se the `limits@openssh.com` extension, yet cannot properly handle the longer mod_sftp lengths. (#1800)
Excellent news. That PR has been merged to master.
We can continue to examine this (and in Issue #1751), but know that I am not aiming to have mod_sftp perform just like OpenSSH. In my mind, requests/questions about "performance" are incredibly subjective. Instead, I think in terms of "what latencies are required for your use case, and why?" Performance optimization covers the operating systems of clients and servers, the application implementations, every single networking component in the path, libraries, designs, etc. In short, one can spend years just on this topic -- and I don't always have the time/patience for performance optimizations. I say this just as a reminder that at some point, I will consider the mod_sftp performance to be "good enough for now". OpenSSH has always performed better, and that hasn't stopped users like yourself from enjoying mod_sftp. |
|
You are right. Time is the most value thing we have. We have to spend it consciously. The implementation of the |
|
@cfiehe I took a look at the provided logs. In comparing the SFTP uploads to ProFTPD vs OpenSSH, the client uses the same buffer lengths in the For the SSH window (used as a backpressure mechanism similar to TCP, since SSH allows for multiplexing multiple concurrent channels; see section "5.2. Data Transfer" of RFC 4254), we see: The initial window size set by mod_sftp is 4GB. I do this deliberately, so that the number of "window adjustment" messages, needed to "open" the window", is low. Fewer messages leads to lower latency, especially over long-lived channels and large data transfers. OpenSSH, on the other hand, starts with a zero window size, then immediately adjusts it, adding 2097152 bytes. We can look at the number of these "adjustment" messages in the two uploads, to see if that contributes to the overall latency: I think this rules out the SSH channel window as a contributing factor to the overall latency difference. What are some other contributing factors? With long-lived channels and large data transfers, the overall algorithms negotiated can have an impact. Let's see what algorithms were negotiated: The difference in host key algorithm won't make difference for data transfer latency. The main factor for data transfer latency will be the cipher and MAC algorithms. Here, we see AES128-GCM used for the mod_sftp transfers, vs ChaChaPoly used for the OpenSSH transfers. Other users have reported similar to negligible timing differences between AES128-GCM and ChaChaPoly algorithms (see #1751 (comment)), but here we get into the OpenSSL versions, and underlying CPUs, used. So you might add/configure ChaChaPoly support to your Everything else looked quite similar, which means that a much more in-depth (and reproducible) investigation would be needed to understand the latency differences. |
|
Thanks for your investigation. I have set the cipher with put-proftpd-20240427.log I do not know if it helps, but that is the output of And my configuration: |
…memset(3)` calls in hot code paths in mod_sftp.
|
@cfiehe #1802 might be of some interest for your latency tests. I used flamegraphs to see where some of the CPU time was being spent, and removed some unnecessary In addition, in the flamegraphs, I saw how much "overhead" occurs when trace logging is enabled (so disable that for less latency). I would also recommend that |
|
@Castaglia Thanks a lot for your analysis. I built a new version of ProFTPD and set |
Excellent, thanks for the tests. I'll commit that PR, and continue looking at these flamegraphs to find other hot code paths, and see what kinds of optimizations I can easily find. |
…memset(3)` calls in hot code paths in mod_sftp. (#1802)
…che the initial results of the `dir_check()` function at the start of the transfer. According to flamegraphs, there is a fair amount of CPU time spent in `dir_check()` and related functions, almost all of which is redundant. Those access checks apply to conditions which do not change over the course of the data transfer.
…che the initial results of the `dir_check()` function at the start of the transfer. According to flamegraphs, there is a fair amount of CPU time spent in `dir_check()` and related functions, almost all of which is redundant. Those access checks apply to conditions which do not change over the course of the data transfer.
|
Great. Thanks a lot @Castaglia. I did some testing, but unfortunately, I could not measure any improvement of the transfer speed. I get: |
…che the initial results of the `dir_check()` function at the start of the transfer. (#1805) According to flamegraphs, there is a fair amount of CPU time spent in `dir_check()` and related functions, almost all of which is redundant. Those access checks apply to conditions which do not change over the course of the data transfer.
What I Did
Hello,
We are have a strange issue regarding upload speeds. Our proFTPD server is installed on a CentOS7 machine and is configured to use LDAP authentication and mod_sftp. It is running on a custom port- 2222.
Downloading is as expected, around 10MB/s when testing with 1GB file.
Uploading is no where near that, running at 300-500kb/s when testing with the same 1GB file.
At the same time, trying the standart SFTP on port 22 doe not have this problem. Upload and Download there is fast, as expected. This should mean that the issue is not in our Network.
No transfer limits have been set and enforcing such did not improve the UPLOAD speed.
I have tried it using FileZilla and Linux CLI as client connections.
What I Expected/Wanted
I have been banging my head for days now. Any help towards troubleshooting this will be much appreciated.
ProFTPD Version and Configuration
VERSION:
CONFOGURATION FILE:
The text was updated successfully, but these errors were encountered: