Slow SFTP transfers #12134
Comments
This is not a dupe of #12072. That one says that CyberDuck doesn't support compression, resulting in slower transfers. While that may be true for WAN transfers, adding compression over LAN transfers will actually slow them down. The issue is that CyberDuck is about 10x slower than the sftp client on a LAN, with compression disabled. Isn't this a really easy one to verify? |
I noticed the same thing. Cyberduck: A few 100 KByte/s. Command line scp: a few MByte/s. |
The problem still seems to exist in the latest pre-release 2.5.5. Download speed through SFTP is still approx. 10% of the available bandwidth. |
Same for me, using the 2.5.5, I get ~85KB/s using Cyberduck, and ~400KB/s using the command line sftp client. |
This is a bug in the SSH I/O code and has nothing to do with the Java language used. Java has a very good network performance in general; e.g. see FTP transfers in Cyberduck. Please don't write any silly annonymous postings here, please. |
Ditto. Fugu transfers files at 11-12MBps whereas CyberDuck transfers up to about 3MBps if I'm transfering multiple items at once or ~1-1.2 MBps for a single transfer. Also, CyberDuck crashes on large file transfers (4-20GB per file) which may or may not be a symptom of the same problem. |
I concur on this one. ~88 kBps with Cduck, and ~7.8MB (yes, that's bytes) with scp. |
I'm experiencing the same problem, 240KB/s with Fugu... 20KB/s with Cyberduck... Any idea when this is going to be fixed? |
I confirm it, 200KB/s with Cyberduck, 1.8MB/s by scp on terminal. |
I may have just entered a duplicate of this bug at http://trac.cyberduck.ch/ticket/656 My symptoms appears to be the same. My connection is not inside a LAN, though. |
Refer to the latest nightly builds at (http://update.cyberduck.ch/nightly/). Try Preferences > SFTP > File Transfers > Use SCP. |
Hi, I'm very thankful that this is fixed however I would STRONGLY suggest that the default setting is set to SCP for SFTP transfers instead of having to dig into these forums to try to determine that this in fact solves the problem. ALSO if someone elects SFTP as the transfer protocol it would be nice to have a message stating that there are issues with large transfers and the order of magnitude may be 10x slower as an FYI due to underlying libraries (not Cyberduck). --Nikolaos |
goodness gosh this is slow. i have set it to SCP and even tiny files are taking ages to upload. the menu to select active over passive connect mode is greyed out. i think i am going to need a different FTP client unless i can fix this. |
Set to SCP and still it's ridiculous, 4-5MB/s vs 25-30MB/s. SFTP is dumbfounding, 200KB/s?!? Why the fuck is this thing so slow? Thank god I'm good with a CLI; but I can't recommend this app to friends in good conscious. |
switch to scp ind sftp settings solved it for me |
when I switched to scp, speed problem was gone. |
Just to say, I've got the same problem with the latest version. For example, downloading from a Linode VPS via SFTP today I get about 50KB/s using Cyberduck, or 800KB/s with Fugu (or rsync, for that matter). Cyberduck is no use to me until this is fixed, so I'd certainly call that a defect rather than an enhancement. Shame, because I like its style. |
I'm still a little confused about whether Cyberduck supports compression over SFTP (as in rsync -z). Do you have to use either SFTP or SCP mode to get this to work? Fugu has a checkbox for this for each connection. Even if it does, the user interface could better clarify this with a "compression" checkbox under "More Options" in the "Open Connection" window. Of course there are times when this compression is desirable and times when it isn't, so a per-connection setting would be really handy. |
Cyberduck was always slow and it is still in the 3.2.1. Just had 35KB/s in comparison to 650KB/s with scp. |
Four years and still nothing. This is marked as an enhancement, yet this is quite a major defect. Enough so to keep me away. |
I'm sorry? I really like cyberduck for FTP, but marking this major defect as an enhancement makes no sense. SFTP transfers are dog slow, and nearly unusable. Why? Why no love for SFTP? Why? Don't you see all the children crying in the street for reasonable-speed encrypted transfers? |
For those complaining how SCP is faster than SFTP: So, when replying to this "enhancement", please use the sftp command for comparison, not scp. |
what's wrong about how filezilla does it. SFTP performance is excellent there. |
Hi, Is there a way to solve these speed issues ? There's still a lot's of problem in the latest version (just downloaded the latest one today 3.5(6066) ). and sftp transfert are around 6 to 10 x slower than using the scp /sftp command line. Regards, Caeies. |
As of Version 3.5.1 (6117) it is still an issue. Regardless of protocol selected (SFTP/SCP) transfer speeds are 10 time slower for SCP and up to 20 times slower for SFTP when compared to command line tools. |
I added a small patch as a POC for upload limitation. This patch will illustrate that doing parallel request for writing is faster than the current code. However, due to my little understanding of your code, This POC is a little bit buggy since it will not clear the waiting queue for the last 64 requests ... And I've got no idea on how to do this with the current framework you're using. Hope this helps. Caeies |
Replying to [comment:56 caeies]:
You could try what impact the cipher has on the transfer speed. We are usually negogiating |
Replying to [comment:63 caeies]:
Thanks for the patch. I have something half-baked here as well and will try to see if I can combine best of both worlds. |
Replying to [comment:62 caeies]:
I don't see how the patch attached changes this behaviour. It will still toggle between writing and reading the status. |
Replying to [comment:65 dkocher]:
Let me know the results with the patch attached and how it compares to the previous upload implementation. |
Replying to [comment:66 dkocher]:
Yes, but after parallelism requests only. So it doesn't wait for the network latency just after sending the first request and so on ... it will read back the answer only after the 64th requests, so when the server starts to reply to the requests and that the requests are back to the client. Anyway, the main problem of this patch is the lasts 64th requests that aren't checked at the end of the download. Hope I'm clear Caeies. |
Replying to [comment:68 caeies]:
I understand and the patch my own patch attached should fix this issue with the same characteristics of sending out multiple write requests before reading any status. Let me know about the results compared to no parallelism. Thanks again for your help here, much appreciated! |
Replying to [comment:67 dkocher]:
Sorry, I don't have the time right now to make the tests, but I will try to look at them this WE. I go quickly through your patch, and I'm not sure of all the changes you made. I will try to look carefully this WE too. Do you have any other changes in the pipe for SFTP client ? So I don't take too much time to change something already changed :). Thanks for your help. Caeies. (PS : whoops, you were quicker than me to reply :). |
Replying to [comment:70 caeies]:
I attached a modified list of the preferred ciphers as a patch. You can experiment with these as well to test if that impacts throughput. I don't have anything else here. |
Replying to [comment:64 dkocher]:
So more info on this : I try sftp with the -o Ciphers=aes256-ctr option. The download speed is around the same than previously. But the window size looks like the same as the one used by cyberduck (which looks great :). So the download speed issue is still there. I will try to switch to weaker cipher as suggested in cyberduck, but not sure it's the issue. regards, Caeies |
Replying to [comment:71 dkocher]:
Regards, Caeies. PS: btw the svn wasn't able to compile this morning due to @OverRide in 3 *Sessions files. |
Hum, still the download speed issue. I tried a higher parallelism factor but that doesn't help. My last guess is that perhaps we are hitting performances issues in the code of the download. The window size is always "low" no more than ~98343 when the window size of sftp is higher + 134K). Unfortunately I don't have the time to profile cyberduck and to look at where is the bottleneck. Probably in the write parts of the download requests (But that could be my computer too :). I guess that cyberduck is near its maximum performance regarding the network layout (As the upload speed seems to indicate). I will be glad to test new code. Regards, Caeies. |
Replying to [comment:73 caeies]:
Changed priority of ciphers In dfbef4b. |
Replying to [comment:68 caeies]:
Commited upload parallelism with ACK queuing in 657339c. |
Replying to [comment:74 caeies]:
I'll leave this closed this until we find clear evidence of other areas for improvement. |
Any ideas on how to profile the java code under Mac OSX ? I tried with Shark, but it doesn't seems to be easy to profile a java application :( |
Replying to [comment:78 caeies]:
Your best bet is a Java Profiler which should be able to connect to the running application. |
If I use the command line sftp client on the local network, I can get transfer speeds of several MB/s.
If I use CyberDuck, the fastest it gets is 600KB/s, which was after I tried increasing the buffer size to 65536.
Is this a known problem? Cyberduck is still nice, but useless for local networks...
Attachments
Ciphers.patch(10.8 KiB)Upload_with_status_queue.patch(9.4 KiB)patch_upload_poc.diff(5.7 KiB)The text was updated successfully, but these errors were encountered: