Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Performance of paramiko is pretty slow (with possible solution) #175
I am using paramiko and doing a sftp file transfer from a twisted sftp server.
So I checked which remote window size each of the tools set
Putty uses: 2147483647
So for a first test I patched the paramiko transport window size to be the same as putty's
and then I reduced the window size back until I saw the first performance degradations.
Here are some (non scientific) performance measurements with client and server on the same localhost
Putty psftp: 10.00 MB/sec
So I think the default winow size of paramiko should be raised to be at least 134217727
Finally I enabled compression in paramiko by calling self.trans.use_compression() which increased the download performance to 52.63 MB/sec for a file containing only zeros and decreased the performance to 6.88 MB/sec for a file containing complete random data.
For reference this was the unmodified paramiko profile, somehow even in the optimized case a lot of time is spent in the aquire method. I don't understand this since this measurement of acquire and release on Win7 returns a much higher performance:
and this with window_size = 134217727
For what it's worth, the default window size in OpenSSH is 2097152.
(see CHAN_TCP_WINDOW_DEFAULT definition in http://www.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/channels.h?rev=1.113)
The following worked for me, performance went from 0.55 MB/sec to 4.5 MB/sec:
class FastTransport(paramiko.Transport): def __init__(self, sock): super(FastTransport, self).__init__(sock) self.window_size = 2147483647 self.packetizer.REKEY_BYTES = pow(2, 40) self.packetizer.REKEY_PACKETS = pow(2, 40) ssh_conn = FastTransport(('host.example.com', 22)) ssh_conn.connect(username='username', password='password') sftp = paramiko.SFTPClient.from_transport(ssh_conn)
referenced this issue
Sep 25, 2013
Specification says that both sides are allowed to send a message to adjust the windows size (sizes up to 2^32-1 is ok or as an int: 4294967295):
The message to send is SSH_MSG_CHANNEL_WINDOW_ADJUST, in paramiko it is called MSG_CHANNEL_WINDOW_ADJUST. If I understand the code correctly, we are only listiening on the server sending this message.
We should probably also look into how we can modify the packet size, if needed.
This is an important bug to fix. Adding this tiny line of code cut the time to transfer one file from 60 seconds to 9 seconds:
2 MB (the OpenSSH default) worked very well too; 3 MB gave me a slight boost (5%). This is on a link with RTT around 250 ms.
Not sure if this is exactly related, but I'm experiencing speed issues with
The file is written correctly (correct contents and modified date), but the connection lags for 10 min before closing.
I haven't tried with larger file sizes and don't know if this lag time scales linearly.
I am still having this bug with paramiko 1.15.2 (and paramiko master). I was reading the strace output, and it looks like paramiko is reading 8 bytes at a time?!
EDIT: It looks like fd 3, the one it keeps reading 8 bytes from, is /dev/urandom
EDIT: I was stracing the main thread, not the one actually doing the download. Here's the strace of the thread:
As an FYI to anyone else who might run into speed issues when writing/uploading files with
Here's an example:
import paramiko ssh_client = SSHClient() ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) ssh_client.connect('host', username='username', password='password') sftp_client = ssh_client.open_sftp() with sftp_client.file('/path/to/file', mode='w') as file: file.set_pipelined() file.writelines([line + '\n' for line in lines])
Hello, Does the window_size solution works with SSHClient? Im new to paramiko. But it looks like its very slow compared to executing a command from the shell client. Im using paramiko to execute hive queries in a hive host from another host . i use python paramiko as below.
ssh = paramiko.SSHClient()
the last line consumes much more time compared to the same command run from shell window
@antoncohen I tried some combinations. And for large data output i did have the hanging error. But window_size does not seem to resolve the issue . But moving
My actual problem i posted above was for any command which returns even tiny data. Any command i execute it takes atleast 5 seconds to finish. So because i use several commands one after other, it amounts to a bigger delay overall.
I'm running into similar issues with running a python Lambda function (code below). It runs for a few minutes and then Lambda times out. The logs show about 14% of the 15MB file transferred. I can increase the Lambda execution time and/or memory and it has no impact.
This issue happens when connecting to our internal dev SFTP; however, when I connect to the client prod SFTP then the file downloads within seconds.
def printTotals(transferred, toBeTransferred):
def lambda_handler(event, context):