Problem downloading files from SFTP Servers that have a "one time download" mechanism

[mwarnes]

Hi

We ran into an issue downloading files from an IBM Sterling Integrator SFTPmailbox, which has a one time download flag (Extractable count=1) set on the file.

Without the Extractable count set to 1 file downloads work as expected

INFO:paramiko.transport.sftp:[chan 0] Opened sftp connection (server version 3)
DEBUG:paramiko.transport.sftp:[chan 0] listdir('/Testa')
DEBUG:paramiko.transport.sftp:[chan 0] stat('/Testa/scc.pk7')
DEBUG:paramiko.transport.sftp:[chan 0] open('/Testa/scc.pk7', 'rb')
DEBUG:paramiko.transport.sftp:[chan 0] open('/Testa/scc.pk7', 'rb') -> 31
DEBUG:paramiko.transport.sftp:[chan 0] stat('/Testa/scc.pk7')
DEBUG:paramiko.transport.sftp:[chan 0] close(31)
INFO:paramiko.transport.sftp:[chan 0] sftp session closed.
DEBUG:paramiko.transport:[chan 0] EOF sent (0)
DEBUG:paramiko.transport:EOF in transport thread

The problem is that when a file has an Extractable count of 1 the moment it is opened the extractable count is set to zero to prevent any further downloads (if the original download attemt fails the count is reet). What we've see is that after the file is opened an additonal SSH_FXP_STAT command is issued but be cause the Extractable count is zero the IBM Sterling Integrator SFTP Server returns a file not found error=2 and the transfer fails.

INFO:paramiko.transport.sftp:[chan 0] Opened sftp connection (server version 3)
DEBUG:paramiko.transport.sftp:[chan 0] listdir('/Testa')
DEBUG:paramiko.transport.sftp:[chan 0] stat('/Testa/scc.pk7')
DEBUG:paramiko.transport.sftp:[chan 0] open('/Testa/scc.pk7', 'rb')
DEBUG:paramiko.transport.sftp:[chan 0] open('/Testa/scc.pk7', 'rb') -> 31
DEBUG:paramiko.transport.sftp:[chan 0] stat('/Testa/scc.pk7')
DEBUG:paramiko.transport.sftp:[chan 0] close(31)
Traceback (most recent call last):
File "testsftp.py", line 71, in
main()
File "testsftp.py", line 24, in main
ftp.retrieve_files(remote_location)
File "testsftp.py", line 61, in retrieve_files
self._retrieve_files(file_list)
File "testsftp.py", line 57, in _retrieve_files
self.ftp.get(remote_file_path, file_location)
File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 720, in get
size = self.getfo(remotepath, fl, callback)
File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 689, in getfo
file_size = self.stat(remotepath).st_size
File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 413, in stat
t, msg = self._request(CMD_STAT, path)
File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 729, in _request
return self._read_response(num)
File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 776, in _read_response
self._convert_status(msg)
File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 802, in _convert_status
raise IOError(errno.ENOENT, text)
IOError: [Errno 2] The message [/Testa/scc.pk7] is not extractable!
DEBUG:paramiko.transport:EOF in transport thread
INFO:paramiko.transport.sftp:[chan 0] sftp session closed.

Upon investigation we found that the STAT command was issued in get() to obatin the file size and again in getfo() and prefetch() which does not work if the SFTP Server is flagging as not available after the initial open.

I made a small modification to use the file size returned from the initial STAT command in get() and pass it to getfo() and prefretch() as a parameter so that further STAT commands are not issued for the remote file.

INFO:paramiko.transport.sftp:[chan 0] Opened sftp connection (server version 3)
DEBUG:paramiko.transport.sftp:[chan 0] listdir('/Testa')
DEBUG:paramiko.transport.sftp:[chan 0] stat('/Testa/scc.pk7')
DEBUG:paramiko.transport.sftp:[chan 0] open('/Testa/scc.pk7', 'rb')
DEBUG:paramiko.transport.sftp:[chan 0] open('/Testa/scc.pk7', 'rb') -> 31
DEBUG:paramiko.transport.sftp:[chan 0] close(31)
INFO:paramiko.transport.sftp:[chan 0] sftp session closed.
DEBUG:paramiko.transport:[chan 0] EOF sent (0)
DEBUG:paramiko.transport:EOF in transport thread

Patch against current repository below:

Cheers .. Martin Warnes

diff --git a/paramiko/sftp_client.py b/paramiko/sftp_client.py
index 6d48e69..eb0ac99 100644
--- a/paramiko/sftp_client.py
+++ b/paramiko/sftp_client.py
@@ -669,7 +669,7 @@ class SFTPClient(BaseSFTP, ClosingContextManager):
with open(localpath, 'rb') as fl:
return self.putfo(fl, remotepath, file_size, callback, confirm)

• def getfo(self, remotepath, fl, callback=None):

• def getfo(self, remotepath, fl, file_size=0, callback=None):
  """
  Copy a remote file (remotepath) from the SFTP server and write to
  an open file or file-like object, fl. Any exception raised by
  @@ -679,6 +679,9 @@ class SFTPClient(BaseSFTP, ClosingContextManager):
  :param object remotepath: opened file or file-like object to copy to
  :param str fl:
  the destination path on the local host or open file object

•    :param int file_size:
  

•                optional size parameter passed to callback. If none is specified,
  

•                size defaults to 0
   :param callable callback:
       optional callback function (form: `func(int, int)`) that accepts
       the bytes transferred so far and the total bytes to be transferred
  

  @@ -687,8 +690,7 @@ class SFTPClient(BaseSFTP, ClosingContextManager):
  .. versionadded:: 1.10
  """
  with self.open(remotepath, 'rb') as fr:

•        file_size = self.stat(remotepath).st_size
  

•        fr.prefetch()
  

•        fr.prefetch(file_size)
       size = 0
       while True:
           data = fr.read(32768)
  

  @@ -716,9 +718,10 @@ class SFTPClient(BaseSFTP, ClosingContextManager):
  .. versionchanged:: 1.7.4
  Added the callback param
  """
  file_size = self.stat(remotepath).st_size
  with open(localpath, 'wb') as fl:

•        size = self.getfo(remotepath, fl, callback)
  

•        size = self.getfo(remotepath, fl, file_size,callback)
   s = os.stat(localpath)
   if s.st_size != size:
       raise IOError('size mismatch in get!  %d != %d' % (s.st_size, size))
  

  diff --git a/paramiko/sftp_file.py b/paramiko/sftp_file.py
  index d0a37da..29e9ca4 100644
  --- a/paramiko/sftp_file.py
  +++ b/paramiko/sftp_file.py
  @@ -379,7 +379,7 @@ class SFTPFile (BufferedFile):
  """
  self.pipelined = pipelined

• def prefetch(self):

• def prefetch(self, size):
  """
  Pre-fetch the remaining contents of this file in anticipation of future
  .read calls. If reading the entire file, pre-fetching can
  @@ -393,7 +393,6 @@ class SFTPFile (BufferedFile):

 .. versionadded:: 1.5.1
 """

•    size = self.stat().st_size
   # queue up async reads for the rest of the file
   chunks = []
   n = self._realpos
  

[lndbrg]

@mwarnes please send it as a pull request, the formatting makes it incredibly hard to decipher the changes.

[mwarnes]

@Indbrg sorry for the delay, pull request created

#579

[smontanaro]

We hit the same problem with a Wells Fargo sftp site. Installing 1.15.2 with @mwarnes patch applied seems to have solve the problem. Of course, we had been using 1.11.0, so we will have to verify that the upgrade doesn't break something else (our app folks don't use virtualenvs, which would simplify things). Thanks for the fix.

[bitprophet]

@smontanaro A) thanks for confirming the patch! B) Would you be able to try out #562 as well? I suspect they address the same issue.

[smontanaro]

Unfortunately, not really. Wells Fargo made no test environment available on their end, and we don't really have a good test capability on our end. We "tested" the #579 patch in production, and our IT folks don't want to rock the boat anymore than they already have.

[mwarnes]

@bitprophet This does look like the same patch as #562 If needed I make a IBM SI Server available for testing, ping me on the side to get details.

[bitprophet]

Going to roll this into #194 which seems to be the oldest ticket about the issue.

@mwarnes, access to an IBM server exhibiting this problem would be handy, thanks! Though if I am able to prove the changes don't break things for vanilla OpenSSH SFTP, I may merge on faith anyways. Depends how fast you get back to me & how fast I wrap up the other tickets for 1.16 :D

Please follow up with me on #194 or at my email (jeff [at] bitprophet.org).