Generator for listing files #131

Merged
merged 3 commits into from Sep 5, 2014

Projects

None yet

3 participants

@jawnb
Contributor
jawnb commented Jan 9, 2013

I ran into some performance issues with SFTPClient.listdir_attr on LARGE directories (100k+ files).

The listdir_attr method of retrieving the file list, leaves a little to be desired. Immediately after sending a READDIR command, listdir_attr tries to read from the socket. This works OKAY for small directory listings where the number of READDIR attempts will be low.

For large directory listings, it results in some issues. The immediate issue is the large number of round trip requests that need to be done to get a full directory listing. The secondary issue is that these listing results are then stored in a list.

I created an listdir_iter that addresses these problems. READDIR requests are batched out before reading from the socket. Which gives a significant speed up. In my testing on a directory listing of 100k files, directory listing times have gone from 58 seconds to 6 seconds with this approach.

The change in functionality from returning a list to being a generator seems like a pretty big one. So I did not incorporate my changes into the existing listdir_attr. It felt like making a new method was the right choice.

Eager to hear any feed back or changes that you'd like to see made to this pull request.

@lndbrg
Contributor
lndbrg commented Jan 21, 2014

LGTM!

@bitprophet bitprophet added Support Feature and removed Support labels Aug 11, 2014
@bitprophet bitprophet added this to the 1.15 milestone Aug 11, 2014
@bitprophet bitprophet merged commit 683b3c2 into paramiko:master Sep 5, 2014

1 check passed

default The Travis build passed
Details
@bitprophet bitprophet added a commit that referenced this pull request Sep 5, 2014
@bitprophet bitprophet Add quick test re #131 e5fc6a6
@bitprophet bitprophet added a commit that referenced this pull request Sep 5, 2014
@bitprophet bitprophet Changelog re #131 35b9d15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment