Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generator for listing files #131

Merged
merged 3 commits into from Sep 5, 2014
Merged

Generator for listing files #131

merged 3 commits into from Sep 5, 2014

Conversation

@jawnb
Copy link
Contributor

@jawnb jawnb commented Jan 9, 2013

I ran into some performance issues with SFTPClient.listdir_attr on LARGE directories (100k+ files).

The listdir_attr method of retrieving the file list, leaves a little to be desired. Immediately after sending a READDIR command, listdir_attr tries to read from the socket. This works OKAY for small directory listings where the number of READDIR attempts will be low.

For large directory listings, it results in some issues. The immediate issue is the large number of round trip requests that need to be done to get a full directory listing. The secondary issue is that these listing results are then stored in a list.

I created an listdir_iter that addresses these problems. READDIR requests are batched out before reading from the socket. Which gives a significant speed up. In my testing on a directory listing of 100k files, directory listing times have gone from 58 seconds to 6 seconds with this approach.

The change in functionality from returning a list to being a generator seems like a pretty big one. So I did not incorporate my changes into the existing listdir_attr. It felt like making a new method was the right choice.

Eager to hear any feed back or changes that you'd like to see made to this pull request.

@lndbrg
Copy link
Contributor

@lndbrg lndbrg commented Jan 21, 2014

LGTM!

@bitprophet bitprophet added this to the 1.15 milestone Aug 11, 2014
@bitprophet bitprophet merged commit 683b3c2 into paramiko:master Sep 5, 2014
1 check passed
1 check passed
@bitprophet
default The Travis build passed
Details
bitprophet added a commit that referenced this pull request Sep 5, 2014
bitprophet added a commit that referenced this pull request Sep 5, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants