New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generator for listing files #131

Merged
merged 3 commits into from Sep 5, 2014

Conversation

Projects
None yet
3 participants
@jawnb
Contributor

jawnb commented Jan 9, 2013

I ran into some performance issues with SFTPClient.listdir_attr on LARGE directories (100k+ files).

The listdir_attr method of retrieving the file list, leaves a little to be desired. Immediately after sending a READDIR command, listdir_attr tries to read from the socket. This works OKAY for small directory listings where the number of READDIR attempts will be low.

For large directory listings, it results in some issues. The immediate issue is the large number of round trip requests that need to be done to get a full directory listing. The secondary issue is that these listing results are then stored in a list.

I created an listdir_iter that addresses these problems. READDIR requests are batched out before reading from the socket. Which gives a significant speed up. In my testing on a directory listing of 100k files, directory listing times have gone from 58 seconds to 6 seconds with this approach.

The change in functionality from returning a list to being a generator seems like a pretty big one. So I did not incorporate my changes into the existing listdir_attr. It felt like making a new method was the right choice.

Eager to hear any feed back or changes that you'd like to see made to this pull request.

@lndbrg

This comment has been minimized.

Contributor

lndbrg commented Jan 21, 2014

LGTM!

@bitprophet bitprophet added Support and removed Support labels Aug 11, 2014

@bitprophet bitprophet added this to the 1.15 milestone Aug 11, 2014

@bitprophet bitprophet merged commit 683b3c2 into paramiko:master Sep 5, 2014

1 check passed

default The Travis build passed
Details

bitprophet added a commit that referenced this pull request Sep 5, 2014

bitprophet added a commit that referenced this pull request Sep 5, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment