Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Generator for listing files #131
I ran into some performance issues with SFTPClient.listdir_attr on LARGE directories (100k+ files).
The listdir_attr method of retrieving the file list, leaves a little to be desired. Immediately after sending a READDIR command, listdir_attr tries to read from the socket. This works OKAY for small directory listings where the number of READDIR attempts will be low.
For large directory listings, it results in some issues. The immediate issue is the large number of round trip requests that need to be done to get a full directory listing. The secondary issue is that these listing results are then stored in a list.
I created an listdir_iter that addresses these problems. READDIR requests are batched out before reading from the socket. Which gives a significant speed up. In my testing on a directory listing of 100k files, directory listing times have gone from 58 seconds to 6 seconds with this approach.
The change in functionality from returning a list to being a generator seems like a pretty big one. So I did not incorporate my changes into the existing listdir_attr. It felt like making a new method was the right choice.
Eager to hear any feed back or changes that you'd like to see made to this pull request.