Thanks, FYI - DirectoryCaching

[bschollnick]

Ben,

I wanted to let you know about this, since I am using Scandir as a building block for this code. Here's a good example of scandir making a radical performance improvement over os.listdir.

https://github.com/bschollnick/Odds-and-Ends (Look for Directory_Caching.py)

I ended up doing a complete rewrite in comparison to my original os.listdir solution, and scandir dramatically sped up the code mainly due to the reduced stat vs lstat calls, I suspect.

I welcome any feedback, comments, suggestions, if you choose to take a look. But as I mentioned, I just wanted to acknowledge your code's contribution to this solution.

[bschollnick]

By the way, here is some benchmark results:

Of course, please keep in mind, this benchmark was on a 2011 iMac. So decent hardware, but getting a bit aged.

Using a directory depth of 4, number of directories of 5, with 50 files, scandir on OS X Mavericks shows a 1.7x speed up over Os.listdir.

To help illustrate it, Scandir Using a directory depth of 4, number of directories of 5, with 2000 files, performance on OS X Mavericks shows a 4.2x speed up over Os.listdir (this was using the "slower ctypes version of scandir").

Scandir Using a directory depth of 4, number of directories of 5, with 1000 files, performance on OS X Mavericks shows a 4.5x speed up over Os.listdir (benchmark did not fall back to "slower" types).

[benhoyt]

Nice, thanks for sharing!