Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Thanks, FYI - DirectoryCaching #19

Closed
bschollnick opened this Issue · 2 comments

2 participants

@bschollnick

Ben,

I wanted to let you know about this, since I am using Scandir as a building block for this code. Here's a good example of scandir making a radical performance improvement over os.listdir.

https://github.com/bschollnick/Odds-and-Ends (Look for Directory_Caching.py)

I ended up doing a complete rewrite in comparison to my original os.listdir solution, and scandir dramatically sped up the code mainly due to the reduced stat vs lstat calls, I suspect.

I welcome any feedback, comments, suggestions, if you choose to take a look. But as I mentioned, I just wanted to acknowledge your code's contribution to this solution.

@bschollnick

By the way, here is some benchmark results:

Of course, please keep in mind, this benchmark was on a 2011 iMac. So decent hardware, but getting a bit aged.

Using a directory depth of 4, number of directories of 5, with 50 files, scandir on OS X Mavericks shows a 1.7x speed up over Os.listdir.

To help illustrate it, Scandir Using a directory depth of 4, number of directories of 5, with 2000 files, performance on OS X Mavericks shows a 4.2x speed up over Os.listdir (this was using the "slower ctypes version of scandir").

Scandir Using a directory depth of 4, number of directories of 5, with 1000 files, performance on OS X Mavericks shows a 4.5x speed up over Os.listdir (benchmark did not fall back to "slower" types).

@benhoyt
Owner

Nice, thanks for sharing!

@benhoyt benhoyt closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.