Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Look Into Filesystem-based Metadata Cacheing #9

Closed
riggsd opened this issue May 10, 2017 · 0 comments
Closed

Look Into Filesystem-based Metadata Cacheing #9

riggsd opened this issue May 10, 2017 · 0 comments

Comments

@riggsd
Copy link
Owner

riggsd commented May 10, 2017

Idea: SQLite3 database or fast key-value database like berkeleydb named .guano.py.cache with index of (filename, filesize, timestamp, hash).

Hash should be a fast non-cryptographic function like crc32, md5, sha1, xxHash.

If we determine that the file hasn't changed, load metadata from cache.

Would this be significantly faster given that we'd need to do full file reads to compute hash? Is (filename, filesize, timestamp) sufficient without a hash?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant