Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use async calls for getting info about subdirectories #139

Conversation

yoel-ross-zip
Copy link
Contributor

When building the model for a directory, there are repeated synchronous uses of the fs.lstat method. I have found that this can cause some serious latency when working with larger folders in s3, and would like to offer a couple of improvements:

  1. get the metadata for each file once, and use both for removing deleted markers and getting last modified time.
  2. use the underlying s3fs._info method instead of lstat. The _info method is async, and all the calls can be made concurrently with the existing s3fs event loop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants