Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

swift: speed up directory listings with lots of big files #1791

Open
ncw opened this issue Nov 2, 2017 · 0 comments
Open

swift: speed up directory listings with lots of big files #1791

ncw opened this issue Nov 2, 2017 · 0 comments

Comments

@ncw
Copy link
Member

ncw commented Nov 2, 2017

Directory listings with lots of big files in can take a very long time to read because fetching the metadata for each file is done serially with reading the directory listing.

This could be sped up by fetching the metadata where necessary in parallel.

See the forum for original discussion

Update This also came up here and the idea of deferring the reading of the metadata until Size() is called. That won't help ls etc, but it will parallelize the reads and defer them until they are used. Unfortunately the API signature of Size doesn't allow an error return.

@ncw ncw added the enhancement label Nov 2, 2017
@ncw ncw changed the title swift: parallelize metadata reading in directory listing swift: speed up directory listings with lots of big files Dec 15, 2017
@ncw ncw added this to the Soon milestone Dec 15, 2017
@ivandeex ivandeex modified the milestones: Soon, Help Wanted Nov 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants