Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a single queue to install dependencies in installDeps #195

Closed
schomatis opened this issue Aug 8, 2018 · 4 comments · Fixed by #197
Closed

Use a single queue to install dependencies in installDeps #195

schomatis opened this issue Aug 8, 2018 · 4 comments · Fixed by #197
Assignees

Comments

@schomatis
Copy link
Collaborator

Increasing ratelim (e.g., to 20 goroutines working simultaneously) seems to improve the download speed (considerably, but more tests are needed to state that with some certainty) up until the first ~100 packages (direct dependencies of go-ipfs) but then it slows down again to download speeds similar to the default ratelim of 2 scenario, when new levels of dependencies are processed in a different call of installDeps which normally has a small number of child dependencies and the parallel fetch is not leveraged.

@schomatis
Copy link
Collaborator Author

This will need to split installDeps into the fetch and the post-install hook process, as the current recursive implementation traversed the dependency DAG in DFS order to run the gx-go doRewrite function (through the hook) from bottom to top (because doRewrite needs the dependencies of a package already available). Since the network fetch seems like the most expensive operation (in contrast with the local rewrite/disk-install) delaying the second process to unify the first one wouldn't defeat the performance gain mentioned above.

@schomatis
Copy link
Collaborator Author

Also, this separation would allow to parallelize the packages rewrite process which is now happening in DFS sequence (although I need to check the doRewrite logic to see if that parallelism is supported).

@Stebalien
Copy link
Collaborator

we currently parallelize rewriting files within a package but yeah, rewriting multiple packages at the same time would be nice.

@schomatis
Copy link
Collaborator Author

Actually testing a recent version of gx-go with your commit improving the rewrite logic already makes a huge performance improvement, so more parallelization there won't help much, the fetching part seems like the only bottleneck now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants