You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Increasing ratelim (e.g., to 20 goroutines working simultaneously) seems to improve the download speed (considerably, but more tests are needed to state that with some certainty) up until the first ~100 packages (direct dependencies of go-ipfs) but then it slows down again to download speeds similar to the default ratelim of 2 scenario, when new levels of dependencies are processed in a different call of installDeps which normally has a small number of child dependencies and the parallel fetch is not leveraged.
The text was updated successfully, but these errors were encountered:
This will need to split installDeps into the fetch and the post-install hook process, as the current recursive implementation traversed the dependency DAG in DFS order to run the gx-godoRewrite function (through the hook) from bottom to top (because doRewrite needs the dependencies of a package already available). Since the network fetch seems like the most expensive operation (in contrast with the local rewrite/disk-install) delaying the second process to unify the first one wouldn't defeat the performance gain mentioned above.
Also, this separation would allow to parallelize the packages rewrite process which is now happening in DFS sequence (although I need to check the doRewrite logic to see if that parallelism is supported).
Actually testing a recent version of gx-go with your commit improving the rewrite logic already makes a huge performance improvement, so more parallelization there won't help much, the fetching part seems like the only bottleneck now.
Increasing
ratelim
(e.g., to 20 goroutines working simultaneously) seems to improve the download speed (considerably, but more tests are needed to state that with some certainty) up until the first ~100 packages (direct dependencies ofgo-ipfs
) but then it slows down again to download speeds similar to the defaultratelim
of 2 scenario, when new levels of dependencies are processed in a different call ofinstallDeps
which normally has a small number of child dependencies and the parallel fetch is not leveraged.The text was updated successfully, but these errors were encountered: