Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
x/vgo: lookup/fetch dependencies concurrently #24316
Ran vgo build for program that imports a large dependency tree (imports k8s.io/kubernetes/something).
It worked for several minutes, printing lines like this at a steady pace:
It looks like vgo works through list dependencies sequentially (recursively appending additional ones as they are discovered). Perhaps this could be parallellized?
For example A depends on B, C, D, E, F, G. D and F depend on X 1.1 and X 1.5 respectively. Use worker pool of size 3. All "finalized" versions are kept in some synchronized structure (or map) - process A. - process B, C, D, add X 1.1 to queue - process E, F, G, add X 1.5 to queue - process X 1.1, X1.5 - if 1.1 gets the lock first, 1.5 overrides the picked version after - if 1.5 gets the lock first, 1.1 is ignored If you have "blacklisted" versions, those are recorded in the store and get handled if the dependency comes up again. In my experience, many packages have >>2 dependencies and the transient dependencies don't overlap too often. Of course, that's from a limited sample size. The implementation would certainly have more edge cases than serial approach. But so far I don't think it's too complex and would benefit some larger projects.…
On Thu, Mar 8, 2018 at 5:45 PM Chotepud Teo ***@***.***> wrote: How would you propose it process dependencies in parallel? If A depends on B and C 1.2 and B depends on C 1.4 There is no way to know to use C 1.4 until it got to B and it won't know to use B until it got to A. /cc @rsc <https://github.com/rsc> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#24316 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABF9l3mwE0xYs09JToC1NR6hrH7JwO_4ks5tcd66gaJpZM4SjgTw> .
Sorry about deleting it, shortly after posting I realized roughly the same thing you posted.
Also based on https://research.swtch.com/vgo-module
So maybe it could (not sure if it does already) just get all the mod files and build the dependency tree and then fetch all the source files completely in parallel. But yeah this wouldn't work with legacy packages so it have to revert to getting the entire package to figure out the dependency tree.