Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/vgo: lookup/fetch dependencies concurrently #24316

Closed
awly opened this issue Mar 8, 2018 · 7 comments
Closed

x/vgo: lookup/fetch dependencies concurrently #24316

awly opened this issue Mar 8, 2018 · 7 comments

Comments

@awly
Copy link
Contributor

@awly awly commented Mar 8, 2018

Ran vgo build for program that imports a large dependency tree (imports k8s.io/kubernetes/something).

It worked for several minutes, printing lines like this at a steady pace:
vgo: finding github.com/kr/pretty v0.0.0-20140723054909-088c856450c0

It looks like vgo works through list dependencies sequentially (recursively appending additional ones as they are discovered). Perhaps this could be parallellized?

@gopherbot gopherbot added this to the vgo milestone Mar 8, 2018
@awly

This comment has been minimized.

Copy link
Contributor Author

@awly awly commented Mar 9, 2018

@awly

This comment has been minimized.

Copy link
Contributor Author

@awly awly commented Mar 9, 2018

Hmm, got an email about someone commenting but perhaps they deleted the comment.
Anyway, the above description still stands.

@AlexRouSg

This comment has been minimized.

Copy link
Contributor

@AlexRouSg AlexRouSg commented Mar 9, 2018

Sorry about deleting it, shortly after posting I realized roughly the same thing you posted.

Also based on https://research.swtch.com/vgo-module
It defines a url to get the go.mod file
GET baseURL/module/@v/version.mod fetches the go.mod file for that version.

So maybe it could (not sure if it does already) just get all the mod files and build the dependency tree and then fetch all the source files completely in parallel. But yeah this wouldn't work with legacy packages so it have to revert to getting the entire package to figure out the dependency tree.

@ALTree

This comment has been minimized.

Copy link
Member

@ALTree ALTree commented Mar 9, 2018

Same request but for go get for reference: #11821.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Mar 27, 2018

Once we are happy with the rest of the code, yes, we should parallelize it. But I want to figure out what the code should do and make it correct before we make it fast.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Apr 9, 2018

Brad asked me to kill uses of NeedsFixButNotYet, so I'm going to call this NeedsInvestigation.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jun 15, 2018

Change https://golang.org/cl/119055 mentions this issue: cmd/go: parallelize module load operations

@golang golang locked and limited conversation to collaborators Jun 15, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
5 participants
You can’t perform that action at this time.