New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: parallelize go get downloads #11821

Open
jacobsa opened this Issue Jul 22, 2015 · 3 comments

Comments

Projects
None yet
4 participants
@jacobsa
Contributor

jacobsa commented Jul 22, 2015

Running go get -u for an app with a decent number of dependencies can be extremely slow from a location with high latency to GitHub (and other code hosts, too -- it's even worse when there are redirects involved). For example, from my machine in Sydney the following command takes almost 90 seconds to download 18 repos:

go get -v -u -t github.com/googlecloudplatform/gcsfuse/...

This is in stark contrast to general use of the go command, which is otherwise pleasantly fast.

I notice that downloads happen serially. It seems like go get could benefit greatly from doing them concurrently. Are there any major obstacles to doing so?

@rsc

This comment has been minimized.

Contributor

rsc commented Jul 22, 2015

The main obstacle is that you have to fetch A to find out that it imports B and C.
But we could parallelize B and C.

@rsc rsc changed the title from cmd/go: serial downloads can be very slow to cmd/go: parallelize go get downloads Jul 22, 2015

@rsc rsc added this to the Go1.6Early milestone Jul 22, 2015

@nightlyone

This comment has been minimized.

Contributor

nightlyone commented Jul 23, 2015

There are also rate-limit issues.

e.g. common setup is to use a shared ssh session to github (sort of a connection pool via ControlPersist yes option) and there are only 10 concurrent connections allowed over that one (hard coded limit in openssh deamon).

So this feature would need a bunch of tuneables.

A different approach to target the increasing complexity of go get and vastly different usage profiles might be a git like plugin architecture for the whole vcs part.

@rsc rsc modified the milestones: Unplanned, Go1.6Early Nov 4, 2015

@powerman

This comment has been minimized.

powerman commented Sep 25, 2017

Is this correct way (in case GOPATH contains exactly one element) to speedup go get -v -t -u all?

find $GOPATH/src -type d -name .git -printf "%h\0" |
    xargs -0 -P64 -I{} sh -c "cd {}; git pull; git submodule update"
go get -v -t all

I suppose it may miss some corner cases like default branch change on GitHub, but I'm not sure is go get -u handle such cases too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment