Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: parallelize go get downloads #11821

jacobsa opened this Issue Jul 22, 2015 · 4 comments


None yet
5 participants
Copy link

jacobsa commented Jul 22, 2015

Running go get -u for an app with a decent number of dependencies can be extremely slow from a location with high latency to GitHub (and other code hosts, too -- it's even worse when there are redirects involved). For example, from my machine in Sydney the following command takes almost 90 seconds to download 18 repos:

go get -v -u -t

This is in stark contrast to general use of the go command, which is otherwise pleasantly fast.

I notice that downloads happen serially. It seems like go get could benefit greatly from doing them concurrently. Are there any major obstacles to doing so?


This comment has been minimized.

Copy link

rsc commented Jul 22, 2015

The main obstacle is that you have to fetch A to find out that it imports B and C.
But we could parallelize B and C.

@rsc rsc changed the title cmd/go: serial downloads can be very slow cmd/go: parallelize go get downloads Jul 22, 2015

@rsc rsc added this to the Go1.6Early milestone Jul 22, 2015


This comment has been minimized.

Copy link

nightlyone commented Jul 23, 2015

There are also rate-limit issues.

e.g. common setup is to use a shared ssh session to github (sort of a connection pool via ControlPersist yes option) and there are only 10 concurrent connections allowed over that one (hard coded limit in openssh deamon).

So this feature would need a bunch of tuneables.

A different approach to target the increasing complexity of go get and vastly different usage profiles might be a git like plugin architecture for the whole vcs part.

@rsc rsc modified the milestones: Unplanned, Go1.6Early Nov 4, 2015


This comment has been minimized.

Copy link

powerman commented Sep 25, 2017

Is this correct way (in case GOPATH contains exactly one element) to speedup go get -v -t -u all?

find $GOPATH/src -type d -name .git -printf "%h\0" |
    xargs -0 -P64 -I{} sh -c "cd {}; git pull; git submodule update"
go get -v -t all

I suppose it may miss some corner cases like default branch change on GitHub, but I'm not sure is go get -u handle such cases too.


This comment has been minimized.

Copy link

bcmills commented Jan 23, 2019

Module downloads are performed in parallel. I don't think that retrofitting GOPATH mode to do so would be worth the effort.

@bcmills bcmills closed this Jan 23, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.