Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go get fails when fetching large repo #27646

Closed
ryandbump opened this issue Sep 12, 2018 · 6 comments

Comments

Projects
None yet
6 participants
@ryandbump
Copy link

commented Sep 12, 2018

What version of Go are you using (go version)?

go version go1.11 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/ryanbump/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/ryanbump/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/ryanbump/code/devolate/test-oversize/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/fn/lp9kz_ms6k353f9ml2j49rlw0000gn/T/go-build776699926=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I created a repo github.com/ryandbump/test-oversize to recreate an issue I ran into when using go get. Thetest-oversize repo has 600M of junk txt files in it to increase it's size. There are also a few branches (junk-580M, junk-560M, etc.) that have different amounts of these junk text files to allow for testing with variable size repo.

In a module aware project run the following steps to reproduce.

  • run go get github.com/ryandbump/test-oversize
  • run go get for each of the different size branches.

What did you expect to see?

I expected to see the package downloaded and cached for use with my project.

What did you see instead?

An error was raised that implied there was a GOPATH issue despite using go get in a module aware context. The command would only succeed once the repo was below 500M in size.

$ go get github.com/ryandbump/test-oversize
go: finding github.com/ryandbump/test-oversize latest
go: downloading github.com/ryandbump/test-oversize v0.0.0-20180912214615-996567b1f368
go get: no install location for directory  outside GOPATH
	For more details see: 'go help gopath'
@agnivade

This comment has been minimized.

Copy link
Member

commented Sep 13, 2018

@agnivade agnivade added this to the Go1.12 milestone Sep 13, 2018

@myitcv

This comment has been minimized.

Copy link
Member

commented Sep 13, 2018

@ryandbump please can you report the output with -x added?

@bcmills bcmills added the modules label Sep 13, 2018

@ryandbump

This comment has been minimized.

Copy link
Author

commented Sep 14, 2018

$ go get -x github.com/ryandbump/test-oversize
# /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869 for git2 https://github.com/ryandbump/test-oversize
cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git ls-remote -q https://github.com/ryandbump/test-oversize
0.296s # cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git ls-remote -q https://github.com/ryandbump/test-oversize
go: finding github.com/ryandbump/test-oversize latest
cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 996567b1f368c1a266b531d0dbbdf78423702b06
0.027s # cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 996567b1f368c1a266b531d0dbbdf78423702b06
cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git cat-file blob 996567b1f368c1a266b531d0dbbdf78423702b06:go.mod
0.019s # cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git cat-file blob 996567b1f368c1a266b531d0dbbdf78423702b06:go.mod
cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git tag -l
0.016s # cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git tag -l
go: downloading github.com/ryandbump/test-oversize v0.0.0-20180912214615-996567b1f368
cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 996567b1f368
0.011s # cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 996567b1f368
cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git cat-file blob 996567b1f368c1a266b531d0dbbdf78423702b06:go.mod
0.011s # cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git cat-file blob 996567b1f368c1a266b531d0dbbdf78423702b06:go.mod
cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ 996567b1f368c1a266b531d0dbbdf78423702b06
3.225s # cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ 996567b1f368c1a266b531d0dbbdf78423702b06
go get: no install location for directory  outside GOPATH
	For more details see: 'go help gopath'
@hyangah

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2018

MaxZipFile = 500 << 20 // maximum size of downloaded zip file

The error message is not very helpful. It seems to be reached the internal limit.

@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 23, 2018

I created a repo github.com/ryandbump/test-oversize to recreate an issue I ran into when using go get.

What is the real issue you ran into? (Did the Go source files and testdata for the module overrun the 500MiB limit, or was our git command producing a zip archive containing irrelevant data?)

@bcmills bcmills modified the milestones: Go1.12, Go1.13 Nov 13, 2018

@gopherbot

This comment has been minimized.

Copy link

commented Nov 23, 2018

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this Nov 23, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.