Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: get / mod: document pruning repos for "module source tree too big" error #32297

Open
paulquerna-okta opened this issue May 29, 2019 · 8 comments

Comments

Projects
None yet
4 participants
@paulquerna-okta
Copy link

commented May 29, 2019

What version of Go are you using (go version)?

go version go1.12.2 darwin/amd64

What did you do?

Hello Gophers. We are attempting to migrate several large proprietary Go ecosystems to using Go Modules. Previously we have used glide and would like to be on track with where the community is going.

When attempting to migrate repo alpha that depends on beta, we received the module source tree too big error. We have already migrated beta to Go modules in a development branch, which is the sha reference in the go get bellow.

This error also prevents commands like go mod vendor from working, and is seemingly hidden from showing up as an error there - it treats the module source tree too big error to imply the module import path is invalid, so some commands "kinda" work, but then we get later failures because the files weren't actually placed in vendor/.

But, you can see the direct error when attempting a go get of the repo:

$ go get github.com/okta/beta@157700ca4b7c2b57e46cde2a531ed3889833349a
go: finding github.com/okta/beta 157700ca4b7c2b57e46cde2a531ed3889833349a
go: downloading github.com/okta/beta v1.39.2
package github.com/okta/beta: unknown import path "github.com/okta/beta": module source tree too big
$ cd beta
$ du -sh .
1020M	.
$ du -sh .git
368M	.git

What did you expect to see?

go mod vendor and go get to work with large repos. Alternatively a flag or environment variable available to increase the limit would be viable for our environment.

What did you see instead?

The module source tree too big error.

Previous reports:
#25470
#29210

In the previous report, #29210 (comment) "Folks can file separate issues if they discover other repositories that don't fit." is the last comment, so here we are.

@agnivade

This comment has been minimized.

Copy link
Member

commented May 29, 2019

Can you provide more details as asked by @bcmills in #29210 ?

What are the file types that contribute to the repo size ? Can you provide a breakdown ? Is all of it coming entirely from .go source files which are required to build it ? If not, have you tried pruning them from being included in the module by placing another go.mod in the right directory ?

@paulquerna-okta

This comment has been minimized.

Copy link
Author

commented Jun 4, 2019

We were able to create go.mod files in a couple sub-directories to get us under the current limit. If the limit is going to stay, is there a place that can document this practice of splicing repos with creating a new go module under a directory? It seems to be the only solution being advised?

@agnivade agnivade removed the WaitingForInfo label Jun 5, 2019

@agnivade

This comment has been minimized.

Copy link
Member

commented Jun 5, 2019

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented Jun 5, 2019

@paulquerna-okta Does the repo just contain source files and test data needed by go build and go test? Are there any large files that can be removed in tagged versions, as there were in #29210? So far, 500MB has been enough space for buildable code, but we'd like to understand if that's appropriate.

The purpose of the limit is to avoid making users download large artifacts (zip files) with files that aren't actually needed. Files that aren't part of the build (large resources, vendor directories, etc) may be removed in tagged versions to get below this limit. These files can usually still be on development branches though.

@paulquerna-okta

This comment has been minimized.

Copy link
Author

commented Jun 5, 2019

We cannot just remove these files from "tagged" versions in git -- we are tagging a whole software project which combines multiple programming languages. You are right as far as the Go compiler / modules are concerned, there are many files that are not relevant. It's pretty common pattern to have a .ignore file, like .gitignore or .dockerignore when you want an ecosystem of tools to ignore things. Has an idea like that been explored for creating the Go Modules zip files?

This is really an intersection of Go Modules and two mono-repositories. Mono-repos like this are less common in open source, so I'm less sure this is an issue about users downloading large files. We're on the path to breaking up the mono-repos, but kinda ironically that path was to first get to Go Modules tooling to make it easier.

@bcmills

This comment has been minimized.

Copy link
Member

commented Jun 5, 2019

I agree that removing the files from tagged versions is not really a viable option.

We've been using go.mod files as the equivalent of .ignore, and at this point that would be very difficult to change: any change in the extracted contents of a module also changes its hash, and we currently have no way to vary that based on the version of the go tool (see #30369).

Right now the supported mechanism is to add go.mod files at the root of any directories that should be pruned out of the Go module. (It should be fine if they're empty, as long as they exist.)

@agnivade

This comment has been minimized.

Copy link
Member

commented Jun 5, 2019

@bcmills - Does it make sense to note this trick in the wiki ? Seems useful. Or is it already documented somewhere ?

@bcmills

This comment has been minimized.

Copy link
Member

commented Jun 5, 2019

@thepudds might know. I don't see it today.

@bcmills bcmills added this to the Go1.14 milestone Jun 5, 2019

@agnivade agnivade changed the title cmd/go: get / mod: "module source tree too big" cmd/go: get / mod: document pruning repos for "module source tree too big" error Jun 15, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.