Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: do not download “modules” that contain no Go files #31866

Open
bcmills opened this issue May 6, 2019 · 6 comments

Comments

@bcmills
Copy link
Member

commented May 6, 2019

At the moment, go mod download will happily try to extract and download any arbitrary repository as long as it can be resolved by some means (through a hard-coded hosting service such as github.com, or using a distinguished extension like .git), even if it does not contain anything even marginally related to building Go code.

I am not aware of any reasonable use-case for such a repository:

  • It's not useful for storing test data, because we currently provide no mechanism for the tests to actually locate that data. (Modules are not guaranteed to be loaded from the module cache — for example, they might be subject to a replace directive — and since the test itself is run within the directory containing its source code, it has no way to locate the data or run go list within the module that invoked it.)

  • It's not useful for C headers (for use with cgo), for the same reason.

  • It might be useful for fetching non-Go inputs to go generate: in theory, the generator could run go mod download $MODULE to locate the sources at the required version. But the output of go generate is intended to be checked in anyway, which makes the use of modules somewhat spurious: if an explicit version of the non-Go inputs appears in the module's requirements, then everyone using the generated package will have an extra module to fetch that is guaranteed to have no effect on the build, and in most cases the go generate program can just as easily git clone (or similar) the input data at a specific revision.

Furthermore, if someone did find a way to make modules without Go source code useful (for the above use-cases or others), it's trivial to add a go.mod file to indicate that the repository really is somehow intended for use with Go source code. (We need to support go.mod-only modules anyway, since they can arise naturally when splitting a large root module into smaller nested modules.)


On the other hand, module proxies tend to rely on the go command to decide what is or is not a valid module, and accepting arbitrary non-Go repositories potentially exposes such proxies to a significant amount of additional load.


Therefore, I propose that we change the go command to explicitly reject any “module” that both contains no .go source files and lacks a go.mod file.

CC @rsc @jayconrod @heschik @hyangah @katiehockman @thepudds @marwan-at-work @ianthehat

@bcmills bcmills added this to the Go1.14 milestone May 6, 2019

@marwan-at-work

This comment has been minimized.

Copy link
Contributor

commented May 6, 2019

Sounds good to me as long as the Go command only checks for .go files and does not validate the import paths when a proxy runs go mod download $MODULE

For more on why: #31458

@bcmills

This comment has been minimized.

Copy link
Member Author

commented May 6, 2019

@marwan-at-work

as long as the Go command only checks for .go files and does not validate the import paths

For that do you mean something like #31662?

@marwan-at-work

This comment has been minimized.

Copy link
Contributor

commented May 6, 2019

@bcmills yes that issue would guarantee a proxy to go mod download a module via its VCS path and not its Vanity import path. I believe this already works as expected for my use case

Thanks!

@vearutop

This comment has been minimized.

Copy link
Contributor

commented May 7, 2019

Another example use case is to deliver supporting scripts and assets with module. Not saying it is the greatest idea, but I included base Makefile from go/pkg/mod/path@ver/Makefile.

Such approach was working with vendor before.

I agree for such case it should be not a problem to at least add go.mod in the repo root.

@beoran

This comment has been minimized.

Copy link

commented May 7, 2019

Actually in the proxy we am writing, we download the repository using the VCS, not using go get or go download. So, I don't know whether the repository is a useful go module or not, I let the proxy user's go decide on that. So actually, a go module without go files doesn't bother me at all. I'd rather not have to decide in the proxy whether a repository contains go files or not, so I am not in favor of this proposal.

@Xe

This comment has been minimized.

Copy link

commented May 30, 2019

3.1 GB of data in go/pkg/mod

Got bit by this today while trying to find out why my disk space was gone. The biggest offender seems to be the linux kernel tree. Like the actual upstream linux kernel somehow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.