proposal: cmd/go: vendor: only copy files that the Go build system recognizes

[ianlancetaylor]

Proposal Details

Currently go mod vendor copies all files in a package except:

• files whose name ends with _test.go
• files named go.mod and go.sum
• files whose names ends with .go and contain a go:build ignore constraint

This means that other random files are copied into the vendored directory. For example, the vendored golang.org/x/sys/unix in the main repo includes mkall.sh, mkerrors.sh, and README.md. There is no reason to vendor those files.

I propose that we change go mod vendor to only copy files that go build recognizes. This is files that end with .go, .c, .cxx, .swig, and so forth. Making this change would also require go mod vendor to look for go:embed directives and copy over all files that they mention (for example, the vendored files in GOROOT/src/cmd/vendor/github.com/google/pprof/internal/driver/html).

[seankhliao]

given #26366, isn't it just a bug that cmd/go copies more than necessary now?

[ianlancetaylor]

Maybe? I wrote this as a proposal because it's clearly a user-visible change. But if we want to just call the current behavior a bug, that is fine with me.

CC @matloob

[bjorndm]

On the other hand, already we have to do all sorts of tricks to force the build system to include certain C and H files. Restricting this more will be a problem. There should also be a way to tell go build that a file is actually necessary. See for example this: https://github.com/go-gl/glfw/blob/master/v3.3/glfw/build_cgo_hack.go

[matloob]

cc @samthanawalla

I think there are some subtleties here that make it more than just a bug. For instance, in the linked issue, #26366 (comment) there's a user who wants to make sure that the LICENSE files are copied in the vendor directory.

[bjorndm]

That too is important. Some open source license require to be distributed together with the code even when vendoring. Some pragma like //go:vendor [filename] would be useful for that.

[ianlancetaylor]

@bjorndm I'm suggesting that we copy all files that the Go build system recognizes, which includes all files with extensions .c and .h.

@matloob We already copy all files that start with metaPrefixes, including LICENSE; I don't think that should change.

[bjorndm]

@ianlancetaylor I think the main problem of go vendor, like for glfw-go is with subdirectories that contain only C and H files without Go files. It would be nice if that could also be supported without hacks.

[matloob]

@ianlancetaylor Ah, got it.

[ianlancetaylor]

@bjorndm Are you saying that that does not work today? That seems like a separate issue from this one.

[bjorndm]

@ianlancetaylor The file I linked above is an example of the hack that is necessary to get go mod vendor to keep C and H files in subdirectories. I'm here to make sure that this hack keeps on working at least. But you are right, not needing this hack is a separate issue.