cmd/go: go mod -vendor prunes non-package directories

[ainar-g]

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

go version devel +8a33045 Fri Jul 13 03:53:00 2018 +0000 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ainar/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ainar/go"
GOPROXY=""
GORACE=""
GOROOT="/home/ainar/go/gotip"
GOTMPDIR=""
GOTOOLDIR="/home/ainar/go/gotip/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build865981061=/tmp/go-build -gno-record-gcc-switches"

What did you do?

#!/bin/sh
set -e
export GO111MODULE=on
mkdir foo
cd foo
cat <<EOF > main.go
package foo // import "foo"

import "github.com/gen2brain/aac-go/aacenc"

func f() { _ = aacenc.VoPidAacMdoule }
EOF
$GO mod -init
$GO mod -vendor
$GO build -getmode vendor

What did you expect to see?

Successful build.

What did you see instead?

# github.com/gen2brain/aac-go/aacenc
vendor/github.com/gen2brain/aac-go/aacenc/aacenc.go:4:19: fatal error: voAAC.h: No such file or directory
 //#include "voAAC.h"
                   ^
compilation terminated.

The problem is that go mod -vendor pruned the external/aacenc folder, which contains the C code needed to build this package.

[bcmills]

The cause here seems to be that we prune out directories that are not Go packages.

There is no voAAC.h in github.com/gen2brain/aac-go/aacenc/, but I do see one in github.com/gen2brain/aac-go/aacenc/external/aacenc/include/.

[kardianos]

Side note from govendor. By default govendor only picks up pkgs referenced, but I had to add an option to grab the whole sub tree for this very reason.

Mostly cgo deps, but sometimes other required resources.

...

Right now I'm considering making a tool that can be run after go mod -vendor to trim unused packages. But I can't leave what isn't there.

[myitcv]

I wonder whether in the world of modules what some/most people are actually intending to do in this situation (namely where they think they want/need to go mod -vendor) is snapshot the current module dependencies as opposed to the packages. Doing so would seem to address all of these points?

So then a go mod -modvendor (command/flags to be 🚲-shed-ed) would create a directory (say ./modvendor) that could then be used by GOPROXY=file:///path/to/modvendor.

That said, I don't use go mod -vendor and likely never will. But I do maintain an append-only cache of the modules that my CI then uses via GOPROXY=file://...

[ainar-g]

I don't think anyone actually wants to vendor everything. There are two problems here, that I think should not be mixed:

1. Non-Go code dependencies vendoring. This is what this issue is about. Whether we like it or not, Go has CGo. And C dependencies are hard. Some approaches to solve this problem (excluding the painful choice of teaching the go tool about C dependencies... the horror):

   • Let module authors declare their non-Go code dependencies in their go.mod. Something like

     external "github.com/gen2brain/aac-go/aacenc/external/aacenc"
     

   • Define and document a convention for such dependencies (e.g. external directory, similar how we have conventions for internal and testdata).

   • Explicitly give up and document that CGo is fundamentally incompatible with go mod -vendor. After all, even if you do have the C code in your vendor, there are no guarantees that that C code doesn't have some dependencies of its own, so hoping that /vendor will save you from build failures is rather naïve.

2. Binary tool dependency vendoring. This may or may not be a problem Go modules want to solve. @rsc said it in #24051.

   It is not a goal to expand into "linux distribution" territory.

   But then this, too, should be documented explicitly. It would be a shame though, as AFAIK other systems, like Ruby's Bundler and JS's NPM support some version of it.

[kardianos]

One way to solve the cgo problem is to require a dummy go file in the c folder that is then imported by another package that uses the c files with a "_" import.

I think this is only a problem because the c and h files live in a separate directory from the go files.

[rsc]

Sorry, but the model for go builds is that what's needed to build a package is in that directory, not subdirectories. The way gen2brain/aac-go is structured, modifying the C source code in the subdirectories will not trigger a rebuild. Instead your build will use stale cache entries. The C code needs to be moved up to the package directory to make it build properly, not to mention vendor properly.

If anyone wants to build a tool that adds to the vendor directory after go mod -vendor, you could do that by reading the modules.txt file and using commands like go list -f '{{.Dir}}' to find out where each package was copied from.

[ainar-g]

@rsc I see two opportunities for improvement here. Firstly, should we explicitly clarify in the cgo documentation that the go tool notices only changes in Go packages? Right now the wording doesn't mention other directories and local includes at all. Something like

<...> Any .h, .hh, .hpp, or .hxx files will not be compiled separately, but, if these header files are changed, the C and C++ files will be recompiled. Changes in files included from other directories will not trigger a rebuild. <...>

(Emphasis = added by me.)

Secondly, should such local includes be a vet warning? There is already -cgocall, why not -cgoinclude?

[gopherbot]

Change https://golang.org/cl/125297 mentions this issue: cmd/cgo: document that #including source files in subdirectories is a bad idea

[pkieltyka]

this tool might be helpful to others: https://github.com/goware/modvendor — I wrote it to easily copy .c, .h, .s, .proto files from module pkgs into my local ./vendor. Feel free to improve :)

[im-kulikov]

go mod vendor drops github.com/ethereum/go-ethereum/crypto/secp256k1/libsecp256k1 files 😔

[myitcv]

I've just added a "proposal" under #27618 (not marked as a proposal because the modules stuff is experimental, so more a discussion issue than anything). Apologies for the modvendor name clash @pkieltyka, I just spotted that.

[nomad-software]

I can't believe the proposed solution to this issue is to write another tool to finish what go mod vendor starts. Either write a tool to replace it or fix it so it copies everything. What is the reason for only vendoring half of the repository?

[myitcv]

@nomad-software Russ outlined a response to this in #26366 (comment).

If instead you want to "vendor" the module that will give you everything. That's the subject of #27618 and a proposed approach is outlined at https://github.com/myitcv/go-modules-by-example/blob/master/012_modvendor/README.md.

[thepudds]

@nomad-software this issue here was initially reported when the original reporter was using cgo.

Are also hitting this with cgo, vs. are you hitting this without using cgo?

If you are hitting this with cgo, and if we set aside vendoring for a moment, do you think you might be hitting problems even just during non-vendor-based building/recompiling due to the behavior described here:

https://golang.org/cmd/cgo/

When the Go tool sees that one or more Go files use the special import "C", it will look for other non-Go files in the directory and compile them as part of the Go package.
...
Note that changes to files in other directories do not cause the package to be recompiled, so all non-Go source code for the package should be stored in the package directory, not in subdirectories.

(It is worth reading the full description at that link, but wanted to include at least a snippet for convenience)

[nomad-software]

go mod vendor drops github.com/ethereum/go-ethereum/crypto/secp256k1/libsecp256k1 files

@im-kulikov I've written a tool to fully copy the entire directory of all dependencies into the vendor folder. You can find it here:

https://github.com/nomad-software/vend

Any suggestions to improve it are welcome, just open an issue there.

[thomasmodeneis]

this still an issue, its crazy how we still need to use vend to solve the package issue....

[hrissan]

We use go mod vendor with a CGo module that references .C and .H files stored in its subdirectory, and they are not copied by the tool.

Perhaps, tool could support an option in go.mod file to list modules that should be fully copied, instead of stripped?

[thepudds]

@hrissan Partially re-using an older comment:

If you are hitting this with cgo, and if we set aside vendoring for a moment, there is a decent chance you are also having a more fundamental problem even just during non-vendor-based building/recompiling due to the behavior described here:

https://golang.org/cmd/cgo/

When the Go tool sees that one or more Go files use the special import "C", it will look for other non-Go files in the directory and compile them as part of the Go package.
...
Note that changes to files in other directories do not cause the package to be recompiled, so all non-Go source code for the package should be stored in the package directory, not in subdirectories.

Does that seem like it applies to you?

[rezaalavi]

We have a legal requirement to add the license file of some packages to their vendor directory. Unfortunately just copying go files and not having an option for other files can become a serious issue.

[jcchavezs]

Any chance we can restart this discussion? A flag to declare assets for a package would make sense? I guess something similar happens when using the embed annotation.

[dolmen]

Looking #26366 (comment), it seems that using a tree of C files will not work correctly.

But I still have to use code from a library that is organized as a tree. So I need to flatten that tree as files in my package directory.

Has anyone built a tool that will take a C file and resolve and inline all #include that are relative to a base directory? This is a subset of what a preprocessor does...

[dolmen]

Here is a workaround that may be useful for other projects:

For package github.com/dolmen-go/hid (which embeds C libraries libusb and hidapi) I've implemented an ad-hoc C preprocessor (triggered by go generate) that concatenates all required C files from the subdirectories and produces 3 .h files (one for each supported platform). The cgo source just includes the the local .h file corresponding to GOOS.

[williamh]

All,

I'm a packager on Gentoo, and I'm hitting this with https://git.sr.ht/~rjarry/aerc.git.

I am required to vendor to build packages under our package manager, and it doesn't look like that is going to change any time soon. Our package manager team sees allowing me to reach out to the network during a build as a massive security hole they don't want to open, and they tell me that having builds that are required to reach out to the network breaks building on unconnected systems.

So, what is the best way to deal with this issue?

Thanks,

William

[williamh]

I have packaged the vend tool from @nomad-software on Gentoo, thanks for writing it.

IMO this is a pretty significant issue since go mod vendor creates vendor directories that are unusable in some situations.

We will use vend for now, but please count me in as someone requesting that this discussion be restarted.

Thanks much for your time,

William

[xuyang2]

@williamh
Is it acceptable to install athens as a private proxy for Go modules?
In this way, only athens needs to access the internet.

• https://github.com/gomods/athens
• https://gomods.io/
• https://docs.gomods.io/intro/protocol/
• https://help.sonatype.com/repomanager3/nexus-repository-administration/formats/go-repositories
• https://jfrog.com/blog/why-goproxy-matters-and-which-to-pick/

[acestronautical]

Why is this closed? This is still a major issue. Go dep was able to handle this with "unused-packages = false" which means many cannot and will not switch over to go mod.