Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go mod -vendor prunes non-package directories #26366

Closed
ainar-g opened this issue Jul 13, 2018 · 56 comments
Closed

cmd/go: go mod -vendor prunes non-package directories #26366

ainar-g opened this issue Jul 13, 2018 · 56 comments
Labels
modules NeedsInvestigation

Comments

@ainar-g
Copy link
Contributor

@ainar-g ainar-g commented Jul 13, 2018

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

go version devel +8a33045 Fri Jul 13 03:53:00 2018 +0000 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ainar/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ainar/go"
GOPROXY=""
GORACE=""
GOROOT="/home/ainar/go/gotip"
GOTMPDIR=""
GOTOOLDIR="/home/ainar/go/gotip/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build865981061=/tmp/go-build -gno-record-gcc-switches"

What did you do?

#!/bin/sh
set -e
export GO111MODULE=on
mkdir foo
cd foo
cat <<EOF > main.go
package foo // import "foo"

import "github.com/gen2brain/aac-go/aacenc"

func f() { _ = aacenc.VoPidAacMdoule }
EOF
$GO mod -init
$GO mod -vendor
$GO build -getmode vendor

What did you expect to see?

Successful build.

What did you see instead?

# github.com/gen2brain/aac-go/aacenc
vendor/github.com/gen2brain/aac-go/aacenc/aacenc.go:4:19: fatal error: voAAC.h: No such file or directory
 //#include "voAAC.h"
                   ^
compilation terminated.

The problem is that go mod -vendor pruned the external/aacenc folder, which contains the C code needed to build this package.

@myitcv myitcv added NeedsInvestigation modules labels Jul 13, 2018
@bcmills
Copy link
Member

@bcmills bcmills commented Jul 13, 2018

The cause here seems to be that we prune out directories that are not Go packages.

There is no voAAC.h in github.com/gen2brain/aac-go/aacenc/, but I do see one in github.com/gen2brain/aac-go/aacenc/external/aacenc/include/.

@bcmills bcmills changed the title cmd/go: go mod -vendor prunes C files cmd/go: go mod -vendor prunes non-package directories Jul 13, 2018
@kardianos
Copy link
Contributor

@kardianos kardianos commented Jul 14, 2018

Side note from govendor. By default govendor only picks up pkgs referenced, but I had to add an option to grab the whole sub tree for this very reason.

Mostly cgo deps, but sometimes other required resources.

...

Right now I'm considering making a tool that can be run after go mod -vendor to trim unused packages. But I can't leave what isn't there.

@myitcv
Copy link
Member

@myitcv myitcv commented Jul 14, 2018

I wonder whether in the world of modules what some/most people are actually intending to do in this situation (namely where they think they want/need to go mod -vendor) is snapshot the current module dependencies as opposed to the packages. Doing so would seem to address all of these points?

So then a go mod -modvendor (command/flags to be 🚲-shed-ed) would create a directory (say ./modvendor) that could then be used by GOPROXY=file:///path/to/modvendor.

That said, I don't use go mod -vendor and likely never will. But I do maintain an append-only cache of the modules that my CI then uses via GOPROXY=file://...

@ainar-g
Copy link
Contributor Author

@ainar-g ainar-g commented Jul 14, 2018

I don't think anyone actually wants to vendor everything. There are two problems here, that I think should not be mixed:

  1. Non-Go code dependencies vendoring. This is what this issue is about. Whether we like it or not, Go has CGo. And C dependencies are hard. Some approaches to solve this problem (excluding the painful choice of teaching the go tool about C dependencies... the horror):

    • Let module authors declare their non-Go code dependencies in their go.mod. Something like

      external "github.com/gen2brain/aac-go/aacenc/external/aacenc"
      
    • Define and document a convention for such dependencies (e.g. external directory, similar how we have conventions for internal and testdata).

    • Explicitly give up and document that CGo is fundamentally incompatible with go mod -vendor. After all, even if you do have the C code in your vendor, there are no guarantees that that C code doesn't have some dependencies of its own, so hoping that /vendor will save you from build failures is rather naïve.

  2. Binary tool dependency vendoring. This may or may not be a problem Go modules want to solve. @rsc said it in #24051.

    It is not a goal to expand into "linux distribution" territory.

    But then this, too, should be documented explicitly. It would be a shame though, as AFAIK other systems, like Ruby's Bundler and JS's NPM support some version of it.

@kardianos
Copy link
Contributor

@kardianos kardianos commented Jul 14, 2018

One way to solve the cgo problem is to require a dummy go file in the c folder that is then imported by another package that uses the c files with a "_" import.

I think this is only a problem because the c and h files live in a separate directory from the go files.

@rsc
Copy link
Contributor

@rsc rsc commented Jul 17, 2018

Sorry, but the model for go builds is that what's needed to build a package is in that directory, not subdirectories. The way gen2brain/aac-go is structured, modifying the C source code in the subdirectories will not trigger a rebuild. Instead your build will use stale cache entries. The C code needs to be moved up to the package directory to make it build properly, not to mention vendor properly.

If anyone wants to build a tool that adds to the vendor directory after go mod -vendor, you could do that by reading the modules.txt file and using commands like go list -f '{{.Dir}}' to find out where each package was copied from.

@rsc rsc closed this as completed Jul 17, 2018
@ainar-g
Copy link
Contributor Author

@ainar-g ainar-g commented Jul 17, 2018

@rsc I see two opportunities for improvement here. Firstly, should we explicitly clarify in the cgo documentation that the go tool notices only changes in Go packages? Right now the wording doesn't mention other directories and local includes at all. Something like

<...> Any .h, .hh, .hpp, or .hxx files will not be compiled separately, but, if these header files are changed, the C and C++ files will be recompiled. Changes in files included from other directories will not trigger a rebuild. <...>

(Emphasis = added by me.)

Secondly, should such local includes be a vet warning? There is already -cgocall, why not -cgoinclude?

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 20, 2018

Change https://golang.org/cl/125297 mentions this issue: cmd/cgo: document that #including source files in subdirectories is a bad idea

gopherbot pushed a commit that referenced this issue Jul 28, 2018
… bad idea

Suggested in discussion on #26366.

Change-Id: Id9ad2e429a915f88b4c4b30fc415c722eebe0ea4
Reviewed-on: https://go-review.googlesource.com/125297
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@pkieltyka
Copy link

@pkieltyka pkieltyka commented Aug 27, 2018

this tool might be helpful to others: https://github.com/goware/modvendor — I wrote it to easily copy .c, .h, .s, .proto files from module pkgs into my local ./vendor. Feel free to improve :)

@im-kulikov
Copy link
Contributor

@im-kulikov im-kulikov commented Sep 10, 2018

go mod vendor drops github.com/ethereum/go-ethereum/crypto/secp256k1/libsecp256k1 files 😔

@myitcv
Copy link
Member

@myitcv myitcv commented Sep 11, 2018

I've just added a "proposal" under #27618 (not marked as a proposal because the modules stuff is experimental, so more a discussion issue than anything). Apologies for the modvendor name clash @pkieltyka, I just spotted that.

@nomad-software
Copy link

@nomad-software nomad-software commented Sep 24, 2018

I can't believe the proposed solution to this issue is to write another tool to finish what go mod vendor starts. Either write a tool to replace it or fix it so it copies everything. What is the reason for only vendoring half of the repository?

@myitcv
Copy link
Member

@myitcv myitcv commented Sep 24, 2018

@nomad-software Russ outlined a response to this in #26366 (comment).

If instead you want to "vendor" the module that will give you everything. That's the subject of #27618 and a proposed approach is outlined at https://github.com/myitcv/go-modules-by-example/blob/master/012_modvendor/README.md.

@thepudds
Copy link

@thepudds thepudds commented Sep 25, 2018

@nomad-software this issue here was initially reported when the original reporter was using cgo.

Are also hitting this with cgo, vs. are you hitting this without using cgo?

If you are hitting this with cgo, and if we set aside vendoring for a moment, do you think you might be hitting problems even just during non-vendor-based building/recompiling due to the behavior described here:

https://golang.org/cmd/cgo/

When the Go tool sees that one or more Go files use the special import "C", it will look for other non-Go files in the directory and compile them as part of the Go package.
...
Note that changes to files in other directories do not cause the package to be recompiled, so all non-Go source code for the package should be stored in the package directory, not in subdirectories.

(It is worth reading the full description at that link, but wanted to include at least a snippet for convenience)

@nomad-software
Copy link

@nomad-software nomad-software commented Oct 22, 2018

go mod vendor drops github.com/ethereum/go-ethereum/crypto/secp256k1/libsecp256k1 files

@im-kulikov I've written a tool to fully copy the entire directory of all dependencies into the vendor folder. You can find it here:

https://github.com/nomad-software/vend

Any suggestions to improve it are welcome, just open an issue there.

@thomasmodeneis
Copy link

@thomasmodeneis thomasmodeneis commented Jun 5, 2021

this still an issue, its crazy how we still need to use vend to solve the package issue....

@hrissan
Copy link

@hrissan hrissan commented Jun 6, 2021

We use go mod vendor with a CGo module that references .C and .H files stored in its subdirectory, and they are not copied by the tool.

Perhaps, tool could support an option in go.mod file to list modules that should be fully copied, instead of stripped?

@thepudds
Copy link

@thepudds thepudds commented Jun 6, 2021

@hrissan Partially re-using an older comment:

If you are hitting this with cgo, and if we set aside vendoring for a moment, there is a decent chance you are also having a more fundamental problem even just during non-vendor-based building/recompiling due to the behavior described here:

https://golang.org/cmd/cgo/

When the Go tool sees that one or more Go files use the special import "C", it will look for other non-Go files in the directory and compile them as part of the Go package.
...
Note that changes to files in other directories do not cause the package to be recompiled, so all non-Go source code for the package should be stored in the package directory, not in subdirectories.

Does that seem like it applies to you?

@rezaalavi
Copy link

@rezaalavi rezaalavi commented Aug 18, 2021

We have a legal requirement to add the license file of some packages to their vendor directory. Unfortunately just copying go files and not having an option for other files can become a serious issue.

nikkon-dev added a commit to NVIDIA/go-dcgm that referenced this issue Oct 20, 2021
This change moves dcgm headers into the pkg/dcgm directory.
Go vendoring does not copy non-go files from subdirectories/sibling directories if there are no go files there. See golang/go#26366 for reference.

Fixes NVIDIA/dcgm-exporter#21

Signed-off-by: Nik Konyuchenko <spaun2002mobile@gmail.com>
nikkon-dev added a commit to NVIDIA/go-dcgm that referenced this issue Oct 20, 2021
This change moves dcgm headers into the pkg/dcgm directory.
Go vendoring does not copy non-go files from subdirectories/sibling directories if there are no go files there. See golang/go#26366 for reference.

Fixes NVIDIA/dcgm-exporter#21

Signed-off-by: Nik Konyuchenko <spaun2002mobile@gmail.com>
@jcchavezs
Copy link

@jcchavezs jcchavezs commented Dec 1, 2021

Any chance we can restart this discussion? A flag to declare assets for a package would make sense? I guess something similar happens when using the embed annotation.

@dolmen
Copy link
Contributor

@dolmen dolmen commented Feb 20, 2022

Looking #26366 (comment), it seems that using a tree of C files will not work correctly.

But I still have to use code from a library that is organized as a tree. So I need to flatten that tree as files in my package directory.

Has anyone built a tool that will take a C file and resolve and inline all #include that are relative to a base directory? This is a subset of what a preprocessor does...

dolmen added a commit to dolmen-go/hid that referenced this issue Feb 20, 2022
There is a rule for go build: all sources files of a given package must
be in the directory of that package. Not in subdirectories.
Which means that {#include "hidapi/mac/hid.c"} is not correct as hid.c
is not in the package directory, so "go build" will not notice its
changes and go mod -vendor will not work (golang/go#26366).

So this patch creates 3 files (hidapi_linux.h, hidapi_windows.h,
hidapi_mac.h) which are go-generated by internal/gen.go. The generator
just inlines any includes from the libusb/ and hidapi/ directories.
This means that the libusb/ and hidapi/ are not needed anymore for
distribution (and for go mod -vendor): they are used only for the "go
generate" phase.
dolmen added a commit to dolmen-go/hid that referenced this issue Feb 20, 2022
There is a rule for go build: all sources files of a given package must
be in the directory of that package. Not in subdirectories.
Which means that {#include "hidapi/mac/hid.c"} is not correct as hid.c
is not in the package directory, so "go build" will not notice its
changes and go mod -vendor will not work (golang/go#26366).

So this patch creates 3 files (hidapi_linux.h, hidapi_windows.h,
hidapi_mac.h) which are go-generated by internal/gen.go. The generator
just inlines any includes from the libusb/ and hidapi/ directories.
This means that the libusb/ and hidapi/ are not needed anymore for
distribution (and for go mod -vendor): they are used only for the "go
generate" phase.
dolmen added a commit to dolmen-go/hid that referenced this issue Feb 20, 2022
To allow 'go mod -vendor' to take directories, we create dummy packages
in each directory.

Basically it does:
for d in $(find internal -type d); do echo "package $(basename $d)" >"$d/pkg.go"; done

A script is used to maintained those dummy package besides hidapi/libusb
upgrades: internal/dummy-packages.sh

See:
- karalabe#31
- golang/go#26366
@dolmen
Copy link
Contributor

@dolmen dolmen commented Feb 20, 2022

Here is a workaround that may be useful for other projects:

For package github.com/dolmen-go/hid (which embeds C libraries libusb and hidapi) I've implemented an ad-hoc C preprocessor (triggered by go generate) that concatenates all required C files from the subdirectories and produces 3 .h files (one for each supported platform). The cgo source just includes the the local .h file corresponding to GOOS.

@williamh
Copy link

@williamh williamh commented Feb 27, 2022

All,

I'm a packager on Gentoo, and I'm hitting this with https://git.sr.ht/~rjarry/aerc.git.

I am required to vendor to build packages under our package manager, and it doesn't look like that is going to change any time soon. Our package manager team sees allowing me to reach out to the network during a build as a massive security hole they don't want to open, and they tell me that having builds that are required to reach out to the network breaks building on unconnected systems.

So, what is the best way to deal with this issue?

Thanks,

William

@williamh
Copy link

@williamh williamh commented Feb 27, 2022

I have packaged the vend tool from @nomad-software on Gentoo, thanks for writing it.

IMO this is a pretty significant issue since go mod vendor creates vendor directories that are unusable in some situations.

We will use vend for now, but please count me in as someone requesting that this discussion be restarted.

Thanks much for your time,

William

@xuyang2
Copy link

@xuyang2 xuyang2 commented Mar 30, 2022

chenyt9 pushed a commit to MotorolaMobilityLLC/external-libcap that referenced this issue May 6, 2022
Remove psx_pthread_create() from libpsx - given the way -lpsx is
linked this is not needed.

Also, as pointed out by Lorenz Bauer, "go mod vendor" support
was unable to vendor a copy of psx_syscall.h because it didn't
reside in the same directory as the *.go code for the psx package.
(General discussion golang/go#26366 .)
Given that we can, avoid the use of a sub-directory in the libcap
tree.

Signed-off-by: Andrew G. Morgan <morgan@kernel.org>
@acestronautical
Copy link

@acestronautical acestronautical commented May 11, 2022

Why is this closed? This is still a major issue. Go dep was able to handle this with "unused-packages = false" which means many cannot and will not switch over to go mod.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
modules NeedsInvestigation
Projects
None yet
Development

No branches or pull requests