New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cmd/go: go mod -vendor prunes non-package directories #26366
Comments
The cause here seems to be that we prune out directories that are not Go packages. There is no |
Side note from govendor. By default govendor only picks up pkgs referenced, but I had to add an option to grab the whole sub tree for this very reason. Mostly cgo deps, but sometimes other required resources. ... Right now I'm considering making a tool that can be run after go mod -vendor to trim unused packages. But I can't leave what isn't there. |
I wonder whether in the world of modules what some/most people are actually intending to do in this situation (namely where they think they want/need to So then a That said, I don't use |
I don't think anyone actually wants to vendor everything. There are two problems here, that I think should not be mixed:
|
One way to solve the cgo problem is to require a dummy go file in the c folder that is then imported by another package that uses the c files with a "_" import. I think this is only a problem because the c and h files live in a separate directory from the go files. |
Sorry, but the model for go builds is that what's needed to build a package is in that directory, not subdirectories. The way gen2brain/aac-go is structured, modifying the C source code in the subdirectories will not trigger a rebuild. Instead your build will use stale cache entries. The C code needs to be moved up to the package directory to make it build properly, not to mention vendor properly. If anyone wants to build a tool that adds to the vendor directory after |
@rsc I see two opportunities for improvement here. Firstly, should we explicitly clarify in the cgo documentation that the go tool notices only changes in Go packages? Right now the wording doesn't mention other directories and local includes at all. Something like
(Emphasis = added by me.) Secondly, should such local includes be a vet warning? There is already |
Change https://golang.org/cl/125297 mentions this issue: |
… bad idea Suggested in discussion on #26366. Change-Id: Id9ad2e429a915f88b4c4b30fc415c722eebe0ea4 Reviewed-on: https://go-review.googlesource.com/125297 Reviewed-by: Ian Lance Taylor <iant@golang.org>
this tool might be helpful to others: https://github.com/goware/modvendor — I wrote it to easily copy .c, .h, .s, .proto files from module pkgs into my local ./vendor. Feel free to improve :) |
|
I've just added a "proposal" under #27618 (not marked as a proposal because the modules stuff is experimental, so more a discussion issue than anything). Apologies for the |
I can't believe the proposed solution to this issue is to write another tool to finish what |
@nomad-software Russ outlined a response to this in #26366 (comment). If instead you want to "vendor" the module that will give you everything. That's the subject of #27618 and a proposed approach is outlined at https://github.com/myitcv/go-modules-by-example/blob/master/012_modvendor/README.md. |
@nomad-software this issue here was initially reported when the original reporter was using cgo. Are also hitting this with cgo, vs. are you hitting this without using cgo? If you are hitting this with cgo, and if we set aside vendoring for a moment, do you think you might be hitting problems even just during non-vendor-based building/recompiling due to the behavior described here:
(It is worth reading the full description at that link, but wanted to include at least a snippet for convenience) |
@im-kulikov I've written a tool to fully copy the entire directory of all dependencies into the vendor folder. You can find it here: https://github.com/nomad-software/vend Any suggestions to improve it are welcome, just open an issue there. |
this still an issue, its crazy how we still need to use |
We use go mod vendor with a CGo module that references .C and .H files stored in its subdirectory, and they are not copied by the tool. Perhaps, tool could support an option in go.mod file to list modules that should be fully copied, instead of stripped? |
@hrissan Partially re-using an older comment: If you are hitting this with cgo, and if we set aside vendoring for a moment, there is a decent chance you are also having a more fundamental problem even just during non-vendor-based building/recompiling due to the behavior described here:
Does that seem like it applies to you? |
We have a legal requirement to add the license file of some packages to their vendor directory. Unfortunately just copying go files and not having an option for other files can become a serious issue. |
This change moves dcgm headers into the pkg/dcgm directory. Go vendoring does not copy non-go files from subdirectories/sibling directories if there are no go files there. See golang/go#26366 for reference. Fixes NVIDIA/dcgm-exporter#21 Signed-off-by: Nik Konyuchenko <spaun2002mobile@gmail.com>
This change moves dcgm headers into the pkg/dcgm directory. Go vendoring does not copy non-go files from subdirectories/sibling directories if there are no go files there. See golang/go#26366 for reference. Fixes NVIDIA/dcgm-exporter#21 Signed-off-by: Nik Konyuchenko <spaun2002mobile@gmail.com>
Any chance we can restart this discussion? A flag to declare assets for a package would make sense? I guess something similar happens when using the |
Looking #26366 (comment), it seems that using a tree of C files will not work correctly. But I still have to use code from a library that is organized as a tree. So I need to flatten that tree as files in my package directory. Has anyone built a tool that will take a C file and resolve and inline all |
There is a rule for go build: all sources files of a given package must be in the directory of that package. Not in subdirectories. Which means that {#include "hidapi/mac/hid.c"} is not correct as hid.c is not in the package directory, so "go build" will not notice its changes and go mod -vendor will not work (golang/go#26366). So this patch creates 3 files (hidapi_linux.h, hidapi_windows.h, hidapi_mac.h) which are go-generated by internal/gen.go. The generator just inlines any includes from the libusb/ and hidapi/ directories. This means that the libusb/ and hidapi/ are not needed anymore for distribution (and for go mod -vendor): they are used only for the "go generate" phase.
There is a rule for go build: all sources files of a given package must be in the directory of that package. Not in subdirectories. Which means that {#include "hidapi/mac/hid.c"} is not correct as hid.c is not in the package directory, so "go build" will not notice its changes and go mod -vendor will not work (golang/go#26366). So this patch creates 3 files (hidapi_linux.h, hidapi_windows.h, hidapi_mac.h) which are go-generated by internal/gen.go. The generator just inlines any includes from the libusb/ and hidapi/ directories. This means that the libusb/ and hidapi/ are not needed anymore for distribution (and for go mod -vendor): they are used only for the "go generate" phase.
To allow 'go mod -vendor' to take directories, we create dummy packages in each directory. Basically it does: for d in $(find internal -type d); do echo "package $(basename $d)" >"$d/pkg.go"; done A script is used to maintained those dummy package besides hidapi/libusb upgrades: internal/dummy-packages.sh See: - karalabe#31 - golang/go#26366
Here is a workaround that may be useful for other projects: For package github.com/dolmen-go/hid (which embeds C libraries |
All, I'm a packager on Gentoo, and I'm hitting this with https://git.sr.ht/~rjarry/aerc.git. I am required to vendor to build packages under our package manager, and it doesn't look like that is going to change any time soon. Our package manager team sees allowing me to reach out to the network during a build as a massive security hole they don't want to open, and they tell me that having builds that are required to reach out to the network breaks building on unconnected systems. So, what is the best way to deal with this issue? Thanks, William |
I have packaged the vend tool from @nomad-software on Gentoo, thanks for writing it. IMO this is a pretty significant issue since go mod vendor creates vendor directories that are unusable in some situations. We will use vend for now, but please count me in as someone requesting that this discussion be restarted. Thanks much for your time, William |
@williamh |
Remove psx_pthread_create() from libpsx - given the way -lpsx is linked this is not needed. Also, as pointed out by Lorenz Bauer, "go mod vendor" support was unable to vendor a copy of psx_syscall.h because it didn't reside in the same directory as the *.go code for the psx package. (General discussion golang/go#26366 .) Given that we can, avoid the use of a sub-directory in the libcap tree. Signed-off-by: Andrew G. Morgan <morgan@kernel.org>
Why is this closed? This is still a major issue. Go dep was able to handle this with "unused-packages = false" which means many cannot and will not switch over to go mod. |
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
What operating system and processor architecture are you using (
go env
)?What did you do?
What did you expect to see?
Successful build.
What did you see instead?
The problem is that
go mod -vendor
pruned theexternal/aacenc
folder, which contains the C code needed to build this package.The text was updated successfully, but these errors were encountered: