Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: clarify best practice for tool dependencies #25922

Closed
myitcv opened this Issue Jun 16, 2018 · 36 comments

Comments

Projects
None yet
@myitcv
Copy link
Member

myitcv commented Jun 16, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10.3 linux/amd64
vgo commit 22e23900224f03be49670113d5781e4d89090f45

Does this issue reproduce with the latest release?

Yes; and latest vgo commit (per above)

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN="/tmp/tmp.VQw1O3x8Wy/hello/bin"
GOCACHE="/home/myitcv/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/tmp/tmp.VQw1O3x8Wy"
GORACE=""
GOROOT="/home/myitcv/gos"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build414355570=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Off the back of #25624 (comment), I'd like to confirm that the following represents the "best practice" advice when adding and installing tool dependencies:

cd `mktemp -d`
export GOPATH=$PWD

mkdir hello
cd hello
vgo mod -init -module example.com/hello

# this could be anywhere but for convenience...
export GOBIN=$PWD/bin

# add a dependency on golang.org/x/tools/cmd/stringer
cat <<EOD > tools.go
// +build tools

package tools

import (
        _ "golang.org/x/tools/cmd/stringer"
)
EOD

vgo install golang.org/x/tools/cmd/stringer

The go.mod and .Target for stringer look fine:

$ cat go.mod
module example.com/hello

require golang.org/x/tools v0.0.0-20180615195736-465e6f399236
$ vgo list -f "{{.Target}}" golang.org/x/tools/cmd/stringer
/tmp/tmp.VQw1O3x8Wy/hello/bin/stringer

The issue however is that running vgo mod -sync then removes our module requirement on golang.org/x/tools - I suspect this is a bug:

$ vgo mod -json
{
        "Module": {
                "Path": "example.com/hello",
                "Version": ""
        },
        "Require": [
                {
                        "Path": "golang.org/x/tools",
                        "Version": "v0.0.0-20180615195736-465e6f399236"
                }
        ],
        "Exclude": null,
        "Replace": null
}
$ vgo mod -sync
warning: "ALL" matched no packages
$ vgo mod -json
{
        "Module": {
                "Path": "example.com/hello",
                "Version": ""
        },
        "Require": null,
        "Exclude": null,
        "Replace": null
}

If we assume this is a bug and ignore it for now, I also wonder whether we can improve this workflow for adding tool dependencies somehow. The following steps feel a bit "boilerplate" and unnecessary:

  • define a fake tools.go file
  • add a dependency on the tool (which will never compile because it's a main package, so I'm not sure we can ever safely verify tools.go is "good"?)
  • set GOBIN to an appropriate location
  • vgo install tool
  • ensure PATH includes GOBIN

I wonder whether we could in fact obviate all of this by having something like:

vgo install -tool golang.org/x/tools/cmd/stringer
vgo run golang.org/x/tools/cmd/stringer

Thoughts?

vgo run tool is possible as a result of #22726, but because of #25416 it effectively requires a link step each time.

What did you expect to see?

With respect to what I think is a bug with vgo mod -sync

go.mod unchanged by the vgo mod -sync

What did you see instead?

The golang.org/x/tools requirement removed.

/cc @rsc @bcmills

@gopherbot gopherbot added this to the vgo milestone Jun 16, 2018

@myitcv myitcv modified the milestones: vgo, Go1.11 Jun 16, 2018

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Jun 19, 2018

Let's take a step back: what do you mean by “tool dependency”?

The stringer example suggests that you mean “a tool required for use with go generate”, but note that generate is intended to be run by the author of the module, not its users: a tool needed to run generate shouldn't be a hard requirement for the module or its tests.

@myitcv

This comment has been minimized.

Copy link
Member Author

myitcv commented Jun 19, 2018

Let's take a step back: what do you mean by “tool dependency”?

Roughly:

  • I'm writing a module; or, more generally, me and my team are writing a module
  • we want to use a tool (e.g. stringer) whilst writing that code
  • we want to reproducibly re-generate code, or put another way, we want the whole team to consistently generate the same code at any given point in time
  • hence I want to ensure that everyone is using the same version of any tool we require

but note that generate is intended to be run by the author of our module, not its users: a tool needed to run generate shouldn't be a hard requirement for the module or its tests.

Correct, this would only be a dependency for the authors of the module (as you point out, the users of our module probably don't care whether or not we've used stringer or other tools).

As I understand it, these tool dependencies do not introduce any additional requirements for users of our module: they (the tool dependencies) can only ever be referenced by running vgo install or vgo run in the context of our module. So as a user of our module, I would, by definition be in a different context (specifically the module that references our module) when running vgo install or vgo run.

I'm using the term "context" here to mean "the go.mod that vgo uses." I'm sure there must be a better, more precise term

@jimmyfrasche

This comment has been minimized.

Copy link
Member

jimmyfrasche commented Jun 19, 2018

I haven't played around with vgo much yet or looked under the hood.

I know a lot of package managers handle this by sorting dependencies into buckets: regular dependencies, test dependencies, dev dependencies. That seems like overkill and there can also be dependencies by build tag, like, say, importing a module for internal use but only on windows, and dependencies may only be for a certain package in a module that you may not be using.

Could vgo record all the dependencies for a project—for all build tags, for all packages, for tests, for development—but lazily fetch them as needed? (This may require more invasive integration but it would be an optimization as the semantics are the same for lazy-fetch and fetch-everything-upfront)

@myitcv

This comment has been minimized.

Copy link
Member Author

myitcv commented Jun 20, 2018

@jimmyfrasche

Could vgo record all the dependencies for a project—for all build tags, for all packages, for tests, for development—but lazily fetch them as needed?

My understanding is that this is exactly what vgo does today, for any dependency, test/tool or otherwise.

And, to your earlier point, I agree it means the delineation between the buckets is unnecessary (for anyone wanting to know why a dependency exists, vgo can tell us that answer).

Just to clarify one point above. My proposal for vgo install -tool golang.org/x/tools/cmd/stringer would, however, add a section to go.mod (or equivalent) where the tool packages are listed. This is simply to avoid the need for the tools.go file referenced in my example.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Jun 20, 2018

A couple of observations:

  1. Some tools required by Go builds are not themselves built using the go tool. (For example, Google's protocol compiler is written in C++; only the Go code generator plugin is written in Go.) So even if we add support for tooling to the go tool, it will be at best a partial solution.

  2. If you are assuming the use of the go tool anyway, then you can write a Go program that invokes it with appropriate arguments.

  3. Because a command-line tool is a top-level package, its versions are not context-sensitive: if you're not mixing in other packages to customize the tool, you don't need to resolve version constraints from those packages.

To me, those suggest a fairly straightforward solution: go run pkg@version. In combination with //go:generate, that would allow you to express the dependency (at the exact version used) directly in the source file:

//go:generate go run golang.org/x/tools/cmd/stringer@v1.11.0 -type=Pill

In contrast, if you are building a custom version of the tool, then you should be able to customize it using blank-imports as you describe.

@jimmyfrasche

This comment has been minimized.

Copy link
Member

jimmyfrasche commented Jun 20, 2018

@bcmills that sounds good but it would be awkward to write that each time and easy for things to go out of sync and you have some code using 1.11.0 and some using 1.12.3 but probably want .

go generate let's you define aliases with //go:generate -command alias the-command (I've not used it personally and unsure of the scope: file? package? (I assume file.))

Perhaps a more vgo-aware variant would be helpful, like

//go:generate -require golang.org/x/tools/cmd/stringer@v1.11.0

which is module scoped. It falls somewhere between an ephemeral go install -tool and a fictitious blank import.

There are a some details to iron out in such an approach, of course.

@myitcv

This comment has been minimized.

Copy link
Member Author

myitcv commented Jun 21, 2018

@bcmills

Some tools required by Go builds are not themselves built using the go tool.

Completely. As you say, this can either be solved entirely orthogonally, or with Go wrappers (wrappers that could even take care of installing the right version).

Because a command-line tool is a top-level package, its versions are not context-sensitive: if you're not mixing in other packages to customize the tool, you don't need to resolve version constraints from those packages.

I'm not entirely sure I follow your point here. The point I was trying to make is that where (i.e. in what directory) you run vgo install golang.org/x/tools/cmd/stringer is relevant because that will determine which go.mod you reference and therefore which version of the golang.org/x/tools module you use.

To me, those suggest a fairly straightforward solution: go run pkg@version.

As @jimmyfrasche pointed out this has the major downside of littering the required version across all the places you require go:generate directives, which seems to be somewhat counter to having go.mod be the single point of reference for versions.

Just one point to note about the "proposal":

vgo install -tool golang.org/x/tools/cmd/stringer
vgo run golang.org/x/tools/cmd/stringer

Despite the explanation in #25416, I think vgo run golang.org/x/tools/cmd/stringer could actually cache the linked binary, because the cache can very precisely be trimmed given we know exactly what version is being referenced at a given point in time.

@jimmyfrasche

This comment has been minimized.

Copy link
Member

jimmyfrasche commented Jun 21, 2018

Running a tool at a specific version depending on the module would have to go through vgo.

Running stringer lets $SHELL pick which stringer to run, so it would need to be something like go run-dev-tool stringer so that vgo could dispatch the correct version of the command.

Something like that could be stuffed into go run but that doesn't seem to fit with the spirit of that command. Arguably go tool would be a better fit but that could cause namespacing issues so it seems like a separate command built to purpose would be best.

go generate could handle this transparently but not all tools are necessarily for code generation. You may also want a specific version of, say, a linter.

An IDE/editor could use tool dependencies to figure out if this is a tool dependency or a regular program and run it through vgo if need be. From the command line, though, it would be up to the user to know whether to run stringer vs go run-dev-tool stringer so there would still be potential skew with the version documented. I don't think there's much to do about that, though.

A tool module could have multiple commands. Look up would be slow if these weren't included somewhere in the metadata.

A tool module could have multiple dependencies, some shared with the module under development. Unlike with other dependencies, you probably don't want those to be solved together: using foobar shouldn't affect your version baz or the version of baz used by other tools you depend on. They should all be siloed.

This is all starting to sound mostly unrelated to vgo but perhaps it's something that can be built on top of vgo.

Going back on what I said about buckets, let's say there was as separate tool godev and a separate file dev.mod.

godev install foo updates dev.mod recording the version and the exposed commands. Instead of installing the command globally it installs the binaries outside of $PATH but somewhere it can easily grab the correct version. It updates dev.mod with the vgo style import and version path but also records the names of the commands. For the most part it's a wrapper around go get that does some extra work.

godev run foo invokes the appropriate foo from dev.mod.

Is there anything that would need to change in vgo to allow something like this?

A downside of not having it integrated in vgo itself would be that go:generate directives would need to explicitly prefix godev run.

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Jul 6, 2018

What is the decision to be made here?
I think the tools.go file is in fact the best practice for tool dependencies, certainly for Go 1.11.
I like it because it does not introduce new mechanisms.
It simply reuses existing ones.

@myitcv

This comment has been minimized.

Copy link
Member Author

myitcv commented Jul 6, 2018

I think the tools.go file is in fact the best practice for tool dependencies, certainly for Go 1.11.

Indeed, we don't need to "solve" this now.

What is the decision to be made here?

For Go 1.11

  • is the vgo mod -sync behaviour observed in the description a bug or not? It feels like it is.

For later:

  • is there a better way to ensure the fake tools.go is valid? Because it will never compile. We could do something with vgo list and a bit of shell, but it feels like we're side stepping the issue
  • is there a way to reduce the "boilerplate" setup described in the issue, to make the experience for module maintainers nicer?

@rsc rsc modified the milestones: vgo, Go1.11 Jul 12, 2018

@rsc rsc added the modules label Jul 12, 2018

@rsc rsc changed the title x/vgo: clarify best practice for tool dependencies cmd/go: clarify best practice for tool dependencies Jul 12, 2018

@myitcv

This comment has been minimized.

Copy link
Member Author

myitcv commented Aug 14, 2018

Adding another comment to this thread to give the equivalent, and current, go commands

This is a go-based rewrite of #25922 (comment)

cd $(mktemp -d)
mkdir hello
cd hello
go mod init example.com/hello

# Either rely on GOBIN=GOPATH/bin or set it explicitly
export GOBIN=$PWD/bin

# add a dependency on golang.org/x/tools/cmd/stringer
# the build constraint ensures this file is ignored
cat <<EOD > tools.go
// +build tools

package tools

import (
        _ "golang.org/x/tools/cmd/stringer"
)
EOD

go install golang.org/x/tools/cmd/stringer

cat go.mod

results in:

module example.com/hello

require golang.org/x/tools v0.0.0-20180813205110-a434f64ace81 // indirect
@joeshaw

This comment has been minimized.

Copy link
Contributor

joeshaw commented Aug 15, 2018

Similarly, converting an existing project from dep to the modules system and running go mod vendor results in the stringer tool and its immediate dependencies being removed from the vendor directory.

Sorry, this is actually working. My tools.go file was missing the package directive at the top of the file and because it's never built the compiler never got a chance to complain. This, ironically, is one of @myitcv's concerns in his initial comment.

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Aug 17, 2018

Best practice remains creating the tools.go file.
It's true that go mod init does not auto-create a tools.go from dep's config,
but I think doing so is getting a bit beyond scope.

@rsc rsc modified the milestones: Go1.11, Go1.12 Aug 17, 2018

@AlekSi

This comment has been minimized.

Copy link
Contributor

AlekSi commented Aug 21, 2018

Probably it's worth to highlight why tools.go should contain some build tag like tools (but not ignore): to avoid side-effects of init() functions in tool dependencies.

@thepudds

This comment has been minimized.

Copy link

thepudds commented Aug 24, 2018

A related comment: as far as I am aware, creating a tools.go as described above in #25922 (comment) seems to work with go mod vendor for Go-based tool dependencies.

In addition, there is also this separate comment in #26244 (comment) :

vgo vendor (now vgo mod -vendor or go mod -vendor) does not copy entire repositories. By design, it copies only the directories needed to satisfy imports in builds of the top-level module. If your module is not importing github.com/antlr/grammars-v4 then that directory will not be copied. And if there are directories with only non-Go files, there is no way to copy those directories into vendor. That's not what vendor is for. Vendor is only about preserving the code needed to build the main module

...which on the one hand seems to say go mod vendor is not intended to cover all use cases around non-Go pieces of a repository...

...but that comment also seems to suggest that if there is a piece of Go code that will be vendored, the operation seems to function at the granularity of directories (e.g., it "copies only the directories needed to satisfy imports in builds of the top-level module").

In practice, this currently seems to mean that non-Go tool dependencies (such as a shell script) are copied into the vendor directory by go mod vendor if they are in the same directory as a vendored piece of Go code. So a tools.go-based approach could in theory be extended to pulling in non-Go based tool dependencies (which would happen naturally if the non-Go tools are intermixed at the directory level with Go code, or perhaps even by adding something like a vendorme.go to a particular directory if needed).

However, as far as I can see, the go mod vendor documentation currently does not say it operates at the directory level, so perhaps that is behavior that should not be relied upon, or perhaps this would otherwise be considered an undesirable approach.

EDIT: note that a fair amount of care is required if you are pulling in additional directories beyond what go mod vendor does naturally, and this is less about vendoring behavior and more about following the go tool's build model. See for example comment from Russ in #26366 (comment), as well as https://golang.org/cl/125297, and #26366 (comment))

@caarlos0

This comment has been minimized.

Copy link

caarlos0 commented Aug 29, 2018

I've tried the solution proposed by @myitcv, it more or less works, seems like stringer can't solve the dependencies in the package that has the //go:generate:

~/Code/goreleaser/goreleaser vgo*
λ go list -f "{{.Target}}" golang.org/x/tools/cmd/stringer
/Users/carlos/Code/goreleaser/goreleaser/bin/stringer

~/Code/goreleaser/goreleaser vgo*
λ go generate ./...
stringer: checking package: artifact.go:11:2: could not import github.com/apex/log (type-checking package "github.com/apex/log" failed (/Users/carlos/go/pkg/mod/github.com/apex/log@v0.0.0-20180702155952-941dea75d3eb/stack.go:3:8: could not import github.com/pkg/errors (cannot find package "github.com/pkg/errors" in any of:
	/usr/local/Cellar/go/1.11/libexec/src/github.com/pkg/errors (from $GOROOT)
	/Users/carlos/go/src/github.com/pkg/errors (from $GOPATH))))
internal/artifact/artifact.go:15: running "stringer": exit status 1

Not sure if it is a bug in stringer or somewhere else, happy to provide more info if needed though.


Not sure if useful, but:

@izumin5210

This comment has been minimized.

Copy link

izumin5210 commented Dec 10, 2018

I've also tried to implement a tools.go-based approach as a tiny go mod / dep command.
https://github.com/izumin5210/gex

I used it in some projects(e.g. wraperr, grapi, ...).
And I found some cons of this approach:

go.mod (Gopkg.toml) contains tool dependencies.
When a project depends on another package that has tools.go , the project's go.mod also contains tools.
If this approach is implemented in go mod, we will think that we should split a manifest file(e.g. go.mod and go-tools.mod).

aswild added a commit to aswild/wolssh that referenced this issue Dec 31, 2018

add bin2go infrastructure
I'll use my bin2go tool to embed a default config file. This commit just
figures out how to import and build it locally.

There's no canonical way to handle this, especially with modules, but
the best option seems to be creating a dummy tools.go with imports so
that "go mod tidy" doesn't remove the dependency.

See golang/go#25922
@tsheaff

This comment has been minimized.

Copy link

tsheaff commented Jan 1, 2019

I made a related issue here: #29494 In my case it's about a hot reloader tool for development. Some parts of this discussion seem to assume this is only about code generators like stringer, but it applies to linters, hot reloader tools, or any other go module that I don't want compiled up into the application executable, but instead want to use from the command line before the application starts (or even the tool that does the starting-up of the application)

I like the go install -tool option as a UI and the go.tools file suggested by @rogpeppe — that seems explicit and clear.

@tjsampson

This comment has been minimized.

Copy link

tjsampson commented Jan 2, 2019

We are running into a similar issue. In our Org, we aren't allowed to download dependencies from the internet during CI. Historically this would require us to Fork/Mirror a repo and bring it in house. But with go modules, we can just vendor the deps and check them into source control. That way, our CI servers aren't trying to pull packages/code from github/golang.org/google.golang.org etc.

This approach works perfectly for packages that are actually being used in the code, but for build/tooling packages, it falls a little short. I really like what @tsheaff and @rogpeppe laid out with go install -tool and the go.tools file. Being able to explicitly declare a package as a tool/build dependency would be very nice. And ideally, that code would not be a part of the final built binary, just used during the build/test/cover steps (i.e. CI/CD process).

@andig

This comment has been minimized.

Copy link

andig commented Jan 3, 2019

Best practice remains creating the tools.go file.

I've failed to get that working:

tools.go

// +build tools

package sdm630

import (
	_ "github.com/mjibson/esc"
)

go.mod

module github.com/gonium/gosdm630

require (
	github.com/mjibson/esc v0.1.0 // indirect
)

being used by

//go:generate esc -private -o assets.go -pkg sdm630 -prefix assets assets

results in

❯ go generate
http.go:21: running "esc": exec: "esc": executable file not found in $PATH
@myitcv

This comment has been minimized.

Copy link
Member Author

myitcv commented Jan 3, 2019

@andig you need to install the tool first, per https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md.

@tjsampson - you can use the vendor directory for tool dependencies too, but you need to tell the go tool to use the vendor when installing the tool. So in the link I provided above, instead of go install your.com/tool you would do go install -mod=vendor your.com/tool.

@tsheaff - there is nothing to prevent any arbitrary tool from being added as a dependency via the mechanism described here. I have a file watcher added as one of my tools for example. The approach is the same.

If however you definitely want to "globally" install a tool, there is support for that in the beta release of Go 1.12 (added in response to #24250) or you could consider github.com/myitcv/gobin (example)

@andig

This comment has been minimized.

Copy link

andig commented Jan 3, 2019

@andig you need to install the tool first, per https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md.

I'm probably missing something here, but when using build-only dependencies without runtime use like https://github.com/mjibson/esc or stringer, what is the point of go.mod'ing them if they still need to be installed? Only selecting the specific version?

@myitcv

This comment has been minimized.

Copy link
Member Author

myitcv commented Jan 3, 2019

Only selecting the specific version?

Correct.

To avoid installing you can modify your //go:generate directive to something like:

//go:generate go run golang.org/x/tools/cmd/stringer ARGS

or else for faster execution use gobin:

//go:generate gobin -m -run golang.org/x/tools/cmd/stringer ARGS

(the reason for using gobin over go run is covered in the gobin wiki)

aswild added a commit to aswild/wolssh that referenced this issue Jan 5, 2019

add bin2go infrastructure
I'll use my bin2go tool to embed a default config file. This commit just
figures out how to import and build it locally.

There's no canonical way to handle this, especially with modules, but
the best option seems to be creating a dummy tools.go with imports so
that "go mod tidy" doesn't remove the dependency.

See golang/go#25922
@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Jan 18, 2019

I notice that the tools.go approach is now documented at http://golang.org/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module.

@bcomnes

This comment has been minimized.

Copy link

bcomnes commented Feb 24, 2019

go.mod (Gopkg.toml) contains tool dependencies.
When a project depends on another package that has tools.go , the project's go.mod also contains tools.
If this approach is implemented in go mod, we will think that we should split a manifest file(e.g. go.mod and go-tools.mod).

@izumin5210 is there an open issue for this? I couldn't fine one.

EDIT: looks like myitcv/gobin#44 contains some discussion

@prymitive

This comment has been minimized.

Copy link

prymitive commented Feb 26, 2019

tools.go pattern worked for me too, but I've noticed that go mod tidy removes all dependencies recorded in tools.go from go.mod & go.sum, which doesn't seem to be mentioned in this thread.

@tv42

This comment has been minimized.

Copy link

tv42 commented Feb 26, 2019

@prymitive Just tested and that does not happen for me, on Go 1.12

@prymitive

This comment has been minimized.

Copy link

prymitive commented Mar 7, 2019

@tv42 sorry, my bad, I had an incorrectly formatted comment in my tools.go so it wasn't working and because it had // +build tools I didn't get any compile / mod tidy errors. Fixing the comment fixed tidying.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.