cmd/go: track tool dependencies in go.mod #48429

mtibben · 2021-09-17T03:33:03Z

UPDATE: 2024-07-29: the latest proposal can be found here.

Background

The current best-practice to track tool dependencies for a module is to add a tools.go file to your module that includes import statements for the tools of interest. This has been extensively discussed in #25922 and is the recommended approach in the Modules FAQ

This approach works, but managing the tool dependencies still feels like a missing piece in the go mod toolchain. For example, the instructions for getting a user set up with a new project using gqlgen (a codegen tool) looks like this

# Initialise a new go module
mkdir example
cd example
go mod init example

# Add gqlgen as a tool
printf '// +build tools\npackage tools\nimport _ "github.com/99designs/gqlgen"' | gofmt > tools.go
go mod tidy

# Initialise gqlgen config and generate models
go run github.com/99designs/gqlgen init

The printf line above really stands out as an arbitrary command to "add a tool" and reflects a poor developer experience when managing tools. For example, an immediate problem is that the printf line will only work on unix systems and not windows. And what happens if tools.go already exists?

So while we have some excellent tools for managing dependencies within the go.mod file using go get and go mod edit, there is no such equivalent for managing tools in the tools.go file.

Proposed Solution

The go.mod file uses the // indirect comment to track some dependencies. An // indirect comment indicates that no package from the required module is directly imported by any package in the main module (source).

I propose that this same mechanism be used to add tool dependencies, using a // tool comment.

Users could add a tool with something like

go get -tool github.com/99designs/gqlgen@v0.14.0

or

go mod edit -require=github.com/99designs/gqlgen -tool

A go.mod would then look something like

module example

go 1.17

require (
	github.com/99designs/gqlgen v0.14.0 // tool
)

And would allow users to subsequently run the tool with go run github.com/99designs/gqlgen

This would mean a separate tools.go file would no longer be required as the tool dependency is tracked in the go.mod file.

Go modules would be "tool" aware. For example:

go mod tidy would not remove the // tool dependency, even though it is not referenced directly in the module
Perhaps if a module with a // tool dependency is imported by another module, Go modules understands that the // tool dependency is not required as an indirect dependency. Currently when using tools.go, go modules does not have that context and the tool is treated like any other indirect dependency
go get -tool [packages] would only add a dependency with a main package

The text was updated successfully, but these errors were encountered:

fsouza · 2021-09-17T03:58:34Z

I like this, I find it annoying to use the tools.go solution, though I'll admit I don't have a better complaint than it being annoying/weird.

If this proposal moves forward, where does the dependency go in the go.mod file? (assuming the 1.17 format with multiple require blocks). Will it have a dedicated block for tools? Or are tools treated like // indirect and placed in the same block?

ianlancetaylor · 2021-09-17T04:53:11Z

CC @bcmills @jayconrod

mtibben · 2021-09-17T05:25:37Z

If this proposal moves forward, where does the dependency go in the go.mod file? (assuming the 1.17 format with multiple require blocks). Will it have a dedicated block for tools? Or are tools treated like // indirect and placed in the same block?

Good question! I'm not so familiar with the reasoning behind the multiple blocks... something to do with lazy loading? I'd defer to those with more experience in this area

mvdan · 2021-09-17T06:51:56Z

Personally, I think #42088 is already a pretty good solution. With it, one can write go generate lines like:

//go:generate go run golang.org/x/tools/cmd/stringer@1a7ca93429 -type=Foo

Similarly, go run pkg@version can be used in scripts, makefiles, and so on. Plus, it doesn't even require a go.mod file to be used; you can use this method anywhere, just like go install pkg@version.

Another big advantage is that you can pick specific versions of tools, and they won't interfere with your main go.mod module dependency graph. Perhaps I want to use a generator that pulls in an unstable master version of a library that my project also uses, and I don't want my project to be forced into using the same newer unstable version.

mvdan · 2021-09-17T06:54:58Z

The only downside to #42088 is that, if you repeat the same go run pkg@version commands across multiple files, it can get a bit repetitive. Luckily, you have multiple solutions at hand: sed scripts to keep the versions in sync, short script files to deduplicate the commands, or even a module-aware tool that could sync go run pkg@version strings with a go.mod file, if you wanted to do that.

seankhliao · 2021-09-17T07:02:13Z

Or GOBIN=local-dir go install pkg@version, always run from the local directory and not clobber whatever version the user may have globally installed.
I think it would be a mistake for modules to implicitly rely on shared mutable global bin dir for a first class workflow

mtibben · 2021-09-17T07:07:00Z

Oh interesting, thanks @mvdan I wasn't aware of that solution. 🤔

A few concerns immediately come to mind...

You mean go run hack.me/now@v1.0.0 will just download and run some random go code 😱 That is slightly unexpected to me, equivalent to a curl | bash command. My assumption was always that go run ran local code or modules already specified in go.mod, but seems that assumption is incorrect
Should gqlgen instructions always be to specify version with go run github.com/99designs/gqlgen@0.14.0? That seems verbose
Repetition across multiple files, keeping version in sync, yep your comment above nails it

mtibben · 2021-09-17T07:08:43Z

Also this go run solution should probably be added to the Go Modules FAQ if this is now considered best-practice for go:generate tools

mvdan · 2021-09-17T07:10:41Z

In module mode, go run can always download, build, and run arbitrary code. The difference between go run pkg relying on go.mod and go run pkg@version is how you specify the version and how it's verified. With a go.mod, you are forced into a specific version recorded in go.mod and go.sum. Without one, it's up to you what version you specify; @master is obviously risky, @full-commit-hash is safest, and @v1.2.3 is a middle ground that would probably be best for most people. Even if a malicious upstream rewrites a tag to inject code, GOPROXY and GOSUMDB should protect you from that.

mvdan · 2021-09-17T07:12:52Z

Also this go run solution should probably be added to the Go Modules FAQ if this is now considered best-practice for go:generate tools

It certainly warrants a mention. I'm not sure we should bless it as the only best practice, though, because there can be legitimate reasons for versioning, downloading, and running tools some other way. Perhaps some of your tools aren't written in Go, such as protoc, so you use a "tool bundler" that's entirely separate to Go. Or perhaps you do need your tools to share the same MVS graph with your main module for proper compatibility, so you want them to share a go.mod file.

mtibben · 2021-09-17T07:29:20Z

Gotta say though... go run pkg@version seems like a massive security footgun to me.

go install I understand well that it can download code from a remote location and build a binary. It's not obvious at all that go run directly executes code from a remote location, and I wonder how widely that is understood.

mtibben · 2021-09-17T07:33:46Z

So even with the go run pkg@version approach, I still think this proposal has value for specifying tool dependency versions in the context of a module. This approach avoids requiring a tools.go file (as with the existing best-practice), and avoids specifying the tool version for every file that uses it (with the go run approach)

lwc · 2021-09-17T12:34:30Z

Also worth noting: codegen tools like gqlgen and protobuf are often comprised of a generator command and a runtime, both of which typically need to be versioned in lock-step.

This proposal solves that case rather neatly, allowing go.mod to manage both generator and runtime versions.

fsouza · 2021-09-17T13:00:52Z

Personally, I think #42088 is already a pretty good solution. With it, one can write go generate lines like:
//go:generate go run golang.org/x/tools/cmd/stringer@1a7ca93429 -type=Foo
Similarly, go run pkg@version can be used in scripts, makefiles, and so on. Plus, it doesn't even require a go.mod file to be used; you can use this method anywhere, just like go install pkg@version.

We used to do that. Then people would have that replicated across different files and the version wouldn't always match, and we wanted to automate tool updating, so we figured that migrating to tools.go + having everything in go.mod would be better for compatibility with the ecosystem built around go modules (vs rolling our own tool to keep modules used directly in //go:generate up to date).

Again, tools.go works, but it's weird (not very scientific, I know 🙈). I think this proposal makes version management of tools better because it enables people to manage them using solely go commands (vs things like the bash oneliner shared by the OP).

bcmills · 2021-09-17T14:04:09Z

@jayconrod has previously suggested something similar, using a new directive (perhaps tool?) instead of a // tool comment.

Personally, I prefer the approach of adding a new directive — today we do treat requirements with // indirect comments a bit specially in terms of syntax, but they are semantically still just comments, and I would rather keep them that way at least to the extent possible.

A new tool directive, on the other hand, would allow us to preserve the existing semantics of go mod tidy without special treatment for // tool comments.

mvdan · 2021-09-17T14:10:36Z

@bcmills would such tool requirements be part of the same MVS module graph?

bcmills · 2021-09-17T14:32:11Z

The tool directive would list package paths (not module requirements), and the named packages would be treated as if imported in a .go source file in the main module.

In particular:

go mod tidy would ensure that the packages transitively imported by the named package (and its test) can be resolved from the module graph.
go mod vendor would copy the packages transitively imported by the name package into the vendor directory (but would omit its test code and dependencies as usual).
go list direct (cmd/go: enable listing direct dependency updates #40364) would report the named packages as direct imports.

carldunham · 2021-09-17T17:49:44Z

Or go list tools

jayconrod · 2021-09-20T20:26:52Z

I like this proposal. I've had something similar in my drafts folder for a while. @bcmills touched on the main difference. go.mod would have a tool directive that would name the full package path for the tool. You'd still need a separate require directive for the containing module, and that would be treated like a normal require directive by MVS.

module example.com/use

go 1.18

require golang.org/x/tools v0.1.6

tool golang.org/x/tools/cmd/stringer

I don't think go run tool@version and go install tool@version completely replace go run tool and go install tool. When the @version suffix is used, it ignores the go.mod file for the current module. That's useful most of the time, but not if you want to track your tool dependencies together with other dependencies, or if you want to use a patched version of a tool (applying replace directives).

mtibben · 2021-09-20T22:55:37Z

Yeah I like the tool directive. There might be a couple of tradeoffs with compatibility with older go versions. A tool directive wouldn't be recognised by older go versions, and presumably ignored. A require directive with // tool would be recognised, but would be removed by a go mod tidy.

A tool directive would keep the dependency tree separate - as they should be. For example, I don't think indirect dependencies would need to be tracked for tools, or shared by the module. Essentially a tool directive would specify a version when running go run tool instead of needing go run tool@version

mtibben · 2021-09-20T23:11:43Z

Or have I got that wrong? Is sharing indirect dependencies between tools and other dependencies a desirable feature?

jayconrod · 2021-09-20T23:24:52Z

A tool directive wouldn't be recognised by older go versions, and presumably ignored. A require directive with // tool would be recognised, but would be removed by a go mod tidy.

Right. The go command reports errors for unknown directives in the main module's go.mod file, but it ignores unknown directives in dependencies' go.mod files. So everyone working on a module that used this would need to upgrade to a version of Go that supports it (same as most other new features), but their users would be unaffected.

A tool directive would keep the dependency tree separate - as they should be. For example, I don't think indirect dependencies would need to be tracked for tools, or shared by the module. Essentially a tool directive would specify a version when running go run tool instead of needing go run tool@version

Or have I got that wrong? Is sharing indirect dependencies between tools and other dependencies a desirable feature?

My suggestion is to have tool act as a disembodied import declaration: it's just in go.mod instead of tools.go. You'd still need a require directive for the module providing the tool, and it would be treated as a regular requirement by go mod tidy and everything else.

If you don't want to mix tool and library dependencies in go.mod, it's probably better to either use go run tool@version or to have a separate tools.mod file, then go run -modfile=tools.mod tool.

mtibben · 2021-09-20T23:30:03Z

Yep that makes a lot of sense @jayconrod

rsc · 2021-10-06T22:07:40Z

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

mtibben · 2021-10-06T23:11:57Z

@jayconrod Did you want to write up the tool directive approach that we could incorporate as an option into this proposal? I'm happy to collaborate on it with you. Positive feedback on that approach so far in this thread, and it would be good to compare the options directly against each other, now that this proposal will be considered by the go-powers-that-be

For #48429 Change-Id: I1a7bd8ffddbc65e3b687dc1d40f3853702e1b5dc Reviewed-on: https://go-review.googlesource.com/c/go/+/521958 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Sam Thanawalla <samthanawalla@google.com> Reviewed-by: Michael Matloob <matloob@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>

stevenh · 2024-08-31T21:22:49Z

Nice to see this being worked on.

gopherbot · 2024-09-20T06:20:41Z

Change https://go.dev/cl/614555 mentions this issue: cmd/mod/edit: disallow relative tool paths

Allowing relative paths in `go.mod` introduced an inconsistency as we do not allow relative package paths anywhere else. For #48429 Change-Id: I5ef88aec4fe35f7e94a0cf6288e94099f3ca7a0e Reviewed-on: https://go-review.googlesource.com/c/go/+/614555 Reviewed-by: Sam Thanawalla <samthanawalla@google.com> Reviewed-by: Michael Matloob <matloob@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

Packages referenced by tool lines in go.mod files will now be included in the module graph for the new "tool" package pattern and the "all" package pattern. For #48429 Change-Id: I128f6a50880814bd5395674426c9a7ee2ddc19bf Reviewed-on: https://go-review.googlesource.com/c/go/+/521959 Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Michael Matloob <matloob@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

Running `go get -tool example.com/m1` will add a tool line to your mod file and add any missing dependencies. Running `go get -tool example.com/m1@none` will drop the tool line from your mod file. For #48429 Change-Id: I07b4776f1f55eff588d08cb6649d94cc42a729d2 Reviewed-on: https://go-review.googlesource.com/c/go/+/563175 Reviewed-by: Michael Matloob <matloob@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Sam Thanawalla <samthanawalla@google.com>

mcandre · 2024-10-04T03:42:34Z

Reminder that buildtime dependencies installed via out of band methods are likely missing from SBOM's sent to SCA tools.

govulncheck, Snyk, Dependabot, etc. look primarily in go.mod for information about Go dependencies.

A few container SCA tools like Docker Scout, Red Hat Quay Container Security Operator, Snyk Container may happen to match CVE's for vulnerable buildtime Go dependency tools. But only for Go tools installed via OS package managers, and likely only for Go tools that are not removed from the final image using modern, multi stage images.

gopherbot · 2024-11-13T22:05:26Z

Change https://go.dev/cl/613095 mentions this issue: cmd/go: cache executables built for go run

Running `go tool` with no arguments will now list built in tools followed by module defined tools. Running `go tool X` where X matches either the full package path, or the last segment of the package path, of a defined tool will build the tool to a known location and immediately execute it. For #48429 Change-Id: I02249df8dad12fb74aa244002f82a81af20e732f Reviewed-on: https://go-review.googlesource.com/c/go/+/534817 Reviewed-by: Michael Matloob <matloob@golang.org> Reviewed-by: Sam Thanawalla <samthanawalla@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

This change implements executable caching. It always caches the outputs of link steps used by go run. To do so we need to make a few changes: The first is that we want to cache binaries in a slightly different location than we cache other outputs. The reason for doing so is so that the name of the file could be the name of the program built. Instead of placing the files in $GOCACHE/<two digit prefix>/<hash>-d, we place them in $GOCACHE/<two digit prefix>/<hash>-d/<executable name>. This is done by adding a new function called PutExecutable that works differently from Put in two ways: first, it causes the binaries to written 0777 rather than 0666 so they can be executed. Second, PutExecutable also writes its outputs to a new location in a directory with the output id based name, with the file named based on p.Internal.ExeName or otherwise the base name of the package (plus the .exe suffix on Windows). The next changes are for writing and reading binaries from the cache. In cmd/go/internal/work.updateBuildID, which updates build ids to the content based id and then writes outputs to the cache, we first make the change to always write the content based id into a binary. This is because we won't be throwing the binaries away after running them. Then, if the action is a link action, and we enabled excutable caching for the action, we write the output to the binary cache. When reading binaries, in the useCache function, we switch to using the binary cache, and we also print the cached link outputs (which are stored using the build action's action id). Finally, we change go run to execute the built output from the cache. The support for caching tools defined in a module that are run by go tool will also use this functionality. Fixes #69290 For #48429 Change-Id: Ic5f1d3b29d8e9786fd0d564460e3a5f53e951f41 Reviewed-on: https://go-review.googlesource.com/c/go/+/613095 Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Alan Donovan <adonovan@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

gopherbot · 2024-11-21T18:26:06Z

Change https://go.dev/cl/630695 mentions this issue: cmd/go/internal/tool: set Internal.ExeName on tool's package

While the cached name of an executable is set based on the base name of the package path, the executable produced as the output of link doesn't have ExeName set on it and is just called a.out (with a .exe suffix on Windows). Set ExeName so that the first time the binary is run, from the directory link is run in, it has the right name for ps. For #48429 Change-Id: Ic049304ec6fd5b23c2f5aaaf91aa58d79fe5a7ba Cq-Include-Trybots: luci.golang.try:gotip-linux-amd64-longtest,gotip-windows-amd64-longtest Reviewed-on: https://go-review.googlesource.com/c/go/+/630695 Reviewed-by: Conrad Irwin <conrad.irwin@gmail.com> Reviewed-by: Hongxiang Jiang <hxjiang@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

meling · 2024-11-25T22:41:27Z

Playing with tip I can do:

go install tool github.com/alta/protopatch/cmd/protoc-gen-go-patch
go get -tool google.golang.org/protobuf/cmd/protoc-gen-go

But these do not appear to work (tools metapackage):

go install tools
go build tools

Is the tools metapackage planned for Go1.24?

matloob · 2024-11-26T00:03:43Z

@meling Apologies- the design doc is now a little out of date.

We changed the naming of the metapackage to tool to reduce user confusion: now users won't have to remember whether it's tool or tools. It's always tool: the go.mod directive is named tool, the metapackage is named tool, the go subcommand is tool, and the argument to go get is tool.

A few other major changes in the final tool support vs. the design doc:

Tools are supported in workspaces: just like within a single module if there's an ambiguous tool name the full package path needs to be specified to go tool an the tool metapackage consists of the union of each of the work modules' tool metapackages.
Tool executables are cached using the go build cache using the new executable caching support (cmd/go: cache link output binaries in the build cache #69290) rather than in the $GOCACHE/tool directory.
Relative tool paths are no longer supported (https://go.dev/cl/614555).

dmitshur · 2024-11-26T18:51:34Z

Adding a release blocker to track mentioning this new cmd/go feature in the Go 1.24 release notes.

GoVeronicaGo · 2024-11-27T18:04:09Z

@matloob @samthanawalla to confirm the code has been completed and only release notes are pending for 1.24.

mtibben · 2024-11-28T20:07:20Z

@dmitshur the Modules FAQ may also need to be updated

gopherbot · 2024-12-01T05:15:13Z

Change https://go.dev/cl/632555 mentions this issue: cmd/go: add tool meta-pattern to go help packages

gopherbot · 2024-12-01T06:34:22Z

Change https://go.dev/cl/632595 mentions this issue: _content/doc: document module tools

gopherbot · 2024-12-01T06:39:17Z

Change https://go.dev/cl/632556 mentions this issue: doc/next: introduce module tools

gopherbot · 2024-12-01T06:46:54Z

Change https://go.dev/cl/632596 mentions this issue: Modules: add go1.24 tools

gopherbot added this to the Proposal milestone Sep 17, 2021

gopherbot added the Proposal label Sep 17, 2021

ianlancetaylor changed the title ~~proposal: track tool dependencies in go.mod~~ proposal: cmd/go: track tool dependencies in go.mod Sep 17, 2021

timothy-king mentioned this issue Jul 25, 2024

x/tools/gopls: move analyzers to an extensible sidecar #59869

Open

hyangah mentioned this issue Aug 21, 2024

cmd/go: stamp the pseudo-version in builds generated by go build #50603

Open

matloob mentioned this issue Sep 6, 2024

cmd/go: cache link output binaries in the build cache #69290

Open

mfridman mentioned this issue Sep 9, 2024

Move parse package internally and modify Makefile mfridman/tparse#132

Merged

dmitshur modified the milestones: Backlog, Go1.24 Nov 23, 2024

dmitshur added the release-blocker label Nov 26, 2024

dmitshur mentioned this issue Nov 26, 2024

doc: write release notes for Go 1.24 #68545

Open

bep mentioned this issue Nov 27, 2024

Go 1.24 notes/TODOs gohugoio/hugo#13091

Open

10 tasks

cmd/go: track tool dependencies in go.mod #48429

cmd/go: track tool dependencies in go.mod #48429

Comments

mtibben commented Sep 17, 2021 • edited by matloob Loading

Background

Proposed Solution

fsouza commented Sep 17, 2021

ianlancetaylor commented Sep 17, 2021

mtibben commented Sep 17, 2021

mvdan commented Sep 17, 2021

mvdan commented Sep 17, 2021

seankhliao commented Sep 17, 2021

mtibben commented Sep 17, 2021

mtibben commented Sep 17, 2021

mvdan commented Sep 17, 2021

mvdan commented Sep 17, 2021

mtibben commented Sep 17, 2021 • edited Loading

mtibben commented Sep 17, 2021

lwc commented Sep 17, 2021

fsouza commented Sep 17, 2021

bcmills commented Sep 17, 2021

mvdan commented Sep 17, 2021

bcmills commented Sep 17, 2021

carldunham commented Sep 17, 2021

jayconrod commented Sep 20, 2021

mtibben commented Sep 20, 2021 • edited Loading

mtibben commented Sep 20, 2021

jayconrod commented Sep 20, 2021

mtibben commented Sep 20, 2021

rsc commented Oct 6, 2021

mtibben commented Oct 6, 2021

stevenh commented Aug 31, 2024

gopherbot commented Sep 20, 2024

mcandre commented Oct 4, 2024 • edited Loading

gopherbot commented Nov 13, 2024

gopherbot commented Nov 21, 2024

meling commented Nov 25, 2024

matloob commented Nov 26, 2024

dmitshur commented Nov 26, 2024

GoVeronicaGo commented Nov 27, 2024

mtibben commented Nov 28, 2024

gopherbot commented Dec 1, 2024

gopherbot commented Dec 1, 2024

gopherbot commented Dec 1, 2024

gopherbot commented Dec 1, 2024

mtibben commented Sep 17, 2021 •

edited by matloob

Loading

mtibben commented Sep 17, 2021 •

edited

Loading

mtibben commented Sep 20, 2021 •

edited

Loading

mcandre commented Oct 4, 2024 •

edited

Loading