proposal: cmd/go: track tool dependencies in go.mod

[mtibben]

Background

The current best-practice to track tool dependencies for a module is to add a tools.go file to your module that includes import statements for the tools of interest. This has been extensively discussed in #25922 and is the recommended approach in the Modules FAQ

This approach works, but managing the tool dependencies still feels like a missing piece in the go mod toolchain. For example, the instructions for getting a user set up with a new project using gqlgen (a codegen tool) looks like this

# Initialise a new go module
mkdir example
cd example
go mod init example

# Add gqlgen as a tool
printf '// +build tools\npackage tools\nimport _ "github.com/99designs/gqlgen"' | gofmt > tools.go
go mod tidy

# Initialise gqlgen config and generate models
go run github.com/99designs/gqlgen init

The printf line above really stands out as an arbitrary command to "add a tool" and reflects a poor developer experience when managing tools. For example, an immediate problem is that the printf line will only work on unix systems and not windows. And what happens if tools.go already exists?

So while we have some excellent tools for managing dependencies within the go.mod file using go get and go mod edit, there is no such equivalent for managing tools in the tools.go file.

Proposed Solution

The go.mod file uses the // indirect comment to track some dependencies. An // indirect comment indicates that no package from the required module is directly imported by any package in the main module (source).

I propose that this same mechanism be used to add tool dependencies, using a // tool comment.

Users could add a tool with something like

go get -tool github.com/99designs/gqlgen@v0.14.0

or

go mod edit -require=github.com/99designs/gqlgen -tool

A go.mod would then look something like

module example

go 1.17

require (
	github.com/99designs/gqlgen v0.14.0 // tool
)

And would allow users to subsequently run the tool with go run github.com/99designs/gqlgen

This would mean a separate tools.go file would no longer be required as the tool dependency is tracked in the go.mod file.

Go modules would be "tool" aware. For example:

• go mod tidy would not remove the // tool dependency, even though it is not referenced directly in the module
• Perhaps if a module with a // tool dependency is imported by another module, Go modules understands that the // tool dependency is not required as an indirect dependency. Currently when using tools.go, go modules does not have that context and the tool is treated like any other indirect dependency
• go get -tool [packages] would only add a dependency with a main package

[fsouza]

I like this, I find it annoying to use the tools.go solution, though I'll admit I don't have a better complaint than it being annoying/weird.

If this proposal moves forward, where does the dependency go in the go.mod file? (assuming the 1.17 format with multiple require blocks). Will it have a dedicated block for tools? Or are tools treated like // indirect and placed in the same block?

[ianlancetaylor]

CC @bcmills @jayconrod

[mtibben]

If this proposal moves forward, where does the dependency go in the go.mod file? (assuming the 1.17 format with multiple require blocks). Will it have a dedicated block for tools? Or are tools treated like // indirect and placed in the same block?

Good question! I'm not so familiar with the reasoning behind the multiple blocks... something to do with lazy loading? I'd defer to those with more experience in this area

[mvdan]

Personally, I think #42088 is already a pretty good solution. With it, one can write go generate lines like:

//go:generate go run golang.org/x/tools/cmd/stringer@1a7ca93429 -type=Foo

Similarly, go run pkg@version can be used in scripts, makefiles, and so on. Plus, it doesn't even require a go.mod file to be used; you can use this method anywhere, just like go install pkg@version.

Another big advantage is that you can pick specific versions of tools, and they won't interfere with your main go.mod module dependency graph. Perhaps I want to use a generator that pulls in an unstable master version of a library that my project also uses, and I don't want my project to be forced into using the same newer unstable version.

[mvdan]

The only downside to #42088 is that, if you repeat the same go run pkg@version commands across multiple files, it can get a bit repetitive. Luckily, you have multiple solutions at hand: sed scripts to keep the versions in sync, short script files to deduplicate the commands, or even a module-aware tool that could sync go run pkg@version strings with a go.mod file, if you wanted to do that.

[seankhliao]

Or GOBIN=local-dir go install pkg@version, always run from the local directory and not clobber whatever version the user may have globally installed.
I think it would be a mistake for modules to implicitly rely on shared mutable global bin dir for a first class workflow

[mtibben]

Oh interesting, thanks @mvdan I wasn't aware of that solution. 🤔

A few concerns immediately come to mind...

1. You mean go run hack.me/now@v1.0.0 will just download and run some random go code 😱 That is slightly unexpected to me, equivalent to a curl | bash command. My assumption was always that go run ran local code or modules already specified in go.mod, but seems that assumption is incorrect

2. Should gqlgen instructions always be to specify version with go run github.com/99designs/gqlgen@0.14.0? That seems verbose

3. Repetition across multiple files, keeping version in sync, yep your comment above nails it

[mtibben]

Also this go run solution should probably be added to the Go Modules FAQ if this is now considered best-practice for go:generate tools

[mvdan]

In module mode, go run can always download, build, and run arbitrary code. The difference between go run pkg relying on go.mod and go run pkg@version is how you specify the version and how it's verified. With a go.mod, you are forced into a specific version recorded in go.mod and go.sum. Without one, it's up to you what version you specify; @master is obviously risky, @full-commit-hash is safest, and @v1.2.3 is a middle ground that would probably be best for most people. Even if a malicious upstream rewrites a tag to inject code, GOPROXY and GOSUMDB should protect you from that.

[mvdan]

Also this go run solution should probably be added to the Go Modules FAQ if this is now considered best-practice for go:generate tools

It certainly warrants a mention. I'm not sure we should bless it as the only best practice, though, because there can be legitimate reasons for versioning, downloading, and running tools some other way. Perhaps some of your tools aren't written in Go, such as protoc, so you use a "tool bundler" that's entirely separate to Go. Or perhaps you do need your tools to share the same MVS graph with your main module for proper compatibility, so you want them to share a go.mod file.

[mtibben]

Gotta say though... go run pkg@version seems like a massive security footgun to me.

go install I understand well that it can download code from a remote location and build a binary. It's not obvious at all that go run directly executes code from a remote location, and I wonder how widely that is understood.

[mtibben]

So even with the go run pkg@version approach, I still think this proposal has value for specifying tool dependency versions in the context of a module. This approach avoids requiring a tools.go file (as with the existing best-practice), and avoids specifying the tool version for every file that uses it (with the go run approach)

[lwc]

Also worth noting: codegen tools like gqlgen and protobuf are often comprised of a generator command and a runtime, both of which typically need to be versioned in lock-step.

This proposal solves that case rather neatly, allowing go.mod to manage both generator and runtime versions.

[fsouza]

Personally, I think #42088 is already a pretty good solution. With it, one can write go generate lines like:

//go:generate go run golang.org/x/tools/cmd/stringer@1a7ca93429 -type=Foo

Similarly, go run pkg@version can be used in scripts, makefiles, and so on. Plus, it doesn't even require a go.mod file to be used; you can use this method anywhere, just like go install pkg@version.

We used to do that. Then people would have that replicated across different files and the version wouldn't always match, and we wanted to automate tool updating, so we figured that migrating to tools.go + having everything in go.mod would be better for compatibility with the ecosystem built around go modules (vs rolling our own tool to keep modules used directly in //go:generate up to date).

Again, tools.go works, but it's weird (not very scientific, I know 🙈). I think this proposal makes version management of tools better because it enables people to manage them using solely go commands (vs things like the bash oneliner shared by the OP).

[bcmills]

@jayconrod has previously suggested something similar, using a new directive (perhaps tool?) instead of a // tool comment.

Personally, I prefer the approach of adding a new directive — today we do treat requirements with // indirect comments a bit specially in terms of syntax, but they are semantically still just comments, and I would rather keep them that way at least to the extent possible.

A new tool directive, on the other hand, would allow us to preserve the existing semantics of go mod tidy without special treatment for // tool comments.

[mvdan]

@bcmills would such tool requirements be part of the same MVS module graph?

[bcmills]

The tool directive would list package paths (not module requirements), and the named packages would be treated as if imported in a .go source file in the main module.

In particular:

• go mod tidy would ensure that the packages transitively imported by the named package (and its test) can be resolved from the module graph.
• go mod vendor would copy the packages transitively imported by the name package into the vendor directory (but would omit its test code and dependencies as usual).
• go list direct (cmd/go: enable listing direct dependency updates #40364) would report the named packages as direct imports.

[carldunham]

Or go list tools

[jayconrod]

I like this proposal. I've had something similar in my drafts folder for a while. @bcmills touched on the main difference. go.mod would have a tool directive that would name the full package path for the tool. You'd still need a separate require directive for the containing module, and that would be treated like a normal require directive by MVS.

module example.com/use

go 1.18

require golang.org/x/tools v0.1.6

tool golang.org/x/tools/cmd/stringer

I don't think go run tool@version and go install tool@version completely replace go run tool and go install tool. When the @version suffix is used, it ignores the go.mod file for the current module. That's useful most of the time, but not if you want to track your tool dependencies together with other dependencies, or if you want to use a patched version of a tool (applying replace directives).

[mtibben]

Yeah I like the tool directive. There might be a couple of tradeoffs with compatibility with older go versions. A tool directive wouldn't be recognised by older go versions, and presumably ignored. A require directive with // tool would be recognised, but would be removed by a go mod tidy.

A tool directive would keep the dependency tree separate - as they should be. For example, I don't think indirect dependencies would need to be tracked for tools, or shared by the module. Essentially a tool directive would specify a version when running go run tool instead of needing go run tool@version

[mtibben]

Or have I got that wrong? Is sharing indirect dependencies between tools and other dependencies a desirable feature?

[jayconrod]

A tool directive wouldn't be recognised by older go versions, and presumably ignored. A require directive with // tool would be recognised, but would be removed by a go mod tidy.

Right. The go command reports errors for unknown directives in the main module's go.mod file, but it ignores unknown directives in dependencies' go.mod files. So everyone working on a module that used this would need to upgrade to a version of Go that supports it (same as most other new features), but their users would be unaffected.

A tool directive would keep the dependency tree separate - as they should be. For example, I don't think indirect dependencies would need to be tracked for tools, or shared by the module. Essentially a tool directive would specify a version when running go run tool instead of needing go run tool@version

Or have I got that wrong? Is sharing indirect dependencies between tools and other dependencies a desirable feature?

My suggestion is to have tool act as a disembodied import declaration: it's just in go.mod instead of tools.go. You'd still need a require directive for the module providing the tool, and it would be treated as a regular requirement by go mod tidy and everything else.

If you don't want to mix tool and library dependencies in go.mod, it's probably better to either use go run tool@version or to have a separate tools.mod file, then go run -modfile=tools.mod tool.

[mtibben]

Yep that makes a lot of sense @jayconrod

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[mtibben]

@jayconrod Did you want to write up the tool directive approach that we could incorporate as an option into this proposal? I'm happy to collaborate on it with you. Positive feedback on that approach so far in this thread, and it would be good to compare the options directly against each other, now that this proposal will be considered by the go-powers-that-be

[mewmew]

Firstly, I also support the proposed dedicated tool directive, as this reduces the use of "magic comments" with semantic meaning, of which there are already enough in the Go ecosystem. We don't need to introduce more :)

re: go install tools by @jayconrod in #48429 (comment)

# Install all tools in GOBIN
go install tools

# Install all tools in the bin/ directory
go build -o bin/ tools

# Update all tools to their latest versions.
go get tools

Just a quick comment regarding tool dependencies used by Go modules. Not all external tools will be developed in Go, some external tools may be developed in arbitrary languages, but we may still wish to pin specific Git commit revisions or tagged versions to ensure that all build requirements of our Go modules are satisfied and don't go out of sync with the required versions of external tools.

A real world example of this is the Textmapper tool (written in Java) used by github.com/llir/llvm to generate lexers and parses for LLVM IR from a BNF grammar. Since the Textmapper tool is not written in Go, it is currently tracked by a Git submodule: https://github.com/llir/ll/tree/master/tools

Just put this out here, to keep in consideration when working on dependency handling of tools (e.g. build tools) required by the Go module.

Should go.mod also pin versions/revisions of tools developed in other languages than Go? If not, simply disregard this commit.

Cheers,
Robin

P.S. @inspirer is working on a Go version of Textmapper (inspirer/textmapper#6), but it has yet to reach feature parity with the Java version. (The above still applies to other tool dependencies used by Go modules and developed in other languages than Go of course.)

[deefdragon]

@rsc This should be able to be taken off hold right?

Also, I am a bit confused why this went on hold in the first place. I understand that there was a lot of 1.18 work, but why did that necessitate this going on hold again?

[ianlancetaylor]

@bcmills Is there more work pending, or should this come off hold? Thanks.

[bcmills]

This should come off hold.

[ConradIrwin]

I like the proposal to add a tool directive to go.mod.

Instead of adding go get -tool, perhaps go install should update these lines when executed within a module (though this may have too many false-positives because I don't pay attention to my working directory when installing things, it would avoid the need for new syntax to learn – and any incidentally added tools could easily be removed).

Unlike the proposal by @jayconrod, I would not merge the tools' dependencies into the package's dependencies (and I'd build the tools ignoring the go modules require and replace directives). Each tool should be built in standalone mode, because it would be surprising for me that adding a tool could affect my main module's build.

As such I'd make the syntax inclusive of a version number:

tool golang.org/x/tools/cmd/stringer@v0.2.0

At this point running go run golang.org/x/tools/cmd/stringer or go install golang.org/x/tools/cmd/stringer in the module directory would always pick up v0.2.0.

[ConradIrwin]

Thinking further...

One problem with the current approach to bundling tools with repositories is that in order to manually run the tool I need to either type go run quite/a/lot/to/type every time (which is tedious),go install it (which means that I no longer can be sure I'm running the right version as I hop from project to project), or write a wrapper script (which is unnecessary).

Maybe a better way of avoiding this is to focus on fixing the ergonomics of running tools that have been bundled with the module, and making it easy to run the correct version (c.f. #57001 for go itself).

I propose the following:

1. go.mod gets a new directive run that lets you define a set of tools for use with the current project. The syntax is run [toolname] => [path-to-run]. The path-to-run must be in the current module (starting with ./) or in a module that is required by the current module.

   require golang.org/x/tools v0.2.0
   run stringer => golang.org/x/tools/cmd/stringer
   run boop => ./cmd/boop.go
   

   The reason that it is restricted to either the current module (or modules you depend on) is so that transitive dependencies appear in your go.mod and go.sum to give you reproducible builds, and it gives you the ability to replace tool dependencies if you wish. It is arguably a bit odd that the tool will use the same version of dependencies as your main module, but it is not likely a problem in practice, and it keeps the mental model simpler.

   The reason to explicitly specify the toolname is to allow for the case where there are multiple commands with the same name. If this isn't a case we want to support, it would be reasonable for the syntax to be run path/to/X where the toolname is inferred from the last path segment.

   Run lines may only contain one [path-to-run], so if you want to write a tool using multiple go files, you'll have to put them in a directory.

2. go run X would look for a run directive with toolname X and (if it exists) would act as though you'd run go run path-to-run in module mode.

   When run in this mode, go run would cache the fully linked binaries so that future runs of go run X do not need to re-link if the built tool is up-to-date (just as go build does). This should mean that running go run X is relatively quick the second time.

   The binary will be cached at $GOCACHE/tool/<current-module-path>/<toolname>, so there will be at most one version of each tool cached per module. Unlike go run in general, this should have a relatively good hit rate. go clean -cache would empty this directory. (We could also add a separate go clean -runcache if we think it's likely people will want to clear this directory without clearing the rest of the cache).

3. go get -run X@version would add a new run line, inferring the toolname from the last path segment; and (if necessary) add a require line for the module containing X at the given version. go get -run X@none would remove the line (and also the require if it's not otherwise needed).

Although I like @jayconrod's idea of the tools meta-package described above, I have removed it from this proposal because I think the use-case is not that clear. It would be somewhat nice to be able to "precompile all tools" so that go run X is fast the first time, but it's not essential (and it would be possible to write a script that did that). It could be a good thing to add later if there's demand.

Edit: this was updated to reflect a slightly tighter scope; and to rename the directive to "run" instead of "tool" to (maybe) reduce confusion (as go tool does something completely different from go run). (Though maybe go tool X should get this behaviour instead of go run X?)

Previous version

I propose the following:

1. go.mod gets a new directive tool that lets you define a set of tools for use with the current project.

   // stringer is a standalone tool
   tool stringer => golang.org/x/tools/cmd/stringer v0.2.0
   
   // boop is a tool in the current module
   tool boop => ./tools/boop
   
   // protoc-gen-go is provided by a required module
   tool protoc-gen-go => google.golang.org/protobuf/cmd/protoc-gen-go
   require google.golang.org/protobuf 1.28.1
   

   If a version number is specified in the tool line then that version of the tool is used, it is built in standalone mode (ignoring the require/replace directives of the current go.mod). If a version number is not specified, then it is built in "companion mode" (respecting the require/replace directives of the current go.mod), and it must either be in the current module or in a module that is required (copying the behavior of go run today).

2. go run learns to pick up tools from the go.mod file: go run stringer would work exactly as if I'd run go run golang.org/x/tools/cmd/stringer@v0.2.0.

3. go get tools would download tool dependencies, go build tools would compile them and cache the result so that go run stringer is fast. go install tools would install them globally (though this may be an anti-pattern).

4. go get -tool golang.org/x/tools/cmd/stringer@latest in module mode would add a tool line with the latest version, and with the toolname inferred from the path of the module in addition to doing the install.
   It is arguably possible to omit the first argument to the tool directive in the go.mod and infer it from the last path segment of the second argument (tool golang.org/x/tools/cmd/stringer@v0.2.0 would be equivalent to tool stringer golang.org/x/tools/cmd/stringer@v0.2.0) but this feels a bit too magic to me...

This would not support non-go tools, as I think specifying a way to version arbitrary binaries is probably out of scope for go's tooling.

I am not sure whether tool stringer golang.org/x/tools/cmd/stringer@v0.2.0 should change the behaviour of running go run golang.org/x/tools/cmd/stringer; it definitely could pick the version up from the go.mod, but it may not be clear why it would given the chosen syntax.

This does require teaching people to use go run stringer instead of stringer or go run golang.org/x/tools/cmd/stringer@v0.2.0, but I think that's a much better tradeoff all around.

See also: #44469, #42088 #33468

[gopherbot]

Change https://go.dev/cl/472755 mentions this issue: cmd/go: support run directive to go.mod

[ConradIrwin]

@rsc is there a way to ask the proposal committee to take a look at this when you next meet?

There are roughly two options proposed here: which seems like the right direction, and what are the remaining things to resolve before something like this could be accepted?
1. the one proposed by @jayconrod: #48429 (comment) which replaces the existing hack with a line in go.mod
2. the version proposed by me: #48429 (comment) that attempts to solve the problem more holistically.

I'd be interested in trying to implement either of these approaches (or an alternate idea) for go 1.22; but I'd love some input from you all on what makes sense as a next step; and if it'd be helpful I'm happy to write a more detailed proposal doc.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[joeblubaugh]

I have some questions about @ConradIrwin 's proposal:

• Should run directives really participate in MVS? One advantage of scripting go install X@v1.0.0 over tools.go is that the version of the installed tool doesn't affect the version of the modules selected for building the module's code.
• go get -run X@none feels hacky - should the user just delete the run directive instead, and run go mod tidy? If not, perhaps a change to go mod? Something like go mod remove?
• What happens if I call go get -run -u X?

I had a thought about a separate tool management system related to go install. A file go.tools that can interact with go install and go.run:

install ./cmd/boop.go
// or, with an alias:
install anyToolName => ./cmd/boop.go
install golang.org/x/tools/cmd/stringer v0.2.0

When run without arguments, in a module with a go.tools file, go install will:

• build main packages that are local to the module like ./cmd/boop.go
• build non-local packages in the same way the go install currently does.

These compiled binaries will be cached in the same way as @ConradIrwin's proposal. When calling go run within a module with a go.tools file, go run will select the cached version of the binary specified by go.tools

I like that this idea cleanly separates tools used to interact with a module (and may do nothing with the Go source at all), and dependency versions required to build and test the module's Go code.

[leighmcculloch]

I think @ConradIrwin's proposal (#48429 (comment)) works for most of the Go repositories that I work in and I think I would use it, but it feels unnecessary given the go run command can run any Go tool already.

+1 @joeblubaugh's concern (#48429 (comment)) about the tools affecting version selection. I think this would create some big surprises. There are times where it would be super useful if I'm using a tool that has a corresponding library that have to be aligned. But there are also times where it would be surprising if my transitive dependencies controlled which version of a tool was in use.

For the most part I've found the go run module/path@version to be a really effective way to run tools. I use that syntax in gogenerate directives and in Makefiles and it works great.

I think the main thing these proposals are adding is aliasing shorter names to paths. If paths are truly too long to be convenient maybe a general path aliasing system would be appropriate. Other systems have done this. For example, Deno added support for aliasing with their import map file (e.g. https://deno.land/manual@v1.31.0/basics/import_maps). I'm not advocating for aliasing, I don't think the go tool should adopt path aliasing as a feature, but that's what it feels like these proposals are adding to the go tool, narrowly for tools.

[bcmills]

I think the main thing these proposals are adding is aliasing shorter names to paths.

I think there are two separate concerns.

One concern is adding tools to the dependency graph, particularly for go mod vendor. It isn't feasible to separate those dependencies from the general dependency graph, because the vendor tree (intentionally) doesn't allow for more than one version of the same package import path, and I don't think it's worth adding more complexity in order to support that.

The other concern is making it easier to run tools; I think that's what the aliasing is getting at. I could see that being useful for, say, tests that run those tools. But I think there is a lot of complexity there that would need to be resolved — for example, if I run go test example.com/m, how would m_test identify the stringer selected by the module from which go test is run (which may in general be different from the one containing example.com/m)?

[ConradIrwin]

@joeblubaugh / @leighmcculloch I went back and forth on "should they contribute to MVS or not". Originally I thought maybe the module author should be able to chose, but the distinction is subtle. I landed on "yes, they should" to give module authors control over which dependencies are pulled in. You can of course still go run x@y (or go install x@y) if you need to not have them intermingle. For the projects I work on, it would make little difference because the dependencies of my tools are mostly distinct from the dependencies of the app (and as @bcmills points out, it would be a very big change to allow tool dependencies to differ from the main module dependencies).

I am not sure whether go get -run is needed, but I liked the idea from @jayconrod's proposal because it gives you a one-line command to run to make the change, which could be copy/pasted into documentation. The @none syntax is already supported by go get, so it made sense to me to support here too. go get -run -u would act just like go get -u (without the -run).

A separate go.tool file seems unnecessary make that much sense if the tools are participating in MVS as then any tool added would need changes in two files. It seems simpler to use the one file.

@joeblubaugh I think that change to go install you propose would be quite intrusive (I mostly use go install to "put the binary produced when building the current package on my path"). @jayconrod had proposed making go install tools do something like what you suggest – though putting them directly in the path – we could expand this proposal to support that (or do later) if it's a common desire.

@leighmcculloch glad to hear you would use this! I do hear your point around aliasing being unnecessary (and indeed I'd be happy to have something that did the versioning and caching without the aliasing). Adding the aliasing I think makes go run significantly more user-friendly (currently I either go install and then use the binary name, which leads to problems making sure the version matches between repos; or I write wrapper scripts to avoid having to type the full path; it'd be nice to just go run X instead).

[ConradIrwin]

@bcmills interesting thought. Currently go test builds the test binary with the current module's dependencies, but runs the tests in the directory containing that module's code. I think this means that it will "do the right thing" in most cases – a test that shells out to go run will pick up the run directives from the module being tested.

It does mean that if you have a different version of the tool required by the main module and by example.com/m then the test will be compiled with one version but go run will run with a different version. (This already is a problem today if you shell out to go run example.com/tool in the tests of example.com/m using the current tools.go hack).

I think it would be theoretically possible to fix the version mismatch in the specific case of go test, but I'm not sure that the cost would be worth it in practice. (We'd either need a new dependency resolution mode, or code to generate a new go.mod that merges two previous ones, and an environment variable to tell go run to change its behaviour).

I'm sure we shouldn't try to fix this in the case that you go build an arbitrary binary and then run it in a different directory – if it shells out to go run then go run would have no specific knowledge of the module used to build the binary; it would just use the working directory. For me that's a pretty strong argument that it should work the same way for tests too.

I don't think this is a problem for other go commands (go run doesn't change directory, go generate only works on the current module), but are there other places it's likely to show up (and cause actual problems)?

[perj]

Overall, this seems useful, the tools.go situation has always been an annoyance to me. I tend to put go run github.xom/x/y in my go:generate comments and thus rely on go.mod for the version info. I don't really mind typing go run github.com/x/y in my shell either but shortcuts are interesting, especially when working with people not that familiar with go.

The proposal so far doesn't mention any go mod edit commands to edit run lines, can I assume those will be added?

Finally, a bit of devil's advocate... currently I can create a vendor/hello/ directory and use go run hello to run it. How does this proposal address that versus the shortcuts? I'm not doing that in practice, but it's possible.

[ConradIrwin]

@perj Good to know that this would be useful to you! And good call on go mod edit; we should add go mod edit -run and go mod edit -droprun.

I hadn't realized about /vendor/x – if you have both (and they point to different things) the ones in go.mod should take precedence; otherwise if you have a go.mod file with run directives when you go mod vendor the behaviour of the directive could change.

[mcandre]

Meanwhile I have published a basic CLI tool to pin Go dev tool versions.

https://github.com/mcandre/accio

As much as I enjoy contributing developer tools, I would prefer to be able to deprecate this workaround and just use the builtin go mod system.

(Would also love to be able to ditch modvendor, and have go mod vendor stop deleting critical cgo source files, for the same reason. But that's uh off topic for this discussion )

[ConradIrwin]

@mcandre Thanks for sharing! Would the change proposed in #48429 (comment) give you the benefit you get from accio?

I notice it takes quite a different approach (installing specific dependency versions into the path, vs requiring go run toolname), but I'm hoping that this change would solve the same problem (having the tools you need to collaborate on a module at your fingertups) albeit in a different way.

[bwplotka]

Thanks for the proposal @ConradIrwin and others! Happy to see this moving forward 💪🏽 Similar to @mcandre I would love to switch to native solution. In the meantime https://github.com/bwplotka/bingo is still up to date and got quite some traction.

My 2c of the discussion so far, assuming we iterate over @ConradIrwin proposal:

1. Having some aliasing is useful, although majority of those tools are used in Makefiles at the end, so we could do it in a separate iteration of the proposal.
   a. Side question: Can I track, within this proposal, multiple tools from same Go module under different version? (Example use case: e2e test that runs through multiple Go binaries of different minor versions).
2. I like the version pinning potential and reusing go.mod semantics.
3. If we are meant to use normal go.mod, I am worried about the dependency hell, so reusing the same MVC/dependency graph as the module. I know @bcmills mentioned it's not trivial, but I think it's must-have. IMO we don't want any Go modules importing my module to have pain of downloading and match all the dependencies for the tools that are not needed to compile my code. Furthermore, while this is probably controversial, many tools require custom replace directives (incompatibilities still) and users don't want to spent hours to craft dependencies of 10 tools to strictly work under the same deps together for no good benefit (unless I miss the benefits, other then downloading less overall?).

Also agree with majority of @leighmcculloch comment, except this one:

I think the main thing these proposals are adding is aliasing shorter names to paths.

If that's true, then we might be missing the point. To me the main problem of the issue we are trying to solve here is to be able to save and track versions of tools and its dependencies (including potential replace directives) in declarative way for the portability of the project development. I think it has to be a separate dependency graph as mentioned above.

[ConradIrwin]

@bwplotka I strongly agree that the main issue to solve is versioning (though I think aliases help, I'd be happy to defer to a second round too). Responding to other points:

Can I track, within this proposal, multiple tools from same Go module under different version? (Example use case: e2e test that runs through multiple Go binaries of different minor versions).

Not as proposed. Similar to how it works for go libraries, you can only have one version of a given module in your go.mod.

IMO we don't want any Go modules importing my module to have pain of downloading and match all the dependencies for the tools that are not needed to compile my code

Agreed! With this proposal if your module depends on a module that has run directives, you will not inherit the dependencies of those run directives in your go.mod (this is different to how it works if you have a tools.go in a depended-on package today).

That said, we still maintain the invariant that your go.mod can only have one version of each module; so if a tool you use depends on a module that your code also depends on, you must chose one version that works with both your code and the tool. (Similarly if multiple tools depend on the same dependency, it must resolve to exactly one version).

There are some advantages to this: you can be sure that a tool used in go:generate has the same version as any library it includes in your code, and you can control the dependencies of your tools with require and replace directives exactly as you do for your own code. Would this be sufficient to solve the problems you've experienced?

(It's maybe worth noting that this proposal does not aim to solve the problem for every tool. You call out prometheus in the bingo blog post, but as you cannot go run github.com/prometheus/prometheus/cmd/prometheus@latest, this proposal isn't directly trying to make it work. Interestingly, you can force it to work by copy-pasting replace directives, but it's probably best to follow their installation instructions).