Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: support module-local install/run of tool dependencies #27653

Closed
myitcv opened this issue Sep 13, 2018 · 23 comments

Comments

Projects
None yet
9 participants
@myitcv
Copy link
Member

commented Sep 13, 2018

What version of Go are you using (go version)?

go version go1.11 linux/amd64

What did you do?

Tied to install and run a tool dependency in a number of my modules (more detail below)

What did you expect to see?

A nice easy way to (install and) run a tool dependency.

What did you see instead?

I needed to set GOBIN and update PATH for each module.

Further detail (on What did you do?)

This issue builds on top of #25922 so if that changes shape in any significant way it may void what follows.

Per #27643 (comment), I think we need to make the workflow around using a module's tool dependencies easier. Let me try to explain by covering my "workflow".

The tools I use on a day-to-day basis fall into two categories:

  1. Tools I want to be controlled by a project's go.mod
  2. Tools that I need globally

Dealing with these in reverse order.

Category 2: I think it's clear that with Go 1.11, there is a "gap" here and that this is covered by #24250. Per the detail in that discussion, there are open questions on how to handle multiple versions, where the installed binaries should be put etc, but it all falls under that issue.

Category 1: by far the largest category of tools for me, made up largely of code generators that I use with go generate and the like. I absolutely want these to be version controlled. And I don't want to be using (via my PATH) a "global" install of such a tool, even if the version just happens to match at that point in time. But both go get and go install currently (i.e. Go 1.11) have a target of $GOPATH/bin (ignoring multi-element `GOPATH values for now).

Hence the workflow I have effectively adopted, building on #25922, is to create a module-local install target:

# create a new module
cd $(mktemp -d)
mkdir hello
cd hello
go mod init example.com/hello

# set GOBIN as a module-local install target
export GOBIN=$PWD/.bin

# update my PATH accordingly (I actually use https://github.com/cxreg/smartcd for this)
export PATH=$GOBIN:$PATH

# add a tool dependency (by definition, category 1 tool) following "best practice" laid out in 
# https://github.com/golang/go/issues/25922#issuecomment-412992431
cat <<EOD > tools.go
// +build tools

package tools

import (
        _ "golang.org/x/tools/cmd/stringer"
)
EOD

# install the tool
go install golang.org/x/tools/cmd/stringer

# verify we are using the module-local binary
which stringer

# which gives something like:
# /tmp/tmp.Hh0BNOF6k2/hello/.bin/stringer

As covered in #27643, one of the disconnects in Go 1.11 is that a go get or go install in a module context modifies the "local" go.mod but installs "globally". This is, as @bcmills put it, "weird". But is to my mind a gap in Go 1.11, just as there not being a "global" tool install is a gap (i.e. #24250).

Potential solutions

Just listing these here as a starting point:

  • go run is a potential alternative to the "local" install here (and a very attractive one to my mind), but we need to find a way to address #25416.
  • There could be some convention that a .bin/ directory, alongside a go.mod, is the target for go get and go install (of main packages) for "local" installs? But this wouldn't obviate the need for everyone to update their PATH and indeed .gitignore the .bin directory for every module they work on.
  • ...

cc @bcmills @rogpeppe @mvdan

@myitcv myitcv added this to the Go1.12 milestone Sep 13, 2018

@bcmills bcmills changed the title cmd/go: need to support module-local install/run of tool dependencies cmd/go: support module-local install/run of tool dependencies Sep 13, 2018

@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 13, 2018

(CC @rsc)

@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 13, 2018

The go run option seems pretty clean to me, and wouldn't expand the command surface at all.

@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 13, 2018

Another option might be some sort of go tool integration: go install -tool golang.org/x/tools/cmd/stringer and go tool stringer $pkg

@gregwebs

This comment has been minimized.

Copy link

commented Oct 11, 2018

Have you tried retool? I am successfully using that for all my tool installs.

@myitcv

This comment has been minimized.

Copy link
Member Author

commented Oct 11, 2018

Have you tried retool? I am successfully using that for all my tool installs.

@gregwebs thanks for the link, yes that had come across my radar before.

I think we should avoid dependencies beyond the standard Go distribution where possible, particularly for, what some might consider, something as fundamental as this.

@mvdan

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

I think this is especially an issue not to fix with third party tools. How would you ensure that the third party tool to install tools can be easily run by all developers? It creates a circular dependency that's entirely avoided if this mechanism is available in the Go toolchain.

@gregwebs

This comment has been minimized.

Copy link

commented Oct 11, 2018

Since you asked, the way I ensure it can be run by all developers is with which retool >/dev/null 2>&1 || go get github.com/twitchtv/retool before running retool do... Both of which are ran in a wrapper; in many projects this would be a (cluttered) Makefile for invoking make check, etc.

I agree with the sentiment though that a package manager should solve this issue. However,

  • I do think it is essential to explore the design space with third party tools first.
  • using just go modules by itself would make go the only programming language I know of that isn't developing a second utility to improve package management. This could be a great thing, but certainly the problem of making a second utility for package management easily run by all developers has been dealt with in other programming languages

There's another desire here that is somewhat related: running any command, not just a tool in the environment that contains tool installs, the packages, and other supported environment modifications. In the stack package manager, this is done with stack exec.

@myitcv

This comment has been minimized.

Copy link
Member Author

commented Oct 11, 2018

I do think it is essential to explore the design space with third party tools first.

Agreed. This is very much the approach being followed by https://github.com/rogpeppe/gohack, too. Depending on the lessons learned, gohack may die, bits may be absorbed back into the go tool.... any number of possible outcomes.

Since you asked, the way I ensure it can be run by all developers is with which retool >/dev/null 2>&1 || go get github.com/twitchtv/retool before running retool do... Both of which are ran in a wrapper; in many projects this would be a (cluttered) Makefile for invoking make check, etc.

I think my main concern is simply that the guidance for every developer globally then becomes:

  1. Install Go - follow the official docs
  2. Install retool - following the docs on this third-party website

at which point you've "broken" one of the key benefits of Go. Not least because this combined guidance will not, by definition, be part of the official docs.

Hence why I'm keen that we get the functionality "interfaces" right here. It's also worth bearing in mind that modules are still officially experimental in 1.11, so there is still time.

@myitcv

This comment has been minimized.

Copy link
Member Author

commented Oct 29, 2018

As something of an experiment in this space, would appreciate feedback etc on https://github.com/myitcv/gobin

@leighmcculloch

This comment has been minimized.

Copy link
Contributor

commented Oct 30, 2018

👏 gobin basically embodies what I expect go install <package> and go run <package> to do. @myitcv is that the intention? If so, it might be worth arranging the tool to operate like that taking install and run commands such that the experience can be experimented with in the way the go tooling would use it.

In regards to the -r option that allows the user to run a specific version of a package, that I assume could be implement something like go run <package>@version, I think it would be simpler to do GOBIN=... go install <module>@version and leave it up to the user to install the binary to a location that won't overlap with other locations because not all tools will be able to be run via the go tooling for a variety of reasons.

@myitcv

This comment has been minimized.

Copy link
Member Author

commented Nov 7, 2018

@leighmcculloch

is that the intention?

A bit more background on what we're trying to experiment with and why given here: https://github.com/myitcv/gobin/wiki/FAQ

In regards to the -r option that allows the user to run a specific version of a package...

(The -r flag is now -run.) The binary that is executed when -run is provided is located in a cache that is guaranteed not to clash (that location is given by -p). So I think that answers your question? If not, probably best to take conversation to an issue over at https://github.com/myitcv/gobin

@rsc

This comment has been minimized.

Copy link
Contributor

commented Nov 20, 2018

Sorry, but I think this is mission creep. It's beyond the scope of the go command to virtualize your whole development environment. The go command is concerned with building code. If you want to manage versions of helper tools and all that, you probably need a more complex build system around it.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Nov 20, 2018

To elaborate on my last comment, another important consideration is that we want Go generally to fit in with the surrounding host system, not try to be its own self-contained bubble. I've worked on self-contained bubbles before, most notably Plan 9, and they can be tremendously elegant and productive. But they are also fundamentally isolating. They make it much harder to use things outside the bubble.

If you want to use tools for development, I want you to use the tools that work best for you, no matter what language they are written in. The protobuf compiler is two binaries, one in C++ and one in Go. You may have other important tools written in C, or Python, or Rust, or any number of other languages. The common denominator is that they are all programs you can exec. I don't want to privilege tools written in Go in some way that creates a bit of a bubble that makes it harder to use tools written in other languages. And that's exactly what I see happening here. The general problem of exec-able tool versioning is not language-specific. It would be best not to give it a language-specific solution. In contrast, the problem of versioning of importable Go packages is absolutely language-specific, so it's reasonable for the go command to have a language-specific solution.

There's probably an interesting system that could be built around versioned bin directories and the like. But that should be a general system, not tied to Go.

@rsc rsc closed this Nov 20, 2018

@gregwebs

This comment has been minimized.

Copy link

commented Nov 20, 2018

I like gobin because it seems to solves 95% of the problem which is tools written in go without requiring other tooling (outside the go bubble!).

I like the concept of being language agnostic even better. The stack build tool (which has several innovations missing in other language build tools) can build in a docker container or a nix environment. So its possible to install all the tools you need into that container or nix environment.

So the solution advocated by @rsc may be satisfieable by nix-shell: https://nixos.org/nixos/nix-pills/developing-with-nix-shell.html

@andig

This comment has been minimized.

Copy link

commented Nov 20, 2018

@rsc I may be missing a point here but I have to disagree.

It is an apparently simple requirement to have a reproducable output path of the generated binary. This will make go depend even less on its bubble of environment variables. Currently thats not possible for tools when using modules [edited] (or for cross-platform builds due to missing support for -o). That in turn leads to complexities around the core tool that should not be necessary.

That said, I only have limited go experience from a couple of smaller projects but have found the whole environment handling to be the most deterring aspect I‘ve encounted (with modules helping a lot to improve).

@myitcv

This comment has been minimized.

Copy link
Member Author

commented Dec 6, 2018

@rsc

There's probably an interesting system that could be built around versioned bin directories and the like. But that should be a general system, not tied to Go.

Thanks for the comprehensive response; I understand and agree with the decision to not do this in the go tool (and hence close this issue).

We will continue with the gobin experiment for now.

@lopezator

This comment has been minimized.

Copy link

commented Jan 9, 2019

There isn't an official solution to simulate the old go get behavior when GO111MODULE=on then? (get code + install binary).

I ended up copying & installing from $GOPATH/pkg/mod

I think I'll use @myitcv gobin experiment.

@leighmcculloch

This comment has been minimized.

Copy link
Contributor

commented Jan 9, 2019

@rsc What's the plan with the behaviour of go get once GOPATH disappears? Will go get error if you run that command in a directory without a go.mod?

@bcmills

This comment has been minimized.

Copy link
Member

commented Jan 9, 2019

go get outside of a module works in module mode in Go 1.12.

Note that this issue is closed. If you are running into a concrete problem that requires a response, please open a new one.

@leighmcculloch

This comment has been minimized.

Copy link
Contributor

commented Jan 9, 2019

@bcmills What's the intended behavior in Go 1.12 of go get outside a module but with modules enabled? Do binaries get installed anywhere? If so, where in the absence of GOPATH?

@bcmills

This comment has been minimized.

Copy link
Member

commented Jan 9, 2019

@leighmcculloch

This comment has been minimized.

Copy link
Contributor

commented Jan 9, 2019

In Go 1.12 the GOPATH is still mentioned in those docs and so I assume binaries will likely be installed in GOPATH/bin. I was asking about when GOPATH disappears entirely. Will go get stop installing binaries?

@dionysius

This comment has been minimized.

Copy link

commented Mar 6, 2019

Erm, I might be a little late and I stumbled upon this issue myself. I am able to run a specific tool with specific version defined in go.mod without "external" tool. All following commands expect GO111MODULE=on

I have to do it this way because of Makefile, you can split the commands to your fit
go list -m -f '{{.Dir}}' github.com/golangci/golangci-lint | xargs -I '{}' go run "{}"/cmd/golangci-lint/main.go run

An example with go:generate
//go:generate sh -c "go run $(go list -m -f '{{.Dir}}' github.com/maxbrunsfeld/counterfeiter/v6)/main.go -- $(go list -m)/$GOPACKAGE.Client"

  • tools.go file as OP described (my example it's just a linter tool), go.mod picks it up
  • go mod download/verify would download it, depending on your steps they're already there
  • no extra cache folder as gobin and retool would do (as far I've seen in examples) - in my opinion just reuse the mod folder...
  • no go mod vendor required, another usually not required export when using go modules...
  • go run also fine for me on a dev tool, as mentioned #27653 (comment)
  • the no "external" tool reason: you can split the command in two, injecting one into another using a variable, except probably depending on os specifics

So basically I wanted a go run way, all I needed was to find a way which path this module is checked out:

go list -m -f '{{.Dir}}' github.com/golangci/golangci-lint
/home/dionysius/Projects/go/pkg/mod/github.com/golangci/golangci-lint@v1.15.0

And then use it as prefix to the corresponding go file. With go run you can still use arguments (as here with the second "run")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.