New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go build should be able to write multiple executables #14295

Open
kardianos opened this Issue Feb 10, 2016 · 20 comments

Comments

Projects
None yet
@kardianos
Contributor

kardianos commented Feb 10, 2016

Right now there is no easy cross platform way to output multiple artifacts from multiple packages into a single directory.

Right now you can do env GOBIN=bin/ go install cmd/... on many platforms, but not when cross compiling or on windows.

Suggesting making go build -o bin/ pkg1 pkg2 build both packages into the "bin" folder.

@robpike thinks this would make the meaning different based on if it is a file or directory and suggests using a different flag like "-odir". That seems fine however I'm not sure how -o would interact with -odir. If they were forbidden from being set at the same time, I would say they should be the same flag, namely -o.

Reference thread: https://groups.google.com/forum/#!topic/golang-dev/a1wvIZdnic0

@robpike

This comment has been minimized.

Contributor

robpike commented Feb 11, 2016

Flag -o would name the file and -odir would name the directory. Seems clear to me. If both are set one could complain if -o is a rooted path.

Not sure this is worth doing at all, but it seems easy to design and understand.

@kardianos

This comment has been minimized.

Contributor

kardianos commented Feb 11, 2016

@robpike That's a fair design. I think it would be useful to have, but I'm unsure if it would pay for a new flag.

I'm curious if others would also find it useful or if it is just me :).

@lmb

This comment has been minimized.

Contributor

lmb commented Feb 12, 2016

FWIW I have been looking for something like this as well. For company policy reasons I need to build a .deb on a build box. There are several binaries, which I currently build individually and then package up. Not a terrible nuisance, but could probably save me some compile time since the binaries share most code.

@minux

This comment has been minimized.

Member

minux commented Feb 12, 2016

@kardianos

This comment has been minimized.

Contributor

kardianos commented Feb 12, 2016

@minux There are times when I want the functionality of "go install", but placed into a local folder.

  • project specific tools that I don't want to install globally, but are needed for a project.
  • project specific binaries where due to the nature of resources or other design, should be built and ran locally.

The above may come up more often for me then others as I work on a contract basis and juggle different customers accounts. I use a single GOPATH, but often architect projects to run within a non-global context. (Sorry, that might not make complete sense).

@bradfitz bradfitz added this to the Unplanned milestone Apr 9, 2016

@rsc

This comment has been minimized.

Contributor

rsc commented Aug 9, 2018

This is so close to working. I'm not sure why it doesn't. We should fix it for Go 1.12.
Clearly -o can't be used with multiple binary targets, but the implicit "write to the current directory" should work, and instead it does nothing.

$ cd /tmp/zzz
$ go build cmd/doc cmd/gofmt
$ ls -l
$ go build cmd/doc
$ go build cmd/gofmt
$ ls -l
total 14808
-rwxr-xr-x  1 rsc  eng  4266200 Aug  9 15:00 doc
-rwxr-xr-x  1 rsc  eng  3310440 Aug  9 15:00 gofmt
$ 

@rsc rsc modified the milestones: Unplanned, Go1.12 Aug 9, 2018

@rsc rsc changed the title from cmd/go: allow "go build <flag>" to output multiple packages to cmd/go: go build should be able to write multiple executables Aug 9, 2018

@robpike

This comment has been minimized.

Contributor

robpike commented Aug 9, 2018

You could imagine -o or a variant specifying a directory rather than or as well as a file.

@pd93

This comment has been minimized.

pd93 commented Sep 3, 2018

This isn't exactly a necessity, but it would be a huge convenience in my workflow - not to mention it would bring it in line with the install command. I never looked this up in the go docs. I just assumed it would work, but it doesn't 😞

I have a few projects with numerous small tools/utils and it would greatly simplify my testing if I could build them all at once to my cwd. Currently I have a bash script that runs consecutive go build ./path/to/tool commands which is less than ideal.

@neilpa

This comment has been minimized.

neilpa commented Sep 4, 2018

Same here, we do most of our local dev on mac but than build and package for our standard linux image so have have similar problems since the GOBIN=... go install ... trick doesn't work. I've hacked together Makefiles to support cross-compiling with explicit go build -o ... for each target binary as a workaround. You could probably take this further with wildcard magic over the cmd/ directory but I decided I like readable makefiles.

This works for now but would prefer a go build or go install solution.

@gopherbot

This comment has been minimized.

gopherbot commented Oct 18, 2018

Change https://golang.org/cl/143139 mentions this issue: cmd/go: make go build write multiple executables

@rasky

This comment has been minimized.

Member

rasky commented Oct 18, 2018

One side effect of this CL is that it breaks the venerable "go build -a std cmd" for Go maintainers (when run from $GOROOT/src, it will complain that "go" is already a directory; when run from any different directory, it will drop many binaries in it). The current workaround is "go build -o /dev/null -a std cmd".

Comments? /cc @rsc @robpike

@kardianos

This comment has been minimized.

Contributor

kardianos commented Oct 24, 2018

There is a bit of discussion on the proposed CL, as well s the comment that the CL as implemented breaks go build -a std cmd. I would also comment the current CL breaks a personal use case I sometimes run go build ./cmd/... just to make sure everything my cmds rely on still build as a quick smoke test while in development.

I would make the following proposal:

Make go build without -o and multiple packages specified behave as before. Only output binaries from go build with multiple packages are specified when -o is specified. This prevents go build -a std cmd from breaking, preserves all existing behavior, while adding useful method such as go build -o ./bin/ ./cmd/.... Either that, or we could teach go install a -o switch that points to the output directory.


In the CL there are comments that suggest either:
go build $(go list -f '{{if eq .Name "main"}}{{.ImportPath}}{{end}}' ./...)
or for pkg in $(go list -f '{{if eq .Name "main"}}{{.ImportPath}}{{end}}' ./...); go build $pkg; end . Yeah, that's missing the point. I can get around this behavior, On *nix I can have a couple of helper scripts and on Windows another helper scripts, or I could make it part of my build system for individual projects. Again, that's kinda missing the point.

Sometimes it would be really useful to install a bunch of commands into a specific directory. Most of the semantics are already defined.

@rasky

This comment has been minimized.

Member

rasky commented Oct 24, 2018

Another option is to add an explicit -dryrun option (or similar name) to avoid writing binaries; similar to the undocumented -o /dev/null but more explicit. This would help those who want to “just test it compiles”, in an explicit way.

Then, building multiple executables would just work, by default. This would make the behavior of go build more consistent; by default, it would always build executables, rather than changing behavior depending on how many packages were specified. It would be easier to explain and document, and less surprising for beginners.

@rasky

This comment has been minimized.

Member

rasky commented Nov 4, 2018

Ping, can we get this unstuck? It just needs a decision on a small detail, we all want the main feature in.
/cc @andybons

@bcmills

This comment has been minimized.

Member

bcmills commented Nov 13, 2018

go build ./... is useful today, and its meaning should not change. We can either take the -o approach that @kardianos suggests, or require an explicit list of targets per my comment on the CL. I'll talk to @rsc to decide which.

@bcmills

This comment has been minimized.

Member

bcmills commented Nov 16, 2018

@kardianos In module mode, it may be useful to issue a go get command outside of any module (#24250) with the path to a binary plus some specific versions of dependencies to upgrade (or downgrade). In those cases, it seems redundant to require the user to pass -o $GOBIN or -o $GOPATH/bin explicitly just to get the binary to install in the usual place.

@kardianos

This comment has been minimized.

Contributor

kardianos commented Nov 16, 2018

@bcmills Thank you for working on defining how go get works outside of any module. I think I may be missing something. Here's how I could envision the go command working in module mode:

## Outside of a module

# In all cases, if $GOBIN is not defined, it assumes $GOBIN is $HOME/bin

# Outside a module. Fetches cmd1 and cmd2, compiles it, and installs it in $GOBIN.
go get github.com/user/repo/cmd/...

# (maybe) Outside a module. Fetches cmd1 and cmd2 installs in ./bin.
go get -o ./bin github.com/user/repo/cmd/...

# Outside a module. Fails.
go build github.com/user/repo/cmd/...

# (maybe) Outside a module. Builds cmd1 and cmd1 and outputs to ./bin
go build -o ./bin /home/user1/code/moduleroot/cmd/...

# Outside a module. Fails.
go install github.com/user/repo/cmd/...

# (maybe) Outside a module. Builds cmd1 and cmd1 and outputs to $GOBIN.
go install /home/user1/code/moduleroot/cmd/...

## Inside a module.

# Inside a module. Fetches cmd1 and cmd2, constrains get to local go.mod file, installs in $GOBIN.
go get github.com/user/repo/cmd/...

# (maybe) Inside a module. Fetches cmd1 and cmd2, constrains get to local go.mod file, installs in ./bin.
go get -o ./bin github.com/user/repo/cmd/...

# Inside a module, builds the local commands cmd1 and cmd2, and discards the results
go build ./cmd/...

# Inside a module. Installs cmd1 and cmd2 in ./bin.
go build -o ./bin ./cmd/...

# (maybe) Inside a module. Installs cmd1 in ./bin (assuming bin is a folder).
# This would be useful in reusing the same script for windows and linux.
go build -o ./bin ./cmd/cmd1

# Inside a module, builds the local commands cmd1 and cmd2, installs in $GOBIN.
go install ./cmd/...

Can you explain why you would need to pass in -o $GOBIN to go get with this change?

@rsc

This comment has been minimized.

Contributor

rsc commented Nov 20, 2018

We may all want this "in", but it's too late for Go 1.12.

@rsc rsc modified the milestones: Go1.12, Go1.13 Nov 20, 2018

@rsc rsc added the early-in-cycle label Nov 20, 2018

@rasky

This comment has been minimized.

Member

rasky commented Nov 20, 2018

This is very sad.

It is the second release that this feature is postponed because the community doesn't get answers from the Go team regarding it in time for the release cycle (see also #23616 where I didn't get an answer for months). I have even sent a CL well in time, to try to unblock things.

How can we break this cycle? Is there a better channel than this? The community can't do development for things we care about if the Go team doesn't even prioritize answering to questions. /cc @andybons @bradfitz @bcmills @FiloSottile @cassandraoid

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Nov 20, 2018

I'm sorry this one got stuck. As you know there has been a great deal of work on cmd/go last cycle and this one, and that has made looking at CLs for cmd/go even slower than usual. Also it seems that this feature interacts with modules, which makes it even worse since modules is in a steady state of flux.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment