Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go build should be able to write multiple executables #14295

Closed
kardianos opened this issue Feb 10, 2016 · 31 comments

Comments

@kardianos
Copy link
Contributor

commented Feb 10, 2016

Right now there is no easy cross platform way to output multiple artifacts from multiple packages into a single directory.

Right now you can do env GOBIN=bin/ go install cmd/... on many platforms, but not when cross compiling or on windows.

Suggesting making go build -o bin/ pkg1 pkg2 build both packages into the "bin" folder.

@robpike thinks this would make the meaning different based on if it is a file or directory and suggests using a different flag like "-odir". That seems fine however I'm not sure how -o would interact with -odir. If they were forbidden from being set at the same time, I would say they should be the same flag, namely -o.

Reference thread: https://groups.google.com/forum/#!topic/golang-dev/a1wvIZdnic0

@robpike

This comment has been minimized.

Copy link
Contributor

commented Feb 11, 2016

Flag -o would name the file and -odir would name the directory. Seems clear to me. If both are set one could complain if -o is a rooted path.

Not sure this is worth doing at all, but it seems easy to design and understand.

@kardianos

This comment has been minimized.

Copy link
Contributor Author

commented Feb 11, 2016

@robpike That's a fair design. I think it would be useful to have, but I'm unsure if it would pay for a new flag.

I'm curious if others would also find it useful or if it is just me :).

@lmb

This comment has been minimized.

Copy link
Contributor

commented Feb 12, 2016

FWIW I have been looking for something like this as well. For company policy reasons I need to build a .deb on a build box. There are several binaries, which I currently build individually and then package up. Not a terrible nuisance, but could probably save me some compile time since the binaries share most code.

@minux

This comment has been minimized.

Copy link
Member

commented Feb 12, 2016

@kardianos

This comment has been minimized.

Copy link
Contributor Author

commented Feb 12, 2016

@minux There are times when I want the functionality of "go install", but placed into a local folder.

  • project specific tools that I don't want to install globally, but are needed for a project.
  • project specific binaries where due to the nature of resources or other design, should be built and ran locally.

The above may come up more often for me then others as I work on a contract basis and juggle different customers accounts. I use a single GOPATH, but often architect projects to run within a non-global context. (Sorry, that might not make complete sense).

@bradfitz bradfitz added this to the Unplanned milestone Apr 9, 2016
@rsc

This comment has been minimized.

Copy link
Contributor

commented Aug 9, 2018

This is so close to working. I'm not sure why it doesn't. We should fix it for Go 1.12.
Clearly -o can't be used with multiple binary targets, but the implicit "write to the current directory" should work, and instead it does nothing.

$ cd /tmp/zzz
$ go build cmd/doc cmd/gofmt
$ ls -l
$ go build cmd/doc
$ go build cmd/gofmt
$ ls -l
total 14808
-rwxr-xr-x  1 rsc  eng  4266200 Aug  9 15:00 doc
-rwxr-xr-x  1 rsc  eng  3310440 Aug  9 15:00 gofmt
$ 
@rsc rsc modified the milestones: Unplanned, Go1.12 Aug 9, 2018
@rsc rsc changed the title cmd/go: allow "go build <flag>" to output multiple packages cmd/go: go build should be able to write multiple executables Aug 9, 2018
@robpike

This comment has been minimized.

Copy link
Contributor

commented Aug 9, 2018

You could imagine -o or a variant specifying a directory rather than or as well as a file.

@pd93

This comment has been minimized.

Copy link

commented Sep 3, 2018

This isn't exactly a necessity, but it would be a huge convenience in my workflow - not to mention it would bring it in line with the install command. I never looked this up in the go docs. I just assumed it would work, but it doesn't 😞

I have a few projects with numerous small tools/utils and it would greatly simplify my testing if I could build them all at once to my cwd. Currently I have a bash script that runs consecutive go build ./path/to/tool commands which is less than ideal.

@neilpa

This comment has been minimized.

Copy link

commented Sep 4, 2018

Same here, we do most of our local dev on mac but than build and package for our standard linux image so have have similar problems since the GOBIN=... go install ... trick doesn't work. I've hacked together Makefiles to support cross-compiling with explicit go build -o ... for each target binary as a workaround. You could probably take this further with wildcard magic over the cmd/ directory but I decided I like readable makefiles.

This works for now but would prefer a go build or go install solution.

@gopherbot

This comment has been minimized.

Copy link

commented Oct 18, 2018

Change https://golang.org/cl/143139 mentions this issue: cmd/go: make go build write multiple executables

@rasky

This comment has been minimized.

Copy link
Member

commented Oct 18, 2018

One side effect of this CL is that it breaks the venerable "go build -a std cmd" for Go maintainers (when run from $GOROOT/src, it will complain that "go" is already a directory; when run from any different directory, it will drop many binaries in it). The current workaround is "go build -o /dev/null -a std cmd".

Comments? /cc @rsc @robpike

@kardianos

This comment has been minimized.

Copy link
Contributor Author

commented Oct 24, 2018

There is a bit of discussion on the proposed CL, as well s the comment that the CL as implemented breaks go build -a std cmd. I would also comment the current CL breaks a personal use case I sometimes run go build ./cmd/... just to make sure everything my cmds rely on still build as a quick smoke test while in development.

I would make the following proposal:

Make go build without -o and multiple packages specified behave as before. Only output binaries from go build with multiple packages are specified when -o is specified. This prevents go build -a std cmd from breaking, preserves all existing behavior, while adding useful method such as go build -o ./bin/ ./cmd/.... Either that, or we could teach go install a -o switch that points to the output directory.


In the CL there are comments that suggest either:
go build $(go list -f '{{if eq .Name "main"}}{{.ImportPath}}{{end}}' ./...)
or for pkg in $(go list -f '{{if eq .Name "main"}}{{.ImportPath}}{{end}}' ./...); go build $pkg; end . Yeah, that's missing the point. I can get around this behavior, On *nix I can have a couple of helper scripts and on Windows another helper scripts, or I could make it part of my build system for individual projects. Again, that's kinda missing the point.

Sometimes it would be really useful to install a bunch of commands into a specific directory. Most of the semantics are already defined.

@rasky

This comment has been minimized.

Copy link
Member

commented Oct 24, 2018

Another option is to add an explicit -dryrun option (or similar name) to avoid writing binaries; similar to the undocumented -o /dev/null but more explicit. This would help those who want to “just test it compiles”, in an explicit way.

Then, building multiple executables would just work, by default. This would make the behavior of go build more consistent; by default, it would always build executables, rather than changing behavior depending on how many packages were specified. It would be easier to explain and document, and less surprising for beginners.

@rasky

This comment has been minimized.

Copy link
Member

commented Nov 4, 2018

Ping, can we get this unstuck? It just needs a decision on a small detail, we all want the main feature in.
/cc @andybons

@bcmills

This comment has been minimized.

Copy link
Member

commented Nov 13, 2018

go build ./... is useful today, and its meaning should not change. We can either take the -o approach that @kardianos suggests, or require an explicit list of targets per my comment on the CL. I'll talk to @rsc to decide which.

@bcmills

This comment has been minimized.

Copy link
Member

commented Nov 16, 2018

@kardianos In module mode, it may be useful to issue a go get command outside of any module (#24250) with the path to a binary plus some specific versions of dependencies to upgrade (or downgrade). In those cases, it seems redundant to require the user to pass -o $GOBIN or -o $GOPATH/bin explicitly just to get the binary to install in the usual place.

@kardianos

This comment has been minimized.

Copy link
Contributor Author

commented Nov 16, 2018

@bcmills Thank you for working on defining how go get works outside of any module. I think I may be missing something. Here's how I could envision the go command working in module mode:

## Outside of a module

# In all cases, if $GOBIN is not defined, it assumes $GOBIN is $HOME/bin

# Outside a module. Fetches cmd1 and cmd2, compiles it, and installs it in $GOBIN.
go get github.com/user/repo/cmd/...

# (maybe) Outside a module. Fetches cmd1 and cmd2 installs in ./bin.
go get -o ./bin github.com/user/repo/cmd/...

# Outside a module. Fails.
go build github.com/user/repo/cmd/...

# (maybe) Outside a module. Builds cmd1 and cmd1 and outputs to ./bin
go build -o ./bin /home/user1/code/moduleroot/cmd/...

# Outside a module. Fails.
go install github.com/user/repo/cmd/...

# (maybe) Outside a module. Builds cmd1 and cmd1 and outputs to $GOBIN.
go install /home/user1/code/moduleroot/cmd/...

## Inside a module.

# Inside a module. Fetches cmd1 and cmd2, constrains get to local go.mod file, installs in $GOBIN.
go get github.com/user/repo/cmd/...

# (maybe) Inside a module. Fetches cmd1 and cmd2, constrains get to local go.mod file, installs in ./bin.
go get -o ./bin github.com/user/repo/cmd/...

# Inside a module, builds the local commands cmd1 and cmd2, and discards the results
go build ./cmd/...

# Inside a module. Installs cmd1 and cmd2 in ./bin.
go build -o ./bin ./cmd/...

# (maybe) Inside a module. Installs cmd1 in ./bin (assuming bin is a folder).
# This would be useful in reusing the same script for windows and linux.
go build -o ./bin ./cmd/cmd1

# Inside a module, builds the local commands cmd1 and cmd2, installs in $GOBIN.
go install ./cmd/...

Can you explain why you would need to pass in -o $GOBIN to go get with this change?

@rsc

This comment has been minimized.

Copy link
Contributor

commented Nov 20, 2018

We may all want this "in", but it's too late for Go 1.12.

@rsc rsc modified the milestones: Go1.12, Go1.13 Nov 20, 2018
@rsc rsc added the early-in-cycle label Nov 20, 2018
@rasky

This comment has been minimized.

Copy link
Member

commented Nov 20, 2018

This is very sad.

It is the second release that this feature is postponed because the community doesn't get answers from the Go team regarding it in time for the release cycle (see also #23616 where I didn't get an answer for months). I have even sent a CL well in time, to try to unblock things.

How can we break this cycle? Is there a better channel than this? The community can't do development for things we care about if the Go team doesn't even prioritize answering to questions. /cc @andybons @bradfitz @bcmills @FiloSottile @cassandraoid

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Nov 20, 2018

I'm sorry this one got stuck. As you know there has been a great deal of work on cmd/go last cycle and this one, and that has made looking at CLs for cmd/go even slower than usual. Also it seems that this feature interacts with modules, which makes it even worse since modules is in a steady state of flux.

@kardianos

This comment has been minimized.

Copy link
Contributor Author

commented Feb 16, 2019

@bcmills Let me know if there is something (proposal or code) that could make this move in 1.13.

Another use case I encountered today is building a main package in a different dir, without needing to worry about if I should append ".exe" at the end go build -o ./bin/ ./cmd/thecmd.

@DisposaBoy

This comment has been minimized.

Copy link

commented Feb 16, 2019

Another use case I encountered today is building a main package in a different dir, without needing to worry about if I should append ".exe" at the end go build -o ./bin/ ./cmd/thecmd.

@kardianos For that specific use-case, you might find go env GOEXE helpful.

~ $ env GOOS=linux go env GOEXE

~ $ env GOOS=windows go env GOEXE
.exe
@kardianos

This comment has been minimized.

Copy link
Contributor Author

commented Feb 17, 2019

I am aware of that. But again, I'm talking about slightly nicer default behavior.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Mar 12, 2019

To move forward with this, let's make -o with an argument that is a directory write a new executable to that directory. Note that the key thing is that the argument is a directory, not that it syntactically ends in slash. That is, go build -o becomes more like mv, not rsync.

Separately, go build with multiple command targets should be able to write multiple binaries to the current directory.

@gopherbot

This comment has been minimized.

Copy link

commented Mar 15, 2019

Change https://golang.org/cl/167679 mentions this issue: cmd/go: allow -o to point to a folder that writes multiple execs

@FiloSottile

This comment has been minimized.

Copy link
Member

commented Aug 19, 2019

It looks like only -o behavior was changed, but go build doesn't yet build multiple packages to the current directory by default, so reopening this.

Separately, go build with multiple command targets should be able to write multiple binaries to the current directory.

@FiloSottile FiloSottile reopened this Aug 19, 2019
@FiloSottile FiloSottile modified the milestones: Go1.13, Go1.14 Aug 19, 2019
@gopherbot

This comment has been minimized.

Copy link

commented Aug 19, 2019

Change https://golang.org/cl/190839 mentions this issue: cmd/go: fix "go help build -o" docs

@rasky

This comment has been minimized.

Copy link
Member

commented Aug 19, 2019

For reference, my CL 143139 did work also without -o. Not sure why it was redone from scratch.

@kardianos

This comment has been minimized.

Copy link
Contributor Author

commented Aug 20, 2019

Sorry for the scattered discussion. CL comment https://go-review.googlesource.com/c/go/+/143139/4#message-fb8adc4368e18ba081fe2fe221448bce17d990bc showed that go build ./... that outputs executables (no -o) breaks things both in current std lib tests, and I'm sure in many tools that rely on that to discard output. Requiring -o . to output to the current directory keeps the current command line compatibility, while still gaining the functionality.

@FiloSottile

This comment has been minimized.

Copy link
Member

commented Aug 20, 2019

Makes sense, thanks for closing the loop.

gopherbot pushed a commit that referenced this issue Aug 20, 2019
The docs refer to "the last two paragraphs", but in fact should refer to
the first two of the previous three paragraphs. Moved up the out of place
paragraph.

Updates #14295

Change-Id: I066da7a665bc6754d246782b941af214a385017a
Reviewed-on: https://go-review.googlesource.com/c/go/+/190839
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Aug 20, 2019

Change https://golang.org/cl/190907 mentions this issue: doc/go1.13: mention '-o <directory>' support for 'go build'

gopherbot pushed a commit that referenced this issue Aug 20, 2019
Fixes #33720
Updates #14295

Change-Id: I9cb6e02bcaccd7971057315163d8810157d465bd
Reviewed-on: https://go-review.googlesource.com/c/go/+/190907
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
t4n6a1ka added a commit to t4n6a1ka/go that referenced this issue Sep 5, 2019
The docs refer to "the last two paragraphs", but in fact should refer to
the first two of the previous three paragraphs. Moved up the out of place
paragraph.

Updates golang#14295

Change-Id: I066da7a665bc6754d246782b941af214a385017a
Reviewed-on: https://go-review.googlesource.com/c/go/+/190839
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
t4n6a1ka added a commit to t4n6a1ka/go that referenced this issue Sep 5, 2019
Fixes golang#33720
Updates golang#14295

Change-Id: I9cb6e02bcaccd7971057315163d8810157d465bd
Reviewed-on: https://go-review.googlesource.com/c/go/+/190907
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.