cmd/go: allow versioned go build commands
#44469
Comments
seankhliao
added
GoCommand
|
On #44203, @jayconrod said:
If the plan is indeed to support versioned variants of multiple commands, perhaps we should use a single issue for all of them at once instead of one per command. |
I think we should definitely do that, regardless of whether we add Can we make this issue about that, and discuss generalizing |
|
Assuming the scope of #44203 is not limited to What will be the expected behavior of
which installs the output in |
|
I would expect Although honestly, in my own use I would probably spell that as: |
The downside of that is that it depends on POSIX Shell syntax, so it wouldn't be portable to e.g. Windows. I see little benefit of I do see the point in flags being consistent between commands, but the whole point of |
|
If there were such a flag, I would not want to name it |
|
Maybe. The flip side of the coin is that |
|
|
IMO yes, but I'd like to see what others think. |
|
I think if Agree that it's redundant with |
|
Currently we are using the sequence of
|
|
If it helps, When you say "for all platforms", are you cross-compiling? If |
|
In our specific case, we don't do cross compiling which simplifies things. To me
|
|
Do I have this right? Though I'm not sure when you'd want
|
|
IMO, the |
I would rather we have consistent meanings for flags than make every subcommand completely orthogonal. It's not obvious to me that “installing a differently named binary in the default location” really needs to be convenient. |
|
Thanks @seankhliao. Is there any command that can provide the default installation directory? If so, I am happy to use it to formulate the right |
Within a module, something like If we accept #44203, that would shorten to |
GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html GH-89
GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html GH-89
…#90) GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html Closes GH-89
…#90) GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html Closes GH-89
`go install` with the version specifier is THE way to install a binary in recent versions of Go. From go1.18, `go get` will not do build & install so shouldn't be used. For `dlv-dap` or `gocode-gomod`, we use `go get` + `go build` because `go install` does not allow to specify output path. (golang/go#44469) Use of `go install` allows us to address another issue: we no longer require a dummy main module in a temp directory, and we can run it from the workspace directory. The use of temp directory broke direnv, asdf users who configured their go to pick a different version of go based on the directory. Thus, we pinned the go binary by selecting the go binary from the current GOROOT's bin directory. Unfortunately, that broke another group of users who are using GOROOT for different purpose. (#1691) Now we can remove the hack to pin the go binary and use the go binary path as it is, and simplifies the installation logic. Reduced the verbose logging printed in the console. Instead, we log the command and the directory path so users could reproduce. In case of errors, the exception still includes all the stdout and stderr. Updates #1825 Change-Id: Idb1604e4484cd73832c24c0077220f8cc57dfaa0 Reviewed-on: https://go-review.googlesource.com/c/vscode-go/+/355974 Trust: Hyang-Ah Hana Kim <hyangah@gmail.com> Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Suzy Mueller <suzmue@golang.org>
|
Hi folks, any progress here? Is there a roadmap or something else to implement this feature ? |
|
@rlanhellas subscribe to this thread; you'll get notified when there is an update. This is not in the immediate roadmap, given that we're in a code freeze for 1.18 for at least another month. |
thanks @mvdan , I would like to help implement this feature, can I send a |
|
We're not even sure if we want this feature or how it would work, see the previous discussion and the |
ok @mvdan , thanks for the feedback. |
|
Repeating here the problem I raised earlier in the hope to get a solution or traction: In order to take advantage of the new in 1.18 fantastic But Example: before: CGO_ENABLED=0 GOOS=windows go build -a -ldflags \
'-s -X fortio.org/fortio/version.version=1.28.0 -X "fortio.org/fortio/version.buildInfo=2022-04-26 00:40 57e9d8f01c342a4c1b5d96f883f81128b04d991a"' \
-o /tmp/fortio.exe fortio.org/fortioafter/now GOPATH=/build CGO_ENABLED=0 GOOS=darwin /usr/local/go/bin/go install -a -ldflags -s fortio.org/fortio@v1.29.0-pre16
mv -f /build/bin/*_*/fortio* /build/bin
rmdir /build/bin/*_*I end up having to use GOPATH to try to set where the binary ends up as well as using some hacks to deal with native vs non native arch |
Alternative:allow to change the installed main package name ingo installcommandSince go1.16,
go installaccepts arguments with version suffixes.go installplaces the output in$GOPATH/binor$GOBIN, but does not allow the output binary's name change.When a user wants to use multiple versions of tools, the user need to be careful not to overwrite other versions.
Different versions may be coming from different module paths.
e.g. VS Code Go extension uses two different versions of
gocode(stamblerre/gocode and mdempsky/gocode). In order to install the former, it does switchingGOBINto use a temporary directory, installing it, and renaming and copying it to the realGOBIN.Alternative: There are other proposals for applying the same to
go run(#42088) andgo list(#44203). Extendinggo buildin the same way (or extendinggo install) will help simplifying the tool installation workflow.cc @jayconrod @bcmills @matloob
The text was updated successfully, but these errors were encountered: