Task for Go toolchain install command
#71
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As of Go version 1.16 `go install $pkg@$version` [1] allows to install commands without affecting the `main` module. Additionally commands like `go build` and `go test` no longer modify `go.mod` and `go.sum` files by default but report an error if a module requirement or checksum needs to be added or updated (as if the `-mod=readonly` flag were used). This can be used as alternative to the already existing `gobin` runner [2]. To support the `go install` command of the Go toolchain [3], a new `Task` [4] has been implemented in the new `install` [5] package that can be used through a Go toolchain `Runner` [6]. The task is customizable through the following functions: - `WithEnv(env map[string]string) install.Option` - sets the task specific environment. - `WithModulePath(path string) install.Option` - sets the module import path. - `WithModuleVersion(version *semver.Version) install.Option` - sets the module version. [1]: https://blog.golang.org/go116-module-changes#TOC_4. [2]: https://pkg.go.dev/github.com/svengreb/wand@v0.4.1/pkg/task/gobin [3]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [4]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Task [5]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task/golang/install [6]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task/golang#Runner GH-70
This was referenced Apr 22, 2021
svengreb
added a commit
that referenced
this pull request
Apr 26, 2021
GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html GH-89
svengreb
added a commit
that referenced
this pull request
Apr 26, 2021
GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html GH-89
svengreb
added a commit
that referenced
this pull request
Apr 26, 2021
…#90) GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html Closes GH-89
svengreb
added a commit
that referenced
this pull request
Apr 26, 2021
…#90) GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html Closes GH-89
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #70