Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buildGoModule: building Go binaries in two phases #57080

Merged
merged 13 commits into from Mar 14, 2019

Conversation

@kalbasit
Copy link
Member

kalbasit commented Mar 8, 2019

Motivation for this change

This change provides a new builder called buildGoModule for building Go packages with go modules support in mind. It does build through two phases:

  1. The first phase consists of building the go-modules derivation: go mod download will be executed within the source directory to download all of the dependencies, and store them at $GOPATH/pkg/mod. After removing the impure directory $GOPATH/pkg/mod/cache/vcs, we persist $GOPATH/pkg/mod as the $out of this derivation.
  2. The second phase consists of building the actual Go module, and this is done by setting GOPROXY to point to the download directory within ${go-modules}/cache/download which instructs Go to skip any network call, and simply use the already-downloaded dependencies.

The benefit of such infrastructure is that we no longer need tools such as dep2nix or vgo2nix to parse the go.mod and go.sum in order to generate Nix dependency file.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date. Rendered.
  • Fits CONTRIBUTING.md.
  • Converted packages
    • Move over hugo.
    • Move over pet.
    • Move over jx.
    • Move over gotools.
    • Move over terminal-parrot.
    • Move over todoist.
    • Move over hetzner-kube.
  • Create new packages
    • Create mod.

cc @ehmry @lethalman @fpletz @Mic92 @nlewo

nix run -f https://github.com/kalbasit/nur-packages/archive/master.tar.gz nix-verify -c nix-verify -I nixpkgs=`pwd` --keep-going \
	-A jx \
	-A pet \
	-A hugo \
    -A gotools \
    -A terminal-parrot \
    -A todoist \
    -A hetzner-kube \
    -A mod

@kalbasit kalbasit force-pushed the kalbasit:nixpkgs_create-build-go-module branch from 53d9cdf to 53829fb Mar 8, 2019

@kalbasit

This comment was marked as resolved.

Copy link
Member Author

kalbasit commented Mar 8, 2019

I got pet to build this way, but there are still problems to solve:

  • Go relies on $GOPATH to figure out where to store the cache, otherwise, it defaults to $HOME/go. It would be nice to have Go write to a different path instead. This seems to work OK!
  • If $GOPATH/go/pkg/mod/cache does not exist, Go will attempt to reach to network even though all the dependencies are available under $GOPATH/go/pkg/mod. Leaving the $GOPATH/go/pkg/mod/cache will not work, at it contains VCS repositories that will change anytime upstream changes.
  • I ended up removing $GOPATH/go/pkg/mod/cache/vcs and leaving $GOPATH/go/pkg/mod/cache/download. Does the download directory change? I will have to test that.

@kalbasit kalbasit force-pushed the kalbasit:nixpkgs_create-build-go-module branch from 53829fb to 1f21995 Mar 8, 2019

@kalbasit kalbasit force-pushed the kalbasit:nixpkgs_create-build-go-module branch 2 times, most recently from 7fc305e to b3437e2 Mar 8, 2019

@kalbasit

This comment has been minimized.

Copy link
Member Author

kalbasit commented Mar 8, 2019

@GrahamcOfBorg build jx pet hugo gotools

@kalbasit kalbasit force-pushed the kalbasit:nixpkgs_create-build-go-module branch from b3437e2 to a1d010c Mar 8, 2019

@kalbasit kalbasit force-pushed the kalbasit:nixpkgs_create-build-go-module branch from 556fe3f to 6f89cdd Mar 9, 2019

@kalbasit kalbasit marked this pull request as ready for review Mar 9, 2019

@kalbasit kalbasit requested review from Mic92, fpletz and nlewo Mar 9, 2019

@kalbasit

This comment has been minimized.

Copy link
Member Author

kalbasit commented Mar 9, 2019

@GrahamcOfBorg build jx pet hugo gotools

@kalbasit kalbasit force-pushed the kalbasit:nixpkgs_create-build-go-module branch from 48efa69 to c717237 Mar 14, 2019

@sondr3 sondr3 referenced this pull request Mar 14, 2019

Merged

hugo: 0.50 -> 0.54.0 #55915

6 of 10 tasks complete

@kalbasit kalbasit force-pushed the kalbasit:nixpkgs_create-build-go-module branch from c717237 to 627b088 Mar 14, 2019

kalbasit added some commits Mar 8, 2019

buildGoModule: function for packaging Go modules
The function buildGoModule builds Go programs managed with Go modules. It builds
a Go module through a two phase build:

- An intermediate fetcher derivation. This derivation will be used to
  fetch all of the dependencies of the Go module.
- A final derivation will use the output of the intermediate derivation
  to build the binaries and produce the final output.

@kalbasit kalbasit force-pushed the kalbasit:nixpkgs_create-build-go-module branch from af208c4 to 8f8e71c Mar 14, 2019

@kalbasit kalbasit changed the title [DO NOT MERGE] buildGoModule: building Go binaries in two phases buildGoModule: building Go binaries in two phases Mar 14, 2019

@kalbasit

This comment has been minimized.

Copy link
Member Author

kalbasit commented Mar 14, 2019

@Mic92 I've addressed your comments. I have also conducted some tests to verify with certainty that the modules downloaded are not affected by changes to the upstream repository, such as with the release of a new verison. This PR is now clear to be merged as well.

PTAL.

@Mic92

Mic92 approved these changes Mar 14, 2019

@kalbasit kalbasit merged commit a0e4f44 into NixOS:master Mar 14, 2019

10 checks passed

grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details

@kalbasit kalbasit deleted the kalbasit:nixpkgs_create-build-go-module branch Mar 14, 2019

@elasticdog elasticdog referenced this pull request Mar 22, 2019

Merged

bazelisk: init at 0.0.3 #56867

10 of 10 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.