Recommend buildGoPackage instead of buildGoModule in the nixpkgs manual

[nlewo]

In the nixpkgs manual, the default function to package a Go application is buildGoModule (since the release 19.03). This function creates one derivation to fetch all dependencies. This is really convenient to use since we only have to provide one hash: the hash of a directory containing all of these dependencies.

However, it has several drawbacks:

• long term reproducibility is harder to achieve because we don't know exactly what sources (revision/checksum) are used by this application (sources mirroring is harder)
• source dependencies are not shared across packages
• users can not know the source dependencies of a package

There is also a Nix issue to restrict fixed-output derivations: it would no longer be possible to create these kind derivations.

We have another function which is buildGoPackage. To use it, we first need to run a tool to generate a derivation per dependency. This is less convenient to use, but cleaner since we are exposing all dependencies, and not only a blob of sources. Unfortunately, this function is tagged as "legacy" in the manual:(

So, I think it would be better to:

1. recommend buildGoPackage to package Go applications,
2. and if buildGoPackage doesn't work, then fallback to buildGoModule.

I think the buildGoModule is a really nice function to quickly package an application. It has to be in the nixpkgs manual, but we should avoid it to package applications in nixpkgs.

I don't know how could I proceed on this. Should I "just" create a PR to upload the manual? WDYT?

Note this could also be applied to buildRustPackage versus Carnix/Crate2Nix.

[doronbehar]

Really? I always thought as a reviewer it was the other way around! Since buildGoPackage is documented as a function to build "legacy" Go programs, not supporting Go modules. I thought Nixpkgs considers Go modules and buildGoModule as the future...

[zimbatm]

For the completeness's sake, there are two main advantages of using buildGoModule:

1. there is much less generated code stored inside of nixpkgs. The larger nixpkgs is, the longer it takes to load a channel.
2. less evaluation is happening while accessing the derivation.

These seem to go against the goal of having all the sources declared though so it has to be a trade-off unless another road is discovered. Point (2) can be mitigated by using the recursive nix feature once it becomes widely available.

[arianvp]

Thank you for bringing this up!

I have had PRs rejected before because of buildRustPackage (which is the rust equivalent of buildGoModule) but rust has the added downside there seems to be no good alternative (there isn't a vetted 2nix tool yet). I would really like an official stance on this as it's hindering packaging of certain applications that I'd love to package but they've been simply blocked by the fact " This uses Fixed output derivations, you can't merge it" or on the other hand "your x2nix tool generates too many nix files, which is slow, you can't merge it"

#68702 (comment)

I've also had PRs rejected for doing the opposite (In this case it was a Node project):

#49082 (comment)

So far this split with no clear decision about fixed-output derivations and 2nix tools is rather frustrating as a contributor

[emilazy]

The nixpkgs documentation implies that buildGoPackage is legacy and doesn't support Go modules, which has made me use buildGoModule in the past, but that doesn't seem quite true: vgo2nix lets us generate deps.nix files from go.mod.

My personal opinion is that I'd happily take an inflated nixpkgs source tree over the flakiness and questionable purity of fixed-output derivations. Ideally we'd have true recursive Nix and could properly integrate with third-party package and build systems...

[zimbatm]

Thanks for bringing this up @arianvp. The positions are clearly untenable if both approaches are being rejected and no alternative is being proposed. Are there any other possible roads that we can take?

[tomberek]

buildGoModule also has a hard time with private repos.

[doronbehar]

Guix shares Go modules and Rust crates for all projects:

https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/crates-io.scm?id=f87fa003617fe990bef4005800a9f40726494b25

https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/golang.scm?id=073c64dc792c79c2a988edae9f4e568d3b09f3b5

[zimbatm]

We know from experience with the python packages that having one version of each dependency doesn't scale. It also means that we are not using the versions of dependencies that were tested by upstream.

That being said, it might be possible to generate a big index of all the dependencies and gain some compression by doing that.

[c00w]

long term reproducibility is harder to achieve because we don't know exactly what sources (revision/checksum) are used by this application (sources mirroring is harder)

The source dependencies are built into a giant bundle of info inside the module download derivation - you have a fully copy of the source in module fetcher derivation. If that's not enough, we could modify it to generate a vendor directory which would also include the full raw source.

source dependencies are not shared across packages

Are they with goBuildPackage?

users can not know the source dependencies of a package

See answer to first question - but I don't think this is true.

CC @kalbasit

[c00w]

and if buildGoPackage doesn't work, then fallback to buildGoModule.

Every single package which builds with buildGoModule should also build with buildGoPackage if you download the same sources AFAIK, this sounds like we're proposing killing buildGoModule.

[c00w]

PR rejections

This sounds pretty painful + ridiculous, I've definitely had conflicting reviewer comments, but nothing so bad that PR's got rejected :(

We have another function which is buildGoPackage. To use it, we first need to run a tool to generate a derivation per dependency. This is less convenient to use, but cleaner since we are exposing all dependencies, and not only a blob of sources.

The ergonomics of a go.deps file are really painful for a go perspective. It makes it so that using go is significantly more taxing, since you have to double specify your dependencies. I guess I could use triply recursive nix in my personal setup, but It would probably change my advice of "oh you're trying to deploy some go code to your server, nix makes this 10 lines of code" to "well you can do it, but honestly it's painful and you'll need to do this multi layered thing to generate some nix files to feed into other nix files and then it'll build"

[c00w]

My personal opinion is that I'd happily take an inflated nixpkgs source tree over the flakiness and questionable purity of fixed-output derivations.

Amusingly enough, go module downloads are actually perfectly reproducible. There is a public certificate transparency log that will alert if you get two downloads with different contents, every go project using go modules, also stores it's local hashes in the repo, so you'll detect conflicts there as well. There was a nix bug with some flakiness initially, but this has all been fixed - if you're still seeing it please let me know. The bug is definitely in nix land, not go :)

Hilariously enough, as I've been auto converting some code from buildGoPackages to buildGoModules, I'm seeing that a bunch of the source hashes in buildGoPackages are incorrect, and don't match what upstream is serving. So I think go modules are doing a better job than nix in terms of making sure you build with what you expect.

[c00w]

Thanks for bringing this up @arianvp. The positions are clearly untenable if both approaches are being rejected and no alternative is being proposed. Are there any other possible roads that we can take?

One middle of the road approach is to modify buildGoModule so you can specify your hashes per dependency, rather than for the who go file. This would satisfy concerns around deduplication, while if we allowed the overall hash as an alternative keep the nice ergonomics. You would probably have to start downloading all go source code from the centralized mirror, and do a mapping to the http download blob, but it should be possible to do without requiring reentrant nix. You would probably have to also specify the version.

[tomberek]

To add a datapoint: a modification to vgo2nix to obtain the full git hash and store the sha256 is enough to allow bulitings.fetchGit to be used in buildGoPackage (almost always, there are edge cases). My specific use case was to allow private repos for a project that required the versions in go.mod, and accepting the non-module dependency resolution was not an option.

nix-community/vgo2nix#34

[c00w]

To add a datapoint: a modification to vgo2nix to obtain the full git hash and store the sha256 is enough to allow bulitings.fetchGit to be used in buildGoPackage (almost always, there are edge cases). My specific use case was to allow private repos for a project that required the versions in go.mod, and accepting the non-module dependency resolution was not an option.

So private repos like this should be fully suported in go modules. If you have a package called bar, stored at a repo at foo.git/thing/foobar, then you can put a replace directive in your go.mod file and everything should work. If buildGoModules doesn't work (and I don't see why it wouldn't since it just reuses the normal go code), let me know and I'd be happy to debug.

More details can be seen at
https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive

You may also want to set
GOPRIVATE= since go will log your code hash + names to the transparency (sum) database otherwise.

https://github.com/golang/go/wiki/Modules#go-113

[nlewo]

The source dependencies are built into a giant bundle of info inside the module download derivation - you have a fully copy of the source in module fetcher derivation. If that's not enough, we could modify it to generate a vendor directory which would also include the full raw source.

The difference is that in one case you know the sources at evaluation time, while in the case of the giant bundle, you need to build this giant bundle derivation, and then writing something to extract urls: you can not write a Nix expression to generate this list of sources.
For instance, this information allows Nix to fallback on mirrors if the original source is no longer available (see the nix.conf option "hashed-mirrors" in the Nix manual). My long term plan is to use Software Heritage as a generic fallback mirror.

source dependencies are not shared across packages
Are they with goBuildPackage?

Since they are in separated derivations, yes they are (when they share the same sha256).

Every single package which builds with buildGoModule should also build with buildGoPackage if you download the same sources AFAIK, this sounds like we're proposing killing buildGoModule.

I often hit issues with tools generating the deps.nix file in the past. In this kind of situations, I think we should be able to use buildGoModule in nixpkgs: my feeling is that in practice, we would still sometimes need buildGoModule.

"oh you're trying to deploy some go code to your server, nix makes this 10 lines of code" to "well you can do it, but honestly it's painful and you'll need to do this multi layered thing to generate some nix files to feed into other nix files and then it'll build"

I'm definitely in favor of keeping the buildGoModule function in nixpkgs because it's more convenient than buildGoPackage. So, if someone wants to package a Go app for its own needs, this person could use the buildGoModule function. However, for a Go app to nixpkgs, I would prefer to recommend buildGoPackage.

Amusingly enough, go module downloads are actually perfectly reproducible.

If the upstream source is no longer available (when a repository is removed from GitHub for instance), I don't think you can still download it (I don't know if the Go community maintain a mirror of GitHub). This could explain why a lot of Go projects "vendorize" their deps in their own repository:/

[c00w]

The difference is that in one case you know the sources at evaluation time, while in the case of the giant bundle, you need to build this giant bundle derivation, and then writing something to extract urls: you can not write a Nix expression to generate this list of sources.

I think you have the ordering here inverted - you list the required modules for a go module by just looking at the go.mod file. The bundle derivation just fetches them :)

I still don't fully understand why you want a list of sources here, but you can pop this out of a derivation pretty easily by importing the go source, then running one go command to list all the items to download.

The other thing that's worth pointing out is that go sources are in a lot of formats - the go specific format, git, svn, and maybe tar(?). I expect more formats will keep getting added here. I don't think there is a lot of value in having to reimplement the go download logic in nix + keep that up to date as stuff changes. It's going to cause some significant churn and be super frustrating when trying to change stuff. I guess you could maybe only implement the go generic download format, and then only download through the proxy API, but then you're losing the ability to download from the source, and the go tool already downloads from the proxies so it's not significantly better.

The difference is that in one case you know the sources at evaluation time, while in the case of the giant bundle, you need to build this giant bundle derivation, and then writing something to extract urls: you can not write a Nix expression to generate this list of sources.
For instance, this information allows Nix to fallback on mirrors if the original source is no longer available (see the nix.conf option "hashed-mirrors" in the Nix manual). My long term plan is to use Software Heritage as a generic fallback mirror.

buildGoModule already does this by using modern go tooling - it first hits the mirror at proxy.golang.org, then flips over to directly trying the source (github etc...). I don't think I've ever seen this fail on the reliability front, but if we're worried, we could add the other 6 mirrors that are listed at
https://github.com/golang/go/wiki/Modules#are-there-always-on-module-repositories-and-enterprise-proxies
But honestly it's pretty unlikely the source and proxy.golang.org are down at the same time. If we have a lot of users in china, adding the china mirror to get around censorship might be useful, but I expect that the nix caches aren't that available in china anyway. If doing these builds were in a realtime serving path for me I'd probably add a 3rd version, but hydra is pretty tolerant here + not critical.

I often hit issues with tools generating the deps.nix file in the past. In this kind of situations, I think we should be able to use buildGoModule in nixpkgs: my feeling is that in practice, we would still sometimes need buildGoModule.

Why? If the tools break we should just hand write it while we wait for the tools to fix. I think switching to a nix format that you're not willing to write by hand probably indicates that we shouldn't be using it. I remember when I was using buildGoPackage for my personal stuff it literally took me a few weeks to figure out how to inject local folders into it, and I had to spend a silly amount of time reading it's implementation.

If the upstream source is no longer available (when a repository is removed from GitHub for instance), I don't think you can still download it (I don't know if the Go community maintain a mirror of GitHub).

https://proxy.golang.org/ should have an indefinite copy of literally anything that's been downloaded by a vanilla version of go. It's not just github specific :) There are another 6 proxies run by other parties if you're worried about that specific proxy.

This could explain why a lot of Go projects "vendorize" their deps in their own repository:/

I don't think I've ever seen a go dependency vanishing. Normally vendoring was done since it overrides your go path and makes multi person collaboration a lot easier if people have different versions of dependencies (this was a hugely frustrating issue in the early days, which was why lots of people ended up just using a Makefile and putting their whole go workplace in a repo). This has been superseeded by modules + vendor folders appear to be getting deleted. I wouldn't be surprised if vendor support is gone by go 0.16 (but I'm just guessing).

[Mic92]

I would suggest to add an optional goDeps to buildGoModule so that we can switch back and forth between a generated deps.nix and a modSha256 attribute based on the package. I am currently looking into vgo2nix to make it fit for that use case.

[emilazy]

Related: #86349

[Mic92]

@nlewo also made some progress on getting vgo2nix to work nicely with buildGoModule: nlewo@09ac010#diff-5cc2ec5eab07fa4ee4e3ba57846f5134R53 This means we no longer have evil fixed-input derivations.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/fixed-output-derivations-to-become-part-of-flake-inputs/8263/1

[blaggacao]

Can't we just write an adapter (parser + transponder) to nix so that it can instantiate and evaluate go.mod properly.

... coming from the serious doubt, that there would be any bits of information missing in go.mod.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/buildgomodule-building-submodule/8911/2

[Mic92]

Can't we just write an adapter (parser + transponder) to nix so that it can instantiate and evaluate go.mod properly.

... coming from the serious doubt, that there would be any bits of information missing in go.mod.

yes: https://github.com/tweag/gomod2nix

[blaggacao]

Hm that tool seems to be in the business of making a 190 lines long file out of a 11 lines file. Judging by the effort put into it, there must be some very serious benefit to doing this, that I can't (yet) see 🤷

I'd have had a builtins.fromGoModAndSum in mind (if that makes sense).

[zimbatm]

Go's lockfile system doesn't translate to Nix so we are forced to take an extra step. Unlike say npm which has a JSON that nix can parse at eval time, and which contains all the sha256 in a compatible format.

[blaggacao]

Oh yes, I can indeed see that.

I was just wondering if a bultins.fromGoMod (akin to a builtins.fromJson) would have been a) conceptionally and b) "implememtationally" cheaper.

But I guess, I'd have had (once again) disregarded the implicit cost of reaching consensus about builtins.fromGoMod in that reasoning.

[Mic92]

Oh yes, I can indeed see that.

I was just wondering if a bultins.fromGoMod (akin to a builtins.fromJson) would have been a) conceptionally and b) "implememtationally" cheaper.

But I guess, I'd have had (once again) disregarded the implicit cost of reaching consensus about builtins.fromGoMod in that reasoning.

Nix releases cycles are too slow to rely on them for heavy builtins like that. It would take ages until we get fixes in case something changes in the go ecosystem.

[blaggacao]

Nix releases cycles are too slow to rely on them for heavy builtins like that. It would take ages until we get fixes in case something changes in the go ecosystem.

Oh thanks! That was the missing link in the argument chain against builtins.

Ceteris paribus, if it was made in nixpkgs/pkgs/pkgs-lib, though?

I innocently assume [in the reasoning for going that extra rewriting step] there was some (psychological) "path dependency" involved, but thinking laterally [out of the box], I guess my critical remarks make sense (if they do).

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/future-of-npm-packages-in-nixpkgs/14285/3

[adisbladis]

Go's lockfile system doesn't translate to Nix so we are forced to take an extra step. Unlike say npm which has a JSON that nix can parse at eval time, and which contains all the sha256 in a compatible format.

Writing a parser for go.mod isn't hard. The biggest problem is the incompatible hash format.
Some of the incompatibilities are described in https://www.tweag.io/blog/2021-03-04-gomod2nix/.

[ShamrockLee]

Considering that

1. buildGoModule is able to handle both vendored and unvendored Go project source.
2. The issue regarding the overriding can be workaround by providing vendorSha256 and related attributes as input attributes.

It is more "overridable" than buildGoPackage.

Should packages inside Nixpkgs use buildGoPackage by default, Go project developers and users will either have to rewrite the expression to use buildGoModule, or run through the *2nix source-to-expression process to re-generate the expression.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rethink-goproxy/23534/10