Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: building requires otherwise-unused .mod files to be present #29410

Closed
stapelberg opened this issue Dec 24, 2018 · 15 comments

Comments

Projects
None yet
5 participants
@stapelberg
Copy link
Contributor

commented Dec 24, 2018

What version of Go are you using (go version)?

$ go version
go version go1.12beta1 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/michael/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/tmp/gp-repro"
GOPROXY=""
GORACE=""
GOROOT="/home/michael/sdk/go1.12beta1"
GOTMPDIR=""
GOTOOLDIR="/home/michael/sdk/go1.12beta1/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/tmp/repro/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build500484946=/tmp/go-build -gno-record-gcc-switches"

What did you do?

% export GOPATH=/tmp/gp-repro
% mkdir /tmp/repro
% cat > repro.go <<'EOT'
package main

import (
	"fmt"

	"golang.org/x/exp/mmap"
	"gonum.org/v1/gonum/unit"
)

func main() {
	fmt.Printf("%v, %v", mmap.ReaderAt{}, unit.Meter)
}
EOT
% go mod init repro
% go build
go: finding gonum.org/v1/gonum v0.0.0-20181223131727-b545e3e77e9e
go: finding golang.org/x/exp v0.0.0-20181221233300-b68661188fbf
go: finding golang.org/x/exp v0.0.0-20180321215751-8460e604b9de
go: finding golang.org/x/tools v0.0.0-20180525024113-a5b4c53f6e8b
go: downloading gonum.org/v1/gonum v0.0.0-20181223131727-b545e3e77e9e
go: downloading golang.org/x/exp v0.0.0-20181221233300-b68661188fbf
go: extracting golang.org/x/exp v0.0.0-20181221233300-b68661188fbf
go: extracting gonum.org/v1/gonum v0.0.0-20181223131727-b545e3e77e9e

As you can see, two versions of golang.org/x/exp are involved here: the repro module uses the latest (v0.0.0-20181221233300-b68661188fbf), whereas the gonum.org/v1/gonum module specifies an older version (v0.0.0-20180321215751-8460e604b9de).

In the build itself, the latest version is picked:

% go list -m -json all
{
	"Path": "repro",
	"Main": true,
	"Dir": "/tmp/repro",
	"GoMod": "/tmp/repro/go.mod",
	"GoVersion": "1.12"
}
{
	"Path": "golang.org/x/exp",
	"Version": "v0.0.0-20181221233300-b68661188fbf",
	"Time": "2018-12-21T23:33:00Z",
	"Dir": "/tmp/gp-repro/pkg/mod/golang.org/x/exp@v0.0.0-20181221233300-b68661188fbf",
	"GoMod": "/tmp/gp-repro/pkg/mod/cache/download/golang.org/x/exp/@v/v0.0.0-20181221233300-b68661188fbf.mod"
}
{
	"Path": "golang.org/x/tools",
	"Version": "v0.0.0-20180525024113-a5b4c53f6e8b",
	"Time": "2018-05-25T02:41:13Z",
	"Indirect": true,
	"GoMod": "/tmp/gp-repro/pkg/mod/cache/download/golang.org/x/tools/@v/v0.0.0-20180525024113-a5b4c53f6e8b.mod"
}
{
	"Path": "gonum.org/v1/gonum",
	"Version": "v0.0.0-20181223131727-b545e3e77e9e",
	"Time": "2018-12-23T13:17:27Z",
	"Dir": "/tmp/gp-repro/pkg/mod/gonum.org/v1/gonum@v0.0.0-20181223131727-b545e3e77e9e",
	"GoMod": "/tmp/gp-repro/pkg/mod/cache/download/gonum.org/v1/gonum/@v/v0.0.0-20181223131727-b545e3e77e9e.mod"
}

However, when preventing internet access by pointing GOPROXY to an unused port, I can see that the old version of golang.org/x/exp needs to be present for the build to succeed:

% rm repro && GOPROXY=http://localhost:1111/ go build
# works
% rm /tmp/gp-repro/pkg/mod/cache/download/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.*
% rm repro && GOPROXY=http://localhost:1111/ go build
go: finding golang.org/x/exp v0.0.0-20180321215751-8460e604b9de
go: golang.org/x/exp@v0.0.0-20180321215751-8460e604b9de: Get http://localhost:1111/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.info: dial tcp [::1]:1111: connect: connection refused
go: error loading module requirements
% echo 'module golang.org/x/exp' > /tmp/gp-repro/pkg/mod/cache/download/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.mod
% GOPROXY=http://localhost:1111/ go build                                                                                         
# works

This came as a surprise to me, so I wanted to check in and see if this was intentional or an oversight. Given that the .mod file only contains “module golang.org/x/exp”, it seems to me that the file could easily be synthesized?

I discovered this issue because I was experimenting with packaging Go modules in Linux distributions (e.g. Debian). It would be a significant downside if we needed to provide (and therefore maintain!) all versions that a build ever references, instead of just the versions which are selected in the end.

Thanks,

@mvdan

This comment has been minimized.

Copy link
Member

commented Dec 24, 2018

It might take a while for this to get a proper reply, as most engineers will be on holiday this week :)

/cc @bcmills @myitcv @thepudds

@thepudds

This comment has been minimized.

Copy link

commented Dec 24, 2018

@stapelberg two quick comments:

  1. you might want to set GOPROXY=off (rather than using a non-existent port). That means go commands in module mode are disallowed from using the network for dependencies. I don't know if that explains your symptoms (and probably it does not), but you might get more standard behavior in general out of GOPROXY=off.

2. you could try with the go 1.12 beta. I think there was at least one fix in 1.12 for avoiding hitting the network unnecessarily (though it might have had to do with replace, which might not help here).

edit: sorry, I see you are using the 1.12 beta.

@thepudds

This comment has been minimized.

Copy link

commented Dec 24, 2018

Setting GOPROXY=off and attempting to reproduce gives a slightly different error, but still fails:

$ go1.12beta1 build
go: golang.org/x/exp@v0.0.0-20180321215751-8460e604b9de: module lookup disabled by GOPROXY=off
go: error loading module requirements

Someone else might need to comment if this is a bug, vs. necessary behavior, vs. perhaps a not-yet-implemented feature could avoid this.

@stapelberg you mentioned:

It would be a significant downside if we needed to provide (and therefore maintain!) all versions that a build ever references, instead of just the versions which are selected in the end.

I am just looking at this quickly, but is it just meta data about the old version 20180321215751-8460e604b9de that currently seems to be required? E.g., after a clean build (with GOPROXY unset):

$ ls $GOPATH/pkg/mod/cache/download/golang.org/x/exp/@v                      
list
v0.0.0-20180321215751-8460e604b9de.info
v0.0.0-20180321215751-8460e604b9de.mod
v0.0.0-20181221233300-b68661188fbf.info
v0.0.0-20181221233300-b68661188fbf.mod
v0.0.0-20181221233300-b68661188fbf.zip
v0.0.0-20181221233300-b68661188fbf.ziphash

and:

$ ls $GOPATH/pkg/mod/golang.org/x
exp@v0.0.0-20181221233300-b68661188fbf

or:

$ cd $GOPATH/pkg/mod
$ find -name "*8460e604b9de*"
./cache/download/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.info
./cache/download/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.mod

@stapelberg

This comment has been minimized.

Copy link
Contributor Author

commented Dec 24, 2018

I am just looking at this quickly, but is it just meta data about the old version 20180321215751-8460e604b9de that currently seems to be required? E.g., after a clean build (with GOPROXY unset):

Yes, it’s just the .mod file. Hence my suspicion that this is unintended.

@bcmills

This comment has been minimized.

Copy link
Member

commented Jan 17, 2019

This is working as designed. Every go.mod file in the transitive requirement graph — even for versions that we know will not be selected — impacts the constraints on which versions we can ultimately choose, so we have to load those files in order to compute the final versions.

The good news is, go.mod files are small and cacheable, so the impact on build times should usually be negligible.

See the cross test-cases for the MVS package for examples:

# Cross-dependency between D and E.
# No matter how it arises, should get result of merging all build lists via max,
# which leads to including both D2 and E2.
name: cross1
A: B C
B: D1
C: D2
D1: E2
D2: E1
build A: A B C D2 E2
name: cross1V
A: B2 C D2 E1
B1:
B2: D1
C: D2
D1: E2
D2: E1
build A: A B2 C D2 E2
name: cross1U
A: B1 C
B1:
B2: D1
C: D2
D1: E2
D2: E1
build A: A B1 C D2 E1
upgrade A B2: A B2 C D2 E2
name: cross1R
A: B C
B: D2
C: D1
D1: E2
D2: E1
build A: A B C D2 E2
name: cross1X
A: B C
B: D1 E2
C: D2
D1: E2
D2: E1
build A: A B C D2 E2
name: cross2
A: B D2
B: D1
D1: E2
D2: E1
build A: A B D2 E2
name: cross2X
A: B D2
B: D1 E2
C: D2
D1: E2
D2: E1
build A: A B D2 E2
name: cross3
A: B D2 E1
B: D1
D1: E2
D2: E1
build A: A B D2 E2
name: cross3X
A: B D2 E1
B: D1 E2
D1: E2
D2: E1
build A: A B D2 E2

@bcmills bcmills closed this Jan 17, 2019

@stapelberg

This comment has been minimized.

Copy link
Contributor Author

commented Jan 17, 2019

This is working as designed. Every go.mod file in the transitive requirement graph — even for versions that we know will not be selected — impacts the constraints on which versions we can ultimately choose, so we have to load those files in order to compute the final versions.

Can you elaborate on the ways in which these files can impact the version selection? Specifically, are you saying it is not sufficient to synthesize these files?

The good news is, go.mod files are small and cacheable, so the impact on build times should usually be negligible.

That’s not my concern, though. In Linux distributions such as Debian, we strive to have precisely one version of a specific module packaged. We can’t keep old packages around just for their .mod files.

If this is really working as intended with no wiggle room, we’ll need to work around that and either modify .mod files at package-time (seems more labor-intensive, but haven’t thought it through entirely) or synthesizing .mod files at build-time (ugh).

@bcmills

This comment has been minimized.

Copy link
Member

commented Jan 17, 2019

In Linux distributions such as Debian, we strive to have precisely one version of a specific module packaged. We can’t keep old packages around just for their .mod files.

Sorry, but Go's version selection algorithm intentionally prioritizes reproducibility over recency: when you build a module-enabled binary, you use the specific versions of the modules it was tested against, not whatever newer version happens to be installed on your system.

Now that #27859 is fixed, you can probably make something work using wildcard replace directives, but the resulting builds will not be as reproducible as ordinary module-enabled builds.

@thepudds

This comment has been minimized.

Copy link

commented Jan 17, 2019

@stapelberg Is this in the context of what I think in the past might have been described as "more or less one giant GOPATH" for Debian packaging of Go libraries, or is this more in the context of trying to use Go modules to break out of that (or perhaps nothing to do with any of that)?

I am not sure how much flexibility you have at package time vs. build time, how that changes for a package vs. a binary, the level of control around setting env variables, or your ability to use an auxiliary directory that contains auxiliary data like go.mod files...

...but depending on your constraints, you might be able to do some variety of using GOPROXY=file:/// and/or temporarily setting GOPATH (which controls the location of GOPATH/pkg/mod cache location). That might not help, but here is a concrete example (from @myitcv):

https://github.com/go-modules-by-example/index/tree/master/012_modvendor

That particular example is showing an alternative to traditional vendoring, and I suspect that is not exactly what you'd want, but it serves as a concrete example of the types of things that are enabled via the flexibility of GOPROXY=file:///.

In any event, good chance GOPROXY=file:/// does not help based whatever your constraints are, but wanted to at least mention it in case it gives you an idea, etc.

@stapelberg

This comment has been minimized.

Copy link
Contributor Author

commented Jan 18, 2019

resulting builds will not be as reproducible as ordinary module-enabled builds.

Debian has its own reproducibility efforts, see https://reproducible-builds.org/. The binaries we build are reproducible, and have been before Go modules :)

@stapelberg Is this in the context of what I think in the past might have been described as "more or less one giant GOPATH" for Debian packaging of Go libraries

Yes. This is the current model and we have no plans to change that.

I am not sure how much flexibility you have at package time vs. build time,

Changes at packaging-time take a long time to trickle down to all our volunteers. Build-time changes are preferable for this particular situation.

how that changes for a package vs. a binary,

No change.

the level of control around setting env variables

We can set arbitrary env variables.

or your ability to use an auxiliary directory that contains auxiliary data like go.mod files...

We can do that, but every step we need to set up introduces additional complexity, which makes builds slower and harder to understand/debug.

...but depending on your constraints, you might be able to do some variety of using GOPROXY=file:/// and/or temporarily setting GOPATH (which controls the location of GOPATH/pkg/mod cache location). That might not help, but here is a concrete example (from @myitcv):

Yeah, we currently already synthesize a GOPATH. From this discussion it seems like we need to synthesize all go.mod files ever referenced by the transitive closure as well. I have not yet looked into how easy or hard this would be to do.

@thepudds

This comment has been minimized.

Copy link

commented Mar 8, 2019

@stapelberg I am curious what your latest thinking here is, at least briefly? Part of the reason for me bouncing this is curiosity, but also some similar questions are coming up elsewhere.

Also, one specific area I do not understand is why you mention synthesizing a large number of go.mod files. Was it specifically limited to synthesizing older go.mod files that are not going to be selected as the final versions? If so, are you thinking they would be empty aside from the module foo line?

Expanding on an earlier point slightly, rather than synthesizing, you might be able use a filesystem-based module cache containing go.mod files. I suspect this is well known to you at this point, but:

  • The module download cache location is controlled by GOPATH. In particular, go mod download, go build, etc. populate the module cache in GOPATH/pkg/mod.
  • In addition, when you want to use a particular module cache, you can tell the go command to use a local module cache by setting GOPROXY=file:///file/path.
  • You can put those two things together:
   # Populate a module download cache in /tmp/gopath-for-cache
   $ GOPATH=/tmp/gopath-for-cache  go mod download

   # Build using the contents of the module download cache in /tmp/gopath-for-cache
   $ GOPROXY=file:///tmp/gopath-for-cache/pkg/mod/cache/download  go build

In other words, that might be an approach to get the actual go.mod files of interest, rather than synthesizing a large number of older go.mod files.

Note that even though you are setting the GOPROXY environment variable in the steps above, there is no actual proxy process involved, and everything is just being read directly from the local filesystem. An even more detailed example is in the link I had mentioned above.

All that said, I don't know if that set of capabilities really helps you or not, and perhaps synthesizing is easier based on your use case.

@stapelberg

This comment has been minimized.

Copy link
Contributor Author

commented Mar 14, 2019

Thanks for your reply.

Also, one specific area I do not understand is why you mention synthesizing a large number of go.mod files. Was it specifically limited to synthesizing older go.mod files that are not going to be selected as the final versions? If so, are you thinking they would be empty aside from the module foo line?

Yes to both questions.

In other words, that might be an approach to get the actual go.mod files of interest

I don’t follow. Where would they come from? Note that Linux distributions generally prohibit internet access during builds, so all files we want to use need to be shipped in other packages (build dependencies) or created by the build infrastructure.

@thepudds

This comment has been minimized.

Copy link

commented Apr 5, 2019

@bcmills @jayconrod Based on the conversation in #30831 (comment) and #30831 (comment) and related comments, is there some chance that a missing older go.mod file (e.g., some of the issues described in this issue here) could also be viewed as a non-fatal error if that older version does not end up in the final build list?

Any brief thoughts on maybe re-opening this, even if it might not be a near-term change here, and even if it might later end up getting closed again?

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented Apr 5, 2019

@thepudds I'm not sure if I followed completely, but a missing go.mod file should not be a fatal error. If there is no go.mod file in a dependency, the Go command will synthesize a trivial .mod file with just the module path and no requirements. That will end up in the cache and will be served by the proxy.

If a go.mod file is missing in an older version but is present with the wrong module path in a newer version (#30831), that's probably a good sign that we were importing packages with the wrong path. We might not need to know that to suppress the error though.

@stapelberg

This comment has been minimized.

Copy link
Contributor Author

commented Apr 6, 2019

[…] but a missing go.mod file should not be a fatal error. If there is no go.mod file in a dependency, the Go command will synthesize a trivial .mod file with just the module path and no requirements.

The reason why I opened this issue is that I’m seeing the opposite behavior (or I’m misunderstanding you). Can you try to reproduce the steps provided in the initial message please?

@thepudds

This comment has been minimized.

Copy link

commented Apr 6, 2019

I think there might be two valid but different uses of the word "missing" here.

At least as I understood it:

  • I think @jayconrod is referring to the case where a repository foo has not yet opted in to modules, and then when a module-aware consumer imports foo the go command will automatically synthesize the "missing" go.mod file that is not present in repository foo (as described in #29410 (comment)).

  • I think @stapelberg is describing a case that arose where he was experimenting on approaches for packaging Go modules in Linux distributions (Debian in particular) where older go.mod files were eliminated from the on-disk module cache to simplify the packaging problem (e.g., rm /tmp/gp-repro/pkg/mod/cache/download/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.*), but then the go command would complain about those old "missing" go.mod files (as seen in #29410 (comment) and the next few comments).

Separately, I had bounced this issue to link it to #30831 (comment). In @stapelberg's case, I suspect the older "missing" go.mod files might not be strictly required, perhaps similar to how in #30831 the old problematic grpc version is not strictly required in the final build list and hence was wondering if a similar solution "treat as a non-fatal error" approach might be possible here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.