cmd/go: compile and link search paths not compatible with multiple GOPATH roots #14271
Here is a day 1 design incompatibility between GOPATH and cmd/go on the one side and cmd/compile's -I option and cmd/link's -L option on the other.
When invoking the compiler or linker, cmd/go must tell it where to find compiled packages. It does so by walking the full dependency graph and collecting the names of pkg directories involved and then passes that list as repeated -I (compiler) or -L (linker) options.
GOPATH defines a preference list. GOPATH=p1:p2 (p1;p2 on Windows) means first look in p1, then p2. It is also valid for both p1 and p2 to have a package foo: p1 wins. But until the fix for #14192, it was possible for cmd/go's walk of the dependency graph to produce
But there is a second kind of possible preference inversion that arises because cmd/go looks for source code and the compiler and linker look for compiled packages. This inversion is harder to fix.
Suppose p1 has source code p1/src/foo, that p2 does not have p2/src/foo, and that p2 does have a stale p2/pkg/goos_goarch/foo.a.
Suppose p2 has source code p2/src/bar, that p1 does not have p1/src/bar, and that p1 does have a stale p1/pkg/goos_goarch/bar.a.
Suppose there is a package top that imports both foo and bar. cmd/go will correctly resolve the imports to the source directories p1/src/foo and p2/src/bar;
After the fix in CL 19385, only bar is needed in the example. Forcing GOPATH order (important for more common scenarios) contradicts what would make bar resolve correctly above. Including foo just makes clear that there's no solution even if you drop the GOPATH order constraint.
The fundamental problem is that, as designed, cmd/go assumes the GOPATH source directory search and the equivalent compiled package search will land on corresponding results. In this example they do not and cannot.
I see three possible resolutions:
(1) is not terribly satisfying. (2) seems reasonable behavior by cmd/go if viewed as “detecting a removed source directory and removing the corresponding package object.” (3) seems like unwarranted complexity.
On the other hand, if we ever turn the pkg/ tree into some kind of general package cache (see #4719), it will necessarily lose the current alignment with import paths and thus require something like (3). I also wonder whether alternate build systems like gb or bazel would be helped by having (3) available (/cc @davecheney, @hanwen). For example, my compiler support for vendoring was to add
With the information I have right now, I'm inclined toward doing (2). In the common case of ≤ 1 GOPATH root, it would be no extra work, and when there are multiple GOPATH roots there are still no more removals of compiled packages than there were failed stats of source directories.
The text was updated successfully, but these errors were encountered:
Bazel sidesteps the Go tool, and manages its own caches, so this scenario isn't a problem for Bazel. Since all the compiles happen in a symlink tree (on Linux it additionally uses a namespace sandbox), there is no possibility of confusion or need of -L options. In fact, it just sets "-L . ":
If you added a -importmap option to the linker, the rules could probably be adapted for that, but it doesn't bring any benefit, just a little extra work.
It's probably too late to change this, but why not have GOPATH just apply to source directories? The artifacts would go then into a pkg/PREFIX/ directory, where you can work all relevant settings (race detector, $GOPATH, target architecture, compiler version, etc.) into PREFIX.
To be clear, -importmap is about something completely different from this
The original motivation for multiple GOPATH entries was to allow
Given GOPATH=p1:p2 and source code of just the right form, the go command could previously end up invoking the compiler with -I p2 -I p1 or the linker with -L p2 -L p1, so that compiled packages in p2 incorrectly shadowed packages in p1. If foo were in both p1 and p2 and the compilation of bar were such that the -I and -L options were inverted in this way, then GOPATH=p2 go install foo GOPATH=p1:p2 go install bar would get the p2 copy of foo instead of the (expected) p1 copy of foo. This manifested in real usage in a few different ways, but in all the root cause was that the -I or -L option sequence did not match GOPATH. Make it match GOPATH. Fixes #14176 (second report). Fixes #14192. Related but less common issue #14271 not fixed. Change-Id: I9c0f69042bb2bf92c9fc370535da2c60a1187d30 Reviewed-on: https://go-review.googlesource.com/19385 Reviewed-by: Ian Lance Taylor <firstname.lastname@example.org> Run-TryBot: Russ Cox <email@example.com>
Allows reading -importmap options from a file instead of putting them all on the command line, and adds the ability to specify the file location of specific packages. In effect, -importcfg is a generalization of and supersedes -importmap, -importsuffix, and -I. Of course, those flags will continue to be supported, for compatibility with other tools. Having this flag in Go 1.9 will let us try some experiments involving package management without needing guinea pigs to build a custom Go toolchain. This flag also helps with #14271 at some later point. For #20579. Change-Id: If005dbc2b01d8fd16cbfd3687dfbe82499f4bc56 Reviewed-on: https://go-review.googlesource.com/44850 Run-TryBot: Russ Cox <firstname.lastname@example.org> Reviewed-by: Ian Lance Taylor <email@example.com>
Adds the ability to specify the file location of each imported package, like in the -importcfg added to cmd/compile in a related CL. In effect, -importcfg is a generalization of and supersedes -installsuffix and -L. Of course, those flags will continue to be supported, for compatibility with other tools. Having this flag in Go 1.9 will let us try some experiments involving package management without needing guinea pigs to build a custom Go toolchain. This flag also helps with #14271 at some later point. For #20579. Change-Id: Ie4c171bcd3aa2faa446ac340e36516f2f9853882 Reviewed-on: https://go-review.googlesource.com/44851 Run-TryBot: Russ Cox <firstname.lastname@example.org> Reviewed-by: Ian Lance Taylor <email@example.com>
Implement importcfg on behalf of gccgo by writing out a tree of symbolic links. In addition to keeping gccgo working with the latest changes, this also fixes a precedence bug in gccgo's cmd/go vendor support (the vendor equivalent of #14271). Change-Id: I0e5645116e1c84c957936baf22e3126ba6b0d46e Reviewed-on: https://go-review.googlesource.com/61731 Run-TryBot: Russ Cox <firstname.lastname@example.org> TryBot-Result: Gobot Gobot <email@example.com> Reviewed-by: David Crawshaw <firstname.lastname@example.org>