Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile, cmd/link: support arbitrary import file map configurations #20579

Closed
rsc opened this issue Jun 5, 2017 · 3 comments

Comments

Projects
None yet
2 participants
@rsc
Copy link
Contributor

commented Jun 5, 2017

Because of multiple versions of packages being used for different builds, proper support for package management in the go command is very likely going to need better support for caching build artifacts (long-standing issue #4719). I have a prototype of this caching that fixes #4719 but also various more specific bugs, like #11193, #14271, #15752, #15799, #18981, #19340, #20137, #20512 and almost certainly others I've missed.

But it's a big change. I'd like to publish an experimental go command (named xgo) that people can download and try, but I want xgo to use an unmodified Go 1.9 release, so that people trying it don't have to opt in to the instability of Go tip toolchain.

The toolchain support required by the caching prototype is fairly straightforward. The compiler and linker both need to support loading a definition of how to resolve imports instead of using the default file-name-construction lookup. For vendoring we added half of what the compiler needs, namely -importmap, which maps from import path to "fully-qualified import path" (the path containing /vendor/). The other half is a map from fully-qualified import path to exact package file to open for that import. The compiler needs this for imports it will see, and the linker needs this for all imports in the program.

The -importmap flag works for a few imports but doesn't scale well, since it must be repeated for every import, and some systems place limits on command-line size. Instead, I propose to add an -importcfg flag to both the compiler and linker that takes a single file name argument. The named file contains a sequence of lines like

importmap path=fqpath
importmap path2=fqpath2
packagefile fqpath=/path/to/file.a

There is one more change required in the compiler: it assumes that it can use the file name extension to decide whether to expect a package archive or a Go object. A trivial change lets it look at the first line of the file to make that decision (it was already reading the first line anyway) instead of using the file name.

Because I have a working prototype of proper build caching using this functionality, I'm confident these two changes are sufficient to enable the "experiment using an extra binary and the standard Go 1.9 toolchain" approach.

The -importcfg change changes very little of the code and is implemented such that it clearly has no effect if the flag is not used. As such, it should be safe to add this in Go 1.9 even now just before the beta.

@rsc rsc added this to the Go1.9 milestone Jun 5, 2017

@rsc rsc self-assigned this Jun 5, 2017

@gopherbot

This comment has been minimized.

Copy link

commented Jun 5, 2017

CL https://golang.org/cl/44850 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

commented Jun 5, 2017

CL https://golang.org/cl/44851 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

commented Jun 5, 2017

CL https://golang.org/cl/44853 mentions this issue.

gopherbot pushed a commit that referenced this issue Jun 6, 2017

cmd/compile: add -importcfg to specify import resolution
Allows reading -importmap options from a file instead of putting
them all on the command line, and adds the ability to specify the
file location of specific packages. In effect, -importcfg is a generalization
of and supersedes -importmap, -importsuffix, and -I.
Of course, those flags will continue to be supported,
for compatibility with other tools.

Having this flag in Go 1.9 will let us try some experiments involving
package management without needing guinea pigs to build a
custom Go toolchain.

This flag also helps with #14271 at some later point.

For #20579.

Change-Id: If005dbc2b01d8fd16cbfd3687dfbe82499f4bc56
Reviewed-on: https://go-review.googlesource.com/44850
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>

gopherbot pushed a commit that referenced this issue Jun 6, 2017

cmd/link: add -importcfg to specify import resolution
Adds the ability to specify the file location of each imported package,
like in the -importcfg added to cmd/compile in a related CL.
In effect, -importcfg is a generalization of and supersedes -installsuffix
and -L. Of course, those flags will continue to be supported, for
compatibility with other tools.

Having this flag in Go 1.9 will let us try some experiments involving
package management without needing guinea pigs to build a custom
Go toolchain.

This flag also helps with #14271 at some later point.

For #20579.

Change-Id: Ie4c171bcd3aa2faa446ac340e36516f2f9853882
Reviewed-on: https://go-review.googlesource.com/44851
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>

@gopherbot gopherbot closed this in 3c1914f Jun 6, 2017

@golang golang locked and limited conversation to collaborators Jun 9, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.