New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: buildmode=shared config should not be read back from $GOROOT/pkg #22196

Open
rsc opened this Issue Oct 10, 2017 · 1 comment

Comments

Projects
None yet
4 participants
@rsc
Contributor

rsc commented Oct 10, 2017

The $GOROOT/pkg (or $GOPATH/pkg) directory is meant to be a cache of precomputed results, all of which can be recomputed as needed. In particular the ideal is that you get the same build results both before and after doing rm -r pkg, and that you get the same build results regardless of what commands have run before.

This property is not true of buildmode=shared: the package groupings passed to go install -buildmode=shared commands are recorded only in the content of the pkg directory. That is, if you run

go install -buildmode=shared runtime sync/atomic
go install -linkshared myprog

then you get a myprog installed that links against pkg/libruntime,sync-atomic.so. It knows to do this because it read that string from both pkg/runtime.shlibname and pkg/sync/atomic.shlibname. If pkg had been removed, the resulting myprog would be different (it wouldn't depend on any shared libraries).

Also, if you run:

go install -buildmode=shared runtime sync/atomic
<modify files in runtime>
go install -linkshared myprog

then the install of myprog is actually expected to (and does) update pkg/libruntime,sync-atomic.so. Even if myprog made no mention of sync/atomic, cmd/go is expected to know what packages are built into pkg/libruntime,sync-atomic.so because it reads the list back out of the .so file.

Both the .shlibname files and the package list in the .so file are irreplaceable: if they are deleted, the go command cannot work out what they should contain. They make pkg not a proper cache but a source of truth for build configuration.

In the long term we need to move all configuration of this sort outside the pkg directory. I don't expect this to happen for Go 1.10, but I'd like to try to figure out a plan for Go 1.11.

It seems to me that the questions that must be answered are (1) what form does the configuration take, and (2) where is it kept? The form is not interesting here; what matters is where we keep the config.

One possibility is that the configuration file lives in a well-known location, like $GOROOT/src/shared.cfg and $GOPATH/src/shared.cfg.

One possibility is that the configuration file must be specified in any particular build:

go install -buildmode=shared -sharedcfg=x.cfg std
go install -linkshared -sharedcfg=x.cfg myprog

One possibility is that there's a well-known location but the file can be overridden, or perhaps augmented, by specifying a config.

I don't know enough about how this buildmode is used to say what the best approach is. Obviously if the people making decisions about the shared library structure are not the owners of $GOROOT, then a file in $GOROOT is not sufficient by itself. But maybe a default file in $GOROOT plus the ability to override makes more sense so that everyone doesn't reinvent the wheel. I don't know.

Thoughts?

/cc @mwhudson @ianlancetaylor @crawshaw

@rsc rsc added this to the Go1.11 milestone Oct 10, 2017

@mwhudson

This comment has been minimized.

Contributor

mwhudson commented Oct 17, 2017

I agree that storing irreplaceable data in pkg/ is bad. I've never really been happy about this aspect of buildmode=shared.

I don't know how relevant it is, but I think the package list in the .so and the .shlibname files basically contain the same information -- the groupings of packages that are intended to go into a shared library together.

From the distro POV, any of your proposals would be OK I think. TBH, from a distro POV assuming each repo (in the sense of a thing you pass to go get) gets its own shared library would be fine and obviate your concerns. Where that falls down is testing, writing misc/cgo/testshared would be a lot harder in this model! And it would be much much slower if it had to build all of the standard library into a shared library.

So I don't know if that helps any, I'm afraid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment