cmd/dist: overwrites $GOROOT_BOOTSTRAP/pkg/xxx/math/big.a

[rsc]

cmd/dist does

$GOROOT_BOOTSTRAP/bin/go install -gcflags -l -tags math_big_pure_go bootstrap/cmd/...

The intended effect of the tags is to set them for building bootstrap/math/big. But they may also be present in $GOROOT_BOOTSTRAP/src/math/big, which will change the set of files used for the standard (that is, standard in $GOROOT_BOOTSTRAP) math/big. The change in the file set will - in Go 1.5 and later - make the standard math/big look out of date, which will cause it to be recompiled and reinstalled.

There are two problems here: (1) the user may not have write access to $GOROOT_BOOTSTRAP, and (2) installing the new math/big.a will make math/big.a look out of date to a regular $GOROOT_BOOTSTRAP/bin/go command not using those tags. One example of situation (1) is when using the default Ubuntu install of Go as GOROOT_BOOTSTRAP.

The fix here is probably to also specify -pkgdir as well pointing at a temporary location (wherever we wrote the bootstrap src would be fine). That will have the unfortunate effect of recompiling the entire $GOROOT_BOOTSTRAP standard library, adding maybe 10 seconds to the build, but it will ensure that nothing is ever written to $GOROOT_BOOTSTRAP/pkg during cmd/dist.

Prior to Go 1.8, the command was just "install -gcflags -l bootstrap/cmd/...", which would not recompile math/big, because the staleness computation doesn't know what compile flags were used. CL 31142 (/cc @mundaym @randall77), which was released in Go 1.8, added the -tags argument and caused these spurious rebuilds.

[rsc]

Unfortunately -pkgdir is only present in Go 1.5 and later, so the bootstrap toolchain will need to be queried. If "go tool" includes the string "compile", then it's Go 1.5 and later. (That check would work even when the more obvious "go version" reports an arbitrary Git hash.)

[rsc]

The go tool check might fail with gccgo; maybe better would be "go build -pkgdir" or "go help build" and look at the output to see if the flag is recognized.

[mdempsky]

This must be a recent regression? I've used a read-only $GOROOT_BOOTSTRAP on one of my machines for a long-time now, and I've only run into this problem this morning.

[rsc]

Are you sure it was read-only before this morning? One unanswered question in my mind is that what the path is from bootstrap/cmd/... to math/big. We know cmd/dist forks the latest math/big into bootstrap/math/big and updates all the bootstrap/cmd/... to refer to bootstrap/math/big instead of the real math/big. So the bootstrap commands must be importing some other package in the standard library - one that cmd/dist doesn't fork into bootstrap/... - that in turn imports math/big, or else why would this command be trying to build math/big at all?

If a change in Go 1.9.1 introduced that missing link to the old math/big, that would explain things, but I don't see such a change. The only changes are in net/smtp and cmd/go, neither of which should be relevant and neither of which mentions math/big.

[mdempsky]

I'm not confident that it was previously RO. I'm reasonably confident though that I would not have taken explicit action to change it from RO to RW though.

[rsc]

It looks like this might be due to local changes we have (inside Google) in our local Go 1.9 that introduces an unexpected math/big dependency where the proper Go 1.9 does not have them. However, the same problem will happen once the go command is more careful about gcflags changes (in Go 1.10, most likely) so this problem will recur on all systems (not just Google's) once Go 1.10 is deployed. Assuming that people are not building old copies of Go using newer GOROOT_BOOTSTRAPs, we only need to fix this by the time Go 1.10 is released. Changing the milestone.

[rsc]

The dist command does install cmd/.... If Go 1.10 is the bootstrap toolchain, then the recent change to install to install only the named targets will avoid reinstalling math/big, so this is only a problem with Go 1.9.x using our local changes inside Google. Not worth fixing.