cmd/go: add test for build -a -tags netgo -installsuffix netgo in test.bash

[ianlancetaylor]

In Go 1.4 we changed "go build -a" so that it no longer builds the standard library (issue #8290). Prior to Go 1.4, programmers who wanted a fully static binary that uses the network routinely built their programs with "go build -a -tags netgo". That no longer works. There is no longer a simple way do achieve this desirable goal. We should fix this for 1.4.1.

[minux]

I propose that we allow go build -a std to rebuild everything in std,
just go install -a std will skip any standard packages.

One interesting observation is that, with 1.4, go install -a std does
re-install something, so the fix for #8290 is actually not correct:

$ go14 version
go version go1.4 linux/amd64
$ go14 install -a -v std
cmd/internal/rsc.io/arm/armasm
cmd/internal/rsc.io/x86/x86asm
cmd/internal/objfile
cmd/addr2line
cmd/nm
cmd/objdump
cmd/pprof

[davecheney]

On 18 Dec 2014, at 08:33, Minux Ma notifications@github.com wrote:

I propose that we allow go build -a std to rebuild everything in std,
just go install -a std will skip any standard packages.

I am against more special cases, especially of this continues to encourage people to favour go build over go install.

One interesting observation is that, with 1.4, go install -a std does
re-install something, so the fix is actually not correct:

$ go14 version
go version go1.4 linux/amd64
$ go14 install -a -v std
cmd/internal/rsc.io/arm/armasm
cmd/internal/rsc.io/x86/x86asm
cmd/internal/objfile
cmd/addr2line
cmd/nm
cmd/objdump
cmd/pprof

Maybe we should just roll back the change to -a. It wasn't working before, but then as now, the REAL fix is to build go from source.

—
Reply to this email directly or view it on GitHub.

[minux]

I'd argue that it's not a new special case. The original fix is a special case, and
my proposal just tries to limit its scope:
To fix the original #8290, we don't need to limit go build -a std.

Yes, this might encourage people to use go build, but sometimes people just
want to build a one-shot statically linked binary, requiring them to build Go
from scratch is too much. (I always use tip, so that problem doesn't affect me
at all, but we've seen more than one issue reports for this problem since 1.4
is released, and this suggests we do need to something here.)

Also, I'm among the one that oppose that fix. I think a better solution for #8290
is to support another meta package that means all-std (alternatively, we can
support negative package selectors at cmd/go command line, but that's probably
too much.)

[tianon]

Given a source release, is there a way to compile in netgo from the start? We can't find a simple one, which is why Docker has resorted to these other tricks to recompile std with netgo.

It was discovered that go clean -i net && go install -tags netgo std works to get us netgo in Go 1.4, but it feels like we're playing chicken and egg here. 😄

[kostix]

Is there a way to make all.bash somehow build two copies of the relevant files, and go build|install -tags netgo then pick the "static" one? So that no rebuilding ever happens -- the user has two copies of the necessary stuff right from the start.

Please note that I'm not familiar with Go internals, just conveying what would seem to be logical to a layman. I mean, until the 1.4 release I had no idea that building a Linux binary with "static DNS lookups" (via -tags netgo) actually implied rebuilding a part of Go itself. Of coure, what I'm talking about looks like a special case but oh well... ;-)

[adg]

Does go install -a -tags netgo std yourpkg work?

[minux]

Does go install -a -tags netgo std yourpkg work?

No. it will reinstall the following packages along with all non-standard
package
dependencies of yourpkg, but not the net package.

cmd/internal/rsc.io/arm/armasm
cmd/internal/rsc.io/x86/x86asm
cmd/internal/objfile
cmd/addr2line
cmd/nm
cmd/objdump
cmd/pprof

[ianlancetaylor]

No, adding std causes the standard programs to be reinstalled but does not cause the standard library to be rebuilt.

[ianlancetaylor]

To be clear, my "no" was in reply to adg, not minux.

[ianlancetaylor]

We used to have a simple rule: the build flag -a meant to rebuild all packages even if they were up to date. We made that flag more complicated by changing it to ignore standard packages for a release build.

I propose that we restore -a to what it used to be, and add a new flag to ignore standard packages. Advantages: 1) fixes the original issue by giving people a flag to do what they want; 2) fixes this issue; 3) simplifies the meaning of -a. Cons: New flag.

[davecheney]

I like that plan.
On 7 Jan 2015 12:29, "Ian Lance Taylor" notifications@github.com wrote:

We used to have a simple rule: the build flag -a meant to rebuild all
packages even if they were up to date. We made that flag more complicated
by changing it to ignore standard packages for a release build.

I propose that we restore -a to what it used to be, and add a new flag to
ignore standard packages. Advantages: 1) fixes the original issue by giving
people a flag to do what they want; 2) fixes this issue; 3) simplifies the
meaning of -a. Cons: New flag.

—
Reply to this email directly or view it on GitHub
#9369 (comment).

[minux]

Could we just support package negation on the cmd/go command line?
go install -a all -std

yes, this is more work, but it's also a new general capability that could
be useful in other scenarios, not limited to just ignoring std packages.

And then we can restore the -a flag, and advise people of binary
distribution
to use go install -a all -std to rebuild all their packages.

Rationale for this is that no only std packages need special treatment,
some packages (commands) in x/tools also need to be skipped because
they are also intended to be installed into $GOROOT (godoc, cover, vet,
for example), so the full command line should be:

go install -a all -std -golang.org/x/tools/cmd/{godoc,cover,vet}

Have this general mechanism will save us the trouble should we introduce
more special commands into the tools sub-repo in the future.

[ianlancetaylor]

https://groups.google.com/d/msg/golang-dev/ajODMUyOhPI/l3Kh-hW3x5AJ

[ianlancetaylor]

I think that negating the list of packages may become more complex when discussing "go build -i". It's also odd in that it only really applies to all or "....". But I'd be OK with that if people prefer it.

[minux]

I imagine that negation only works for explicitly listed packages (or
package sets),
so the exact semantics is:
first generate the packages ignoring the negated packages, and then remove
those negated.

Whether -i applies to negated packages or not is open to debate, though.

[davecheney]

I'd really like to see the behaviour of -a reverted for Go 1.4.1

Once that is done we can workshop more elaborate ideas for Go 1.5.

What I don't want to see is users of Go 1.4 have to wait 8 months for a
resolution of this problem.

On Wed, Jan 7, 2015 at 12:49 PM, Minux Ma notifications@github.com wrote:

Could we just support package negation on the cmd/go command line?
go install -a all -std

yes, this is more work, but it's also a new general capability that could
be useful in other scenarios, not limited to just ignoring std packages.

And then we can restore the -a flag, and advise people of binary
distribution
to use go install -a all -std to rebuild all their packages.

Rationale for this is that no only std packages need special treatment,
some packages (commands) in x/tools also need to be skipped because
they are also intended to be installed into $GOROOT (godoc, cover, vet,
for example), so the full command line should be:

go install all -std -golang.org/x/tools/cmd/{godoc,cover,vet}
http://golang.org/x/tools/cmd/%7Bgodoc,cover,vet%7D
http://golang.org/x/tools/cmd/%7Bgodoc,cover,vet%7D

Have this general mechanism will save us the trouble should we introduce
more special commands into the tools sub-repo in the future.

—
Reply to this email directly or view it on GitHub
#9369 (comment).

[minux]

A more conservative stop-gap solution for 1.4.1 might be that
go install -a std will not do anything, but
go build -a net somepkg will do the correct thing.
i.e. the restriction only applies to go install, but not to go build.

The rationale being that the user should rarely need netgo, so
recompiling the net package when a pure Go net package is
needed is not a big problem.

If the user want to use pure Go net package always, they can
always rebuild from source.

[davecheney]

I think reverting the change which affected the behaviour of -a is the most
conservative option.

On Wed, Jan 7, 2015 at 1:21 PM, Minux Ma notifications@github.com wrote:

A more conservative stop-gap solution for 1.4.1 might be that
go install -a std will not do anything, but
go build -a net somepkg will do the correct thing.
i.e. the restriction only applies to go install, but not to go build.

The rationale being that the user should rarely need netgo, so
recompiling the net package when a pure Go net package is
needed is not a big problem.

If the user want to use pure Go net package always, they can
always rebuild from source.

—
Reply to this email directly or view it on GitHub
#9369 (comment).

[minux]

I agree (to revert that change in 1.4.1). That change is not fully
working due to the introduction of internal packages anyway.

$ go14 version
go version go1.4 linux/amd64
$ go14 install -a -v std
cmd/internal/rsc.io/arm/armasm
cmd/internal/rsc.io/x86/x86asm
cmd/internal/objfile
cmd/addr2line
cmd/nm
cmd/objdump
cmd/pprof

And go install -a all will still try to reinstall
x/tools/cmd/{godoc,vet,cover},
so even if the restriction fully works for std, it's still insufficient to
address
the original issue (need a way to reinstall everything after a new release
of Go, but do not touch anything inside $GOROOT)

[davecheney]

If the problem to be solved is "how can I rebuild my $GOPATH after
replacing my Go install", then can we close that issue with a workaround

 cd $GOPATH
 rm -rf pkg
 cd src
 go install ./...

That will work. In the rare case of multiple GOPATH entries, the procedure
can be repeated.

I don't know why anyone would need to do this anyway.

On Wed, Jan 7, 2015 at 1:31 PM, Minux Ma notifications@github.com wrote:

I agree (to revert that change in 1.4.1). That change is not fully
working due to the introduction of internal packages anyway.

$ go14 version
go version go1.4 linux/amd64
$ go14 install -a -v std
cmd/internal/rsc.io/arm/armasm
cmd/internal/rsc.io/x86/x86asm
cmd/internal/objfile
cmd/addr2line
cmd/nm
cmd/objdump
cmd/pprof

And go install -a all will still try to reinstall
x/tools/cmd/{godoc,vet,cover},
so even if the restriction fully works for std, it's still insufficient to
address
the original issue (need a way to reinstall everything after a new release
of Go, but do not touch anything inside $GOROOT)

—
Reply to this email directly or view it on GitHub
#9369 (comment).

[minux]

On Tue, Jan 6, 2015 at 9:36 PM, Dave Cheney notifications@github.com
wrote:

If the problem to be solved is "how can I rebuild my $GOPATH after
replacing my Go install", then can we close that issue with a workaround

cd $GOPATH
rm -rf pkg
cd src
go install ./...

That will work. In the rare case of multiple GOPATH entries, the procedure
can be repeated.

I don't know why anyone would need to do this anyway.

I agree. Building Go code is generally so fast that even rebuilding
everything
each time will be barely noticeable (unless the program relies on some cgo
packages.....)

[davecheney]

I don't know what you mean by 'every time', if you mean every time you
change your go install (ie, moving between major releases) or between
development builds, this is a once off hit. I'd be surprised if the time to
rebuild everything in your GOPATH was greater than the time it took to
rebuild your go install, or download a packaged one.

On Wed, Jan 7, 2015 at 1:42 PM, Minux Ma notifications@github.com wrote:

On Tue, Jan 6, 2015 at 9:36 PM, Dave Cheney notifications@github.com
wrote:

If the problem to be solved is "how can I rebuild my $GOPATH after
replacing my Go install", then can we close that issue with a workaround

cd $GOPATH
rm -rf pkg
cd src
go install ./...

That will work. In the rare case of multiple GOPATH entries, the
procedure
can be repeated.

I don't know why anyone would need to do this anyway.

I agree. Building Go code is generally so fast that even rebuilding
everything
each time will be barely noticeable (unless the program relies on some cgo
packages.....)

—
Reply to this email directly or view it on GitHub
#9369 (comment).

[minux]

On Tue, Jan 6, 2015 at 9:45 PM, Dave Cheney notifications@github.com
wrote:

I don't know what you mean by 'every time', if you mean every time you
change your go install (ie, moving between major releases) or between
development builds, this is a once off hit. I'd be surprised if the time
to
rebuild everything in your GOPATH was greater than the time it took to
rebuild your go install, or download a packaged one.

A concrete example for what I meant by "every time".
I have a special GOPATH that has a broken symlink as pkg so that
go install will never work for that GOPATH:
$ ln -s $GOPATH/pkg $GOPATH/pkg

Because I don't have any cgo packages in that GOPATH, I don't feel
the need to every go install anything..... I switch go versions very
frequently
for that GOPATH, so even if I installed the packages, go build will almost
always rebuild everything.

[davecheney]

Why would you do something like this ? Is it just for testing ?

This doesn't sound like a use case that the go tool should be optimising
for.

On Wed, Jan 7, 2015 at 1:52 PM, Minux Ma notifications@github.com wrote:

On Tue, Jan 6, 2015 at 9:45 PM, Dave Cheney notifications@github.com
wrote:

I don't know what you mean by 'every time', if you mean every time you
change your go install (ie, moving between major releases) or between
development builds, this is a once off hit. I'd be surprised if the time
to
rebuild everything in your GOPATH was greater than the time it took to
rebuild your go install, or download a packaged one.

A concrete example for what I meant by "every time".
I have a special GOPATH that has a broken symlink as pkg so that
go install will never work for that GOPATH:
$ ln -s $GOPATH/pkg $GOPATH/pkg

Because I don't have any cgo packages in that GOPATH, I don't feel
the need to every go install anything..... I switch go versions very
frequently
for that GOPATH, so even if I installed the packages, go build will almost
always rebuild everything.

—
Reply to this email directly or view it on GitHub
#9369 (comment).

[minux]

As mentioned in the CL (https://golang.org/cl/2770), I'm in favor
of removing the -a check when doing "go build".

For those who want to install a netgo package, they do have a
workaround now:
touch go14 env GOROOT/src/net/*
go14 install -tags netgo std # can also add -a, it doesn't matter

As they are installing into $GOROOT anyway, touching the net
package source files seems like a suitable workaround.

[rsc]

The original report was not 'go install', it was 'go build'. Specifically, 'go build -a -tags netgo my/binary' doesn't work anymore. I believe the correct response to is to use 'go build -a -tags netgo -installsuffix netgo my/binary'. That command should work fine in plain Go 1.4, no changes needed right now.

[tianon]

I believe the correct response to is to use 'go build -a -tags netgo -installsuffix netgo my/binary'.

Ahh, I can confirm that this works for my smaller project that needed this. Going to test it on Docker shortly too. 👍 ❤️

[tianon]

It works! 🎉 (moby/moby#10087)

Any chance we could get that in a test case to make sure it doesn't break for Go 1.5? 👼

[gopherbot]

CL https://golang.org/cl/10761 mentions this issue.