x/build/version: make it trivial to obtain and test code against new go releases

[fgergo]

(was: direct email to adg titled "proposal on getting more people to test beta, -rc releases". Adg suggested to continue on golang-dev, but since this is a proposal, golang-dev shall receive a notification with an issue id on the concrete proposal according to https://github.com/golang/proposal.)

Though the goal is the same (to get better test coverage for beta and rc releases), this is a proposal about a process change, not about the complex tool and process change I sent as an rfc to golang-dev https://groups.google.com/forum/#!topic/golang-dev/kycbtvTIBAA

(Original direct email revised for clarity.)

Proposal

to include immediate value proposition (or expectation or hope...) in beta and rc announcements:

1. if somebody planned to work on something earlier, a personal one-line reply on it (e.g. if it was useful or ready or not useful or whatever)
2. and/or no more than 1 or 2 personal new favourites from 2-3 tip users in the announcement email (e.g. "1.8 has new go bug, contrary to it's name it helps me. signed me.")

Assumption

(This may be a bit off, but I think the big picture for possible non golang-dev beta or -rc testers is something like this.)
In the morning:

1. "hmm, a new go beta. Ok almost all previous final go releases were good or more like exceptionally useful for me."
2. "my current go release is working fine." <--- that's probably high value for a possible non golang-dev beta tester
3. "simple upgrade to a final is easy. Testing a beta sounds like work I didn't plan and I am not used to it. Let's see, I need to at least: download, install, configure, select what to test, select how to test, then test-test, test, then possibly report in some useful way if something is not like I expect it to be, possibly with a minimal repro. Uhh, that's a lot and doing only some subset of these is not even useful in general."
4. "ok, what do I get, hmmm I should probably check the draft release notes in the announcement."
5. ok, let's see... beta.golang.org/doc/go1.8 If I understand correctly, I'll need to read and evaluate all of these to decide if I get any immediate value in testing the beta.
6. "although I get everything for free and I could give back now by testing the beta. Maybe later. I'll test this later, when it's probably more immediately valuable for me."

To confirm,

nice change in value proposition since the last beta announcement:
"There are no known problems or regressions." <--- that's a high value statement. Although this possibly high value is immediately lowered after clicking and seeing "Known Issues".

[rsc]

Hmm, OK. It sounds like the underlying issue you want to solve is to make it easier for people to download and run alternate Go versions? Like go test --goverson=1.8beta1 mypkg? There's a lot to unpack there, but that could be useful. It's not clear what to do though.

This wouldn't need to built into the go command. There could be a separate gotry command that did this, downloading versions as needed.

FWIW, I keep all old versions of Go in my home directory and have scripts like:

$ cat $(which go1.5)
#!/bin/bash
export GOROOT=$HOME/go1.5
export PATH=$HOME/go1.5/bin:$PATH
exec $GOROOT/bin/go "$@"

Could be beta.golang.org/go1.8beta1 or just a unified beta.golang.org/gotry or ...

[rsc]

I retitled this to the general underlying problem that seemed to be the target of the golang-dev thread.

[fgergo]

Thank you for your comment!
Although I tried to separate 2 ideas (1 proposal on process change on value proposition and 1 about a tool change), I was not successful. I've reread everything I wrote on these ideas, I still don't know what I did wrong.

I try to clarify again:
My assumption was after I've seen both adg's and somebody else's tweets/comments on how few tests the last beta received and how little time remained to fix bugs before 1.7, that at least 2 approaches could be considered to increase test coverage for betas.

1. make go programmers interested to test when the go project needs tests
2. make it easy/trivial to make regressions tests with betas on all locally installed packages under $GOPATH/src for anybody with a go installation and publish/report the result without any human assistance.

It's a fact I could not make myself clear on 1., that's why reconsider my proposal for 1: I suggest to forget about that.
About the tool change (2.) in the thread: https://groups.google.com/forum/#!topic/golang-dev/kycbtvTIBAA
I can see there are quite a few details brad had probably already thought about when he replied and most probably rsc as well, I am still collecting thoughts in the google docs document I referred to, before I come up with a proposal.

Please respond how/if I should open/close/merge/whatever this github issue or just leave it as it is.

[minux]

@rsc, I think just symlink $HOME/go1.5/bin/go to your_local_bin/go1.5 suffices. There is no need for a wrapper script because GOROOT is always embedded into the Go command. Of course, that means you will need to build various Go releases in their final position.

[josharian]

Whipped something up: https://github.com/josharian/goversion. All manner of comments welcome.

[rsc]

@josharian Nice. I will leave it to you and @bradfitz to figure out what to do next, but I'd love to have instructions like that in the next beta release.

@fgergo, thanks for the clarification. I agree (1) is hard. I think lowering the barrier to entry might be enough. For (2), I was with you up until "publish/report the result without any human assistance." Many users are working on closed source systems. Publishing the results in a way that (a) helps us identify the underlying problem and (b) doesn't reveal details of their code that they don't want revealed is also hard.

What I think we can do (what @josharian put together seems like the right direction) is "make it easy/trivial to make regressions tests with betas on all locally installed packages under $GOPATH/src for anybody with a go installation".

@josharian / @bradfitz, I wonder if goversion should default to downloading binaries from golang.org/dl instead of doing a full rebuild. Should be faster, makes sure we're not debugging misbuilt toolchain problems.

[fgergo]

@rsc on (1): I imagined to automatically publish something like this:
$GOOS, $GOARCH, $GOHOSTOS
local: 1.7.3
testing: 1.8beta2
number of different test results: 0/444
1.7.3 test duration: 26m 12s
go1.8beta2 test duration: 25m 59s

If this information is too sensitive for somebody to publish, goversion should not be run. Maybe this warning could be announced in the first few goversions and then later goversion changed to only produce output if some error happened, similar to any other go tool.

[josharian]

@rsc yes, goversion should probably download binaries when possible. It is useful to know about build problems, but I guess this is not the venue.

@bradfitz what next, do you think?

[bradfitz]

@josharian, I was going to write that tool today. :-) Thanks!

Yes, please make binary downloads the default. Actually, I'd make build-from-source require a flag. If you run it on plan9, the error message should be something like "go1.8beta1 binary download not available for plan9/386. Use the --build flag.".

And then let's get some testing on golang-nuts / Slack, and then we'll announce it with go1.8beta2.

[josharian]

Where should it live? I don't care particularly, I just want to know what to do with the next round of code.

Looking at golang.org/dl, some questions:

(1) How do I get a list of available versions? Note that e.g. 1.7.2 is not available for download, so I can't just use the git tags like I did for build-from-source.
(2) Is the download naming convention stable enough that I can hard-code it?
(3) Is there a non-screen-scrape way to get checksums, so that I can sanity-check the download?

Basically, I can haz an API for golang.org/dl?

[gopherbot]

CL https://golang.org/cl/34012 mentions this issue.

[bradfitz]

@josharian, ignore /dl/?format=json for now. That's too hard to deploy and iterate on.

Here's an easier API: https://storage.googleapis.com/go-builder-data/dl-index.txt

Future note to self:

$ gsutil ls gs://golang/ | sed 's,gs://,https://storage.googleapis.com/,' > index.txt
$ upload -cacheable=false -public -file=index.txt go-builder-data/dl-index.txt

[bradfitz]

/cc @broady

[fgergo]

@josharian at the moment goversion won't work on windows without gcc obviously. Though on windows it fails silently currently. I understand that when binary downloads are preferred, this won't be a problem.

@ anybody_who_can_answer: Assuming goversion is ready for 1.8beta2, how would the usage instructions look like? Should the new beta tester run go test for src/, then run goversion ..., then manually diff the 2 test runs? What if no difference? What if there is a difference?

[josharian]

@fgergo thanks for testing on windows. Though I will switch it preferring binary downloads, I plan to still support installing from source, and silently failing is very bad. Any chance you can help debug a bit? We can discuss over at josharian/goversion#1. Thanks!

As for how to test using 1.8beta2, there are lots of things that people could usefully do. Here are a few:

• Run your tests using 1.8beta. They ought to pass with 1.7.4. If they still pass with the beta, call it good. This should be fast and easy and safe.
• Run your benchmarks using 1.7.4 and the beta. Report any significant regressions. Maybe slow, pretty easy, safe.
• Re-build a production service and run it as a canary. Monitor it for misbehavior, performance issues, etc. Potentially fairly involved, potentially hard, not 100% safe...better done as we move from betas to RCs.

[fgergo]

@josharian sorry about my (deleted) comment about windows' exit command.
Now I believe I understand better what's happening:
make.bat has an error, last 5 lines from make.bat (both 1.4 and 1.8beta1):
:fail
set GOBUILDFAIL=1
if x%GOBUILDEXIT%==x1 exit %GOBUILDFAIL%

:end

Since GOBUILDEXIT is never set, make.bat finishes without error --> goversion assumes go1.4 is ready. (sidenote: all.bat checks for GOBUILDFAIL, that's why I think nobody noticed until now.)

@brad I am not sure how to progress here, change 1.4 make.bat, change tip make.bat?

If I understand the situation correctly: still goversion needs some other mechanism to check for failed make.bat. Will respond on josharian/goversion#1 when I have some sensible hack/workaround. (Maybe run make.bat from a shell script and check for GOBUILDFAIL.)

[rsc]

You could just check if make.bat managed to build Go or not. if bin/go exists, we're good.

[fgergo]

details: josharian/goversion#1
I admit I am not sure what's the general direction, hence:
simplistic pr created for goversion to check for go.exe after make.bat
If somebody is doing the coordination, I'd be happy to know/help etc.
until that: josharian/goversion#3

[bradfitz]

@josharian, still on track to have this ready for Go 1.8beta2?

[josharian]

No. Fixed the windows build-from-source issue last night, and got as far as writing a list-relevant-downloads function. The install-from-download isn't very much work, but I foresee very little time to work on this in the next couple of days. My apologies.

[fgergo]

@bradfitz @josharian what's the scope of "ready"? Binary installs for all platforms listed here: https://storage.googleapis.com/go-builder-data/dl-index.txt? If yes, osx10.6 or osx10.8? Anything else?

[gopherbot]

CL https://golang.org/cl/34385 mentions this issue.

[bradfitz]

I think golang/build@071a06a is probably sufficient for Go 1.8. We can keep this open to track more.

[bradfitz]

This is sufficiently done. We'll keep using golang.org/x/build/version/NNNNN for the betas.