x/build/version: make it trivial to obtain and test code against new go releases #18136

Closed
fgergo opened this Issue Dec 1, 2016 · 24 comments

Comments

Projects
None yet
6 participants
@fgergo

fgergo commented Dec 1, 2016

(was: direct email to adg titled "proposal on getting more people to test beta, -rc releases". Adg suggested to continue on golang-dev, but since this is a proposal, golang-dev shall receive a notification with an issue id on the concrete proposal according to https://github.com/golang/proposal.)

Though the goal is the same (to get better test coverage for beta and rc releases), this is a proposal about a process change, not about the complex tool and process change I sent as an rfc to golang-dev https://groups.google.com/forum/#!topic/golang-dev/kycbtvTIBAA

(Original direct email revised for clarity.)

Proposal

to include immediate value proposition (or expectation or hope...) in beta and rc announcements:

  1. if somebody planned to work on something earlier, a personal one-line reply on it (e.g. if it was useful or ready or not useful or whatever)
  2. and/or no more than 1 or 2 personal new favourites from 2-3 tip users in the announcement email (e.g. "1.8 has new go bug, contrary to it's name it helps me. signed me.")

Assumption

(This may be a bit off, but I think the big picture for possible non golang-dev beta or -rc testers is something like this.)
In the morning:

  1. "hmm, a new go beta. Ok almost all previous final go releases were good or more like exceptionally useful for me."
  2. "my current go release is working fine." <--- that's probably high value for a possible non golang-dev beta tester
  3. "simple upgrade to a final is easy. Testing a beta sounds like work I didn't plan and I am not used to it. Let's see, I need to at least: download, install, configure, select what to test, select how to test, then test-test, test, then possibly report in some useful way if something is not like I expect it to be, possibly with a minimal repro. Uhh, that's a lot and doing only some subset of these is not even useful in general."
  4. "ok, what do I get, hmmm I should probably check the draft release notes in the announcement."
  5. ok, let's see... beta.golang.org/doc/go1.8 If I understand correctly, I'll need to read and evaluate all of these to decide if I get any immediate value in testing the beta.
  6. "although I get everything for free and I could give back now by testing the beta. Maybe later. I'll test this later, when it's probably more immediately valuable for me."

To confirm,

nice change in value proposition since the last beta announcement:
"There are no known problems or regressions." <--- that's a high value statement. Although this possibly high value is immediately lowered after clicking and seeing "Known Issues".

@bradfitz bradfitz added the Proposal label Dec 1, 2016

@bradfitz bradfitz added this to the Proposal milestone Dec 1, 2016

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Dec 5, 2016

Contributor

Hmm, OK. It sounds like the underlying issue you want to solve is to make it easier for people to download and run alternate Go versions? Like go test --goverson=1.8beta1 mypkg? There's a lot to unpack there, but that could be useful. It's not clear what to do though.

This wouldn't need to built into the go command. There could be a separate gotry command that did this, downloading versions as needed.

FWIW, I keep all old versions of Go in my home directory and have scripts like:

$ cat $(which go1.5)
#!/bin/bash
export GOROOT=$HOME/go1.5
export PATH=$HOME/go1.5/bin:$PATH
exec $GOROOT/bin/go "$@"

Could be beta.golang.org/go1.8beta1 or just a unified beta.golang.org/gotry or ...

Contributor

rsc commented Dec 5, 2016

Hmm, OK. It sounds like the underlying issue you want to solve is to make it easier for people to download and run alternate Go versions? Like go test --goverson=1.8beta1 mypkg? There's a lot to unpack there, but that could be useful. It's not clear what to do though.

This wouldn't need to built into the go command. There could be a separate gotry command that did this, downloading versions as needed.

FWIW, I keep all old versions of Go in my home directory and have scripts like:

$ cat $(which go1.5)
#!/bin/bash
export GOROOT=$HOME/go1.5
export PATH=$HOME/go1.5/bin:$PATH
exec $GOROOT/bin/go "$@"

Could be beta.golang.org/go1.8beta1 or just a unified beta.golang.org/gotry or ...

@rsc rsc changed the title from proposal: to highlight immediate value in testing beta and rc releases to proposal: make it trivial to obtain and test code against new go releases Dec 5, 2016

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Dec 5, 2016

Contributor

I retitled this to the general underlying problem that seemed to be the target of the golang-dev thread.

Contributor

rsc commented Dec 5, 2016

I retitled this to the general underlying problem that seemed to be the target of the golang-dev thread.

@fgergo

This comment has been minimized.

Show comment
Hide comment
@fgergo

fgergo Dec 6, 2016

Thank you for your comment!
Although I tried to separate 2 ideas (1 proposal on process change on value proposition and 1 about a tool change), I was not successful. I've reread everything I wrote on these ideas, I still don't know what I did wrong.

I try to clarify again:
My assumption was after I've seen both adg's and somebody else's tweets/comments on how few tests the last beta received and how little time remained to fix bugs before 1.7, that at least 2 approaches could be considered to increase test coverage for betas.

  1. make go programmers interested to test when the go project needs tests
  2. make it easy/trivial to make regressions tests with betas on all locally installed packages under $GOPATH/src for anybody with a go installation and publish/report the result without any human assistance.

It's a fact I could not make myself clear on 1., that's why reconsider my proposal for 1: I suggest to forget about that.
About the tool change (2.) in the thread: https://groups.google.com/forum/#!topic/golang-dev/kycbtvTIBAA
I can see there are quite a few details brad had probably already thought about when he replied and most probably rsc as well, I am still collecting thoughts in the google docs document I referred to, before I come up with a proposal.

Please respond how/if I should open/close/merge/whatever this github issue or just leave it as it is.

fgergo commented Dec 6, 2016

Thank you for your comment!
Although I tried to separate 2 ideas (1 proposal on process change on value proposition and 1 about a tool change), I was not successful. I've reread everything I wrote on these ideas, I still don't know what I did wrong.

I try to clarify again:
My assumption was after I've seen both adg's and somebody else's tweets/comments on how few tests the last beta received and how little time remained to fix bugs before 1.7, that at least 2 approaches could be considered to increase test coverage for betas.

  1. make go programmers interested to test when the go project needs tests
  2. make it easy/trivial to make regressions tests with betas on all locally installed packages under $GOPATH/src for anybody with a go installation and publish/report the result without any human assistance.

It's a fact I could not make myself clear on 1., that's why reconsider my proposal for 1: I suggest to forget about that.
About the tool change (2.) in the thread: https://groups.google.com/forum/#!topic/golang-dev/kycbtvTIBAA
I can see there are quite a few details brad had probably already thought about when he replied and most probably rsc as well, I am still collecting thoughts in the google docs document I referred to, before I come up with a proposal.

Please respond how/if I should open/close/merge/whatever this github issue or just leave it as it is.

@minux

This comment has been minimized.

Show comment
Hide comment
@minux

minux Dec 6, 2016

Member
Member

minux commented Dec 6, 2016

@josharian

This comment has been minimized.

Show comment
Hide comment
@josharian

josharian Dec 6, 2016

Contributor

Whipped something up: https://github.com/josharian/goversion. All manner of comments welcome.

Contributor

josharian commented Dec 6, 2016

Whipped something up: https://github.com/josharian/goversion. All manner of comments welcome.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Dec 6, 2016

Contributor

@josharian Nice. I will leave it to you and @bradfitz to figure out what to do next, but I'd love to have instructions like that in the next beta release.

@fgergo, thanks for the clarification. I agree (1) is hard. I think lowering the barrier to entry might be enough. For (2), I was with you up until "publish/report the result without any human assistance." Many users are working on closed source systems. Publishing the results in a way that (a) helps us identify the underlying problem and (b) doesn't reveal details of their code that they don't want revealed is also hard.

What I think we can do (what @josharian put together seems like the right direction) is "make it easy/trivial to make regressions tests with betas on all locally installed packages under $GOPATH/src for anybody with a go installation".

@josharian / @bradfitz, I wonder if goversion should default to downloading binaries from golang.org/dl instead of doing a full rebuild. Should be faster, makes sure we're not debugging misbuilt toolchain problems.

Contributor

rsc commented Dec 6, 2016

@josharian Nice. I will leave it to you and @bradfitz to figure out what to do next, but I'd love to have instructions like that in the next beta release.

@fgergo, thanks for the clarification. I agree (1) is hard. I think lowering the barrier to entry might be enough. For (2), I was with you up until "publish/report the result without any human assistance." Many users are working on closed source systems. Publishing the results in a way that (a) helps us identify the underlying problem and (b) doesn't reveal details of their code that they don't want revealed is also hard.

What I think we can do (what @josharian put together seems like the right direction) is "make it easy/trivial to make regressions tests with betas on all locally installed packages under $GOPATH/src for anybody with a go installation".

@josharian / @bradfitz, I wonder if goversion should default to downloading binaries from golang.org/dl instead of doing a full rebuild. Should be faster, makes sure we're not debugging misbuilt toolchain problems.

@fgergo

This comment has been minimized.

Show comment
Hide comment
@fgergo

fgergo Dec 6, 2016

@rsc on (1): I imagined to automatically publish something like this:
$GOOS, $GOARCH, $GOHOSTOS
local: 1.7.3
testing: 1.8beta2
number of different test results: 0/444
1.7.3 test duration: 26m 12s
go1.8beta2 test duration: 25m 59s

If this information is too sensitive for somebody to publish, goversion should not be run. Maybe this warning could be announced in the first few goversions and then later goversion changed to only produce output if some error happened, similar to any other go tool.

fgergo commented Dec 6, 2016

@rsc on (1): I imagined to automatically publish something like this:
$GOOS, $GOARCH, $GOHOSTOS
local: 1.7.3
testing: 1.8beta2
number of different test results: 0/444
1.7.3 test duration: 26m 12s
go1.8beta2 test duration: 25m 59s

If this information is too sensitive for somebody to publish, goversion should not be run. Maybe this warning could be announced in the first few goversions and then later goversion changed to only produce output if some error happened, similar to any other go tool.

@josharian

This comment has been minimized.

Show comment
Hide comment
@josharian

josharian Dec 6, 2016

Contributor

@rsc yes, goversion should probably download binaries when possible. It is useful to know about build problems, but I guess this is not the venue.

@bradfitz what next, do you think?

Contributor

josharian commented Dec 6, 2016

@rsc yes, goversion should probably download binaries when possible. It is useful to know about build problems, but I guess this is not the venue.

@bradfitz what next, do you think?

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Dec 6, 2016

Member

@josharian, I was going to write that tool today. :-) Thanks!

Yes, please make binary downloads the default. Actually, I'd make build-from-source require a flag. If you run it on plan9, the error message should be something like "go1.8beta1 binary download not available for plan9/386. Use the --build flag.".

And then let's get some testing on golang-nuts / Slack, and then we'll announce it with go1.8beta2.

Member

bradfitz commented Dec 6, 2016

@josharian, I was going to write that tool today. :-) Thanks!

Yes, please make binary downloads the default. Actually, I'd make build-from-source require a flag. If you run it on plan9, the error message should be something like "go1.8beta1 binary download not available for plan9/386. Use the --build flag.".

And then let's get some testing on golang-nuts / Slack, and then we'll announce it with go1.8beta2.

@josharian

This comment has been minimized.

Show comment
Hide comment
@josharian

josharian Dec 6, 2016

Contributor

Where should it live? I don't care particularly, I just want to know what to do with the next round of code.

Looking at golang.org/dl, some questions:

(1) How do I get a list of available versions? Note that e.g. 1.7.2 is not available for download, so I can't just use the git tags like I did for build-from-source.
(2) Is the download naming convention stable enough that I can hard-code it?
(3) Is there a non-screen-scrape way to get checksums, so that I can sanity-check the download?

Basically, I can haz an API for golang.org/dl?

Contributor

josharian commented Dec 6, 2016

Where should it live? I don't care particularly, I just want to know what to do with the next round of code.

Looking at golang.org/dl, some questions:

(1) How do I get a list of available versions? Note that e.g. 1.7.2 is not available for download, so I can't just use the git tags like I did for build-from-source.
(2) Is the download naming convention stable enough that I can hard-code it?
(3) Is there a non-screen-scrape way to get checksums, so that I can sanity-check the download?

Basically, I can haz an API for golang.org/dl?

@gopherbot

This comment has been minimized.

Show comment
Hide comment

CL https://golang.org/cl/34012 mentions this issue.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Dec 6, 2016

Member

@josharian, ignore /dl/?format=json for now. That's too hard to deploy and iterate on.

Here's an easier API: https://storage.googleapis.com/go-builder-data/dl-index.txt

Future note to self:

$ gsutil ls gs://golang/ | sed 's,gs://,https://storage.googleapis.com/,' > index.txt
$ upload -cacheable=false -public -file=index.txt go-builder-data/dl-index.txt
Member

bradfitz commented Dec 6, 2016

@josharian, ignore /dl/?format=json for now. That's too hard to deploy and iterate on.

Here's an easier API: https://storage.googleapis.com/go-builder-data/dl-index.txt

Future note to self:

$ gsutil ls gs://golang/ | sed 's,gs://,https://storage.googleapis.com/,' > index.txt
$ upload -cacheable=false -public -file=index.txt go-builder-data/dl-index.txt
@bradfitz

This comment has been minimized.

Show comment
Hide comment
Member

bradfitz commented Dec 6, 2016

/cc @broady

@fgergo

This comment has been minimized.

Show comment
Hide comment
@fgergo

fgergo Dec 6, 2016

@josharian at the moment goversion won't work on windows without gcc obviously. Though on windows it fails silently currently. I understand that when binary downloads are preferred, this won't be a problem.

@ anybody_who_can_answer: Assuming goversion is ready for 1.8beta2, how would the usage instructions look like? Should the new beta tester run go test for src/, then run goversion ..., then manually diff the 2 test runs? What if no difference? What if there is a difference?

fgergo commented Dec 6, 2016

@josharian at the moment goversion won't work on windows without gcc obviously. Though on windows it fails silently currently. I understand that when binary downloads are preferred, this won't be a problem.

@ anybody_who_can_answer: Assuming goversion is ready for 1.8beta2, how would the usage instructions look like? Should the new beta tester run go test for src/, then run goversion ..., then manually diff the 2 test runs? What if no difference? What if there is a difference?

@josharian

This comment has been minimized.

Show comment
Hide comment
@josharian

josharian Dec 7, 2016

Contributor

@fgergo thanks for testing on windows. Though I will switch it preferring binary downloads, I plan to still support installing from source, and silently failing is very bad. Any chance you can help debug a bit? We can discuss over at josharian/goversion#1. Thanks!

As for how to test using 1.8beta2, there are lots of things that people could usefully do. Here are a few:

  • Run your tests using 1.8beta. They ought to pass with 1.7.4. If they still pass with the beta, call it good. This should be fast and easy and safe.
  • Run your benchmarks using 1.7.4 and the beta. Report any significant regressions. Maybe slow, pretty easy, safe.
  • Re-build a production service and run it as a canary. Monitor it for misbehavior, performance issues, etc. Potentially fairly involved, potentially hard, not 100% safe...better done as we move from betas to RCs.
Contributor

josharian commented Dec 7, 2016

@fgergo thanks for testing on windows. Though I will switch it preferring binary downloads, I plan to still support installing from source, and silently failing is very bad. Any chance you can help debug a bit? We can discuss over at josharian/goversion#1. Thanks!

As for how to test using 1.8beta2, there are lots of things that people could usefully do. Here are a few:

  • Run your tests using 1.8beta. They ought to pass with 1.7.4. If they still pass with the beta, call it good. This should be fast and easy and safe.
  • Run your benchmarks using 1.7.4 and the beta. Report any significant regressions. Maybe slow, pretty easy, safe.
  • Re-build a production service and run it as a canary. Monitor it for misbehavior, performance issues, etc. Potentially fairly involved, potentially hard, not 100% safe...better done as we move from betas to RCs.
@fgergo

This comment has been minimized.

Show comment
Hide comment
@fgergo

fgergo Dec 7, 2016

@josharian sorry about my (deleted) comment about windows' exit command.
Now I believe I understand better what's happening:
make.bat has an error, last 5 lines from make.bat (both 1.4 and 1.8beta1):
:fail
set GOBUILDFAIL=1
if x%GOBUILDEXIT%==x1 exit %GOBUILDFAIL%

:end

Since GOBUILDEXIT is never set, make.bat finishes without error --> goversion assumes go1.4 is ready. (sidenote: all.bat checks for GOBUILDFAIL, that's why I think nobody noticed until now.)

@brad I am not sure how to progress here, change 1.4 make.bat, change tip make.bat?

If I understand the situation correctly: still goversion needs some other mechanism to check for failed make.bat. Will respond on josharian/goversion#1 when I have some sensible hack/workaround. (Maybe run make.bat from a shell script and check for GOBUILDFAIL.)

fgergo commented Dec 7, 2016

@josharian sorry about my (deleted) comment about windows' exit command.
Now I believe I understand better what's happening:
make.bat has an error, last 5 lines from make.bat (both 1.4 and 1.8beta1):
:fail
set GOBUILDFAIL=1
if x%GOBUILDEXIT%==x1 exit %GOBUILDFAIL%

:end

Since GOBUILDEXIT is never set, make.bat finishes without error --> goversion assumes go1.4 is ready. (sidenote: all.bat checks for GOBUILDFAIL, that's why I think nobody noticed until now.)

@brad I am not sure how to progress here, change 1.4 make.bat, change tip make.bat?

If I understand the situation correctly: still goversion needs some other mechanism to check for failed make.bat. Will respond on josharian/goversion#1 when I have some sensible hack/workaround. (Maybe run make.bat from a shell script and check for GOBUILDFAIL.)

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Dec 7, 2016

Contributor
Contributor

rsc commented Dec 7, 2016

@fgergo

This comment has been minimized.

Show comment
Hide comment
@fgergo

fgergo Dec 9, 2016

details: josharian/goversion#1
I admit I am not sure what's the general direction, hence:
simplistic pr created for goversion to check for go.exe after make.bat
If somebody is doing the coordination, I'd be happy to know/help etc.
until that: josharian/goversion#3

fgergo commented Dec 9, 2016

details: josharian/goversion#1
I admit I am not sure what's the general direction, hence:
simplistic pr created for goversion to check for go.exe after make.bat
If somebody is doing the coordination, I'd be happy to know/help etc.
until that: josharian/goversion#3

@rsc rsc added Proposal-Accepted and removed Proposal labels Dec 12, 2016

@rsc rsc modified the milestones: Go1.8, Proposal Dec 12, 2016

@rsc rsc changed the title from proposal: make it trivial to obtain and test code against new go releases to dist: make it trivial to obtain and test code against new go releases Dec 12, 2016

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Dec 12, 2016

Member

@josharian, still on track to have this ready for Go 1.8beta2?

Member

bradfitz commented Dec 12, 2016

@josharian, still on track to have this ready for Go 1.8beta2?

@josharian

This comment has been minimized.

Show comment
Hide comment
@josharian

josharian Dec 12, 2016

Contributor

No. Fixed the windows build-from-source issue last night, and got as far as writing a list-relevant-downloads function. The install-from-download isn't very much work, but I foresee very little time to work on this in the next couple of days. My apologies.

Contributor

josharian commented Dec 12, 2016

No. Fixed the windows build-from-source issue last night, and got as far as writing a list-relevant-downloads function. The install-from-download isn't very much work, but I foresee very little time to work on this in the next couple of days. My apologies.

@fgergo

This comment has been minimized.

Show comment
Hide comment
@fgergo

fgergo Dec 13, 2016

@bradfitz @josharian what's the scope of "ready"? Binary installs for all platforms listed here: https://storage.googleapis.com/go-builder-data/dl-index.txt? If yes, osx10.6 or osx10.8? Anything else?

fgergo commented Dec 13, 2016

@bradfitz @josharian what's the scope of "ready"? Binary installs for all platforms listed here: https://storage.googleapis.com/go-builder-data/dl-index.txt? If yes, osx10.6 or osx10.8? Anything else?

@gopherbot

This comment has been minimized.

Show comment
Hide comment

CL https://golang.org/cl/34385 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Dec 15, 2016

version: add programs to download & run go1.8beta1 and go1.8beta2
This permits a user with any halfway recent version of Go to run:

   $ go get golang.org/x/build/version/go1.8beta1

   $ go1.8beta1
   go1.8beta1: not downloaded. Run 'go1.8beta1 download' to install to /home/bradfitz/sdk/go1.8beta1

   $ go1.8beta1 download
   Downloaded 0.0% (15225 / 90818333 bytes) ...
   Downloaded 50.1% (45514752 / 90818333 bytes) ...
   Downloaded 100.0% (90818333 / 90818333 bytes).
   Unpacking /home/bradfitz/sdk/go1.8beta1/go1.8beta1.linux-amd64.tar.gz ...
   Unpacked /home/bradfitz/sdk/go1.8beta1/go1.8beta1.linux-amd64.tar.gz.
   Success. You may now run 'go1.8beta1'

   $ go1.8beta1 version
   go version go1.8beta1 linux/amd64

   $ go1.8beta1 run ~/helloworld.go
   Hello, world from go1.8beta1!

Updates golang/go#18136

Change-Id: I4297df19138facbf90dfab790ab30b942151b096
Reviewed-on: https://go-review.googlesource.com/34385
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Dec 15, 2016

Member

I think golang/build@071a06a is probably sufficient for Go 1.8. We can keep this open to track more.

Member

bradfitz commented Dec 15, 2016

I think golang/build@071a06a is probably sufficient for Go 1.8. We can keep this open to track more.

@bradfitz bradfitz modified the milestones: Go1.9, Go1.8 Dec 15, 2016

@bradfitz bradfitz changed the title from dist: make it trivial to obtain and test code against new go releases to x/build/version: make it trivial to obtain and test code against new go releases Dec 15, 2016

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Jun 8, 2017

Member

This is sufficiently done. We'll keep using golang.org/x/build/version/NNNNN for the betas.

Member

bradfitz commented Jun 8, 2017

This is sufficiently done. We'll keep using golang.org/x/build/version/NNNNN for the betas.

@golang golang locked and limited conversation to collaborators Jun 8, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.