Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/release{,bot}: include long tests in pre-release testing #29252

Open
bcmills opened this issue Dec 14, 2018 · 5 comments

Comments

@bcmills
Copy link
Member

commented Dec 14, 2018

I tested CL 154101 and the subsequent security patches using a combination of go test -run TestScript/[…] and all.bash. Unfortunately, significant parts of go get (including path-to-repository-resolution) are only exercised in non-short tests, and all.bash by default only runs the short tests, despite the name. (I remember that latter point occasionally — but apparently not frequently enough.)

Even more unfortunately, releasebot suggests all.bash for security releases as well, and release runs the same all.bash commands as the regular builders.

As a result, a significant regression (#29241) made it all the way through development, code review, and release building without running the existing tests that should have caught it.

We should ensure that the commands release executes and the instructions releasebot prints for both kinds of releases include the non-short tests on at least one platform.

(CC @bradfitz @dmitshur @FiloSottile)

@gopherbot gopherbot added this to the Unreleased milestone Dec 14, 2018

@gopherbot gopherbot added the Builders label Dec 14, 2018

@bradfitz

This comment has been minimized.

Copy link
Member

commented Dec 14, 2018

Additionally: normally we also look at build.golang.org for a happy row of "ok" before doing a release, but because this was a security release without any of our normal infrastructure, we didn't have build.golang.org and didn't have the existing linux-amd64-longtest builder.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Dec 14, 2018

The longtest builder is highlighted here and shows the breakage, which we would've been much more likely to see using the normal infrastructure:

screen shot 2018-12-14 at 6 03 12 am

@bcmills

This comment has been minimized.

Copy link
Member Author

commented Dec 14, 2018

Yep, there were a lot of factors at play. This one seems like a relatively easy win, though: we don't cut all that many releases, so “run all of the tests one last time to be sure” seems like a reasonable step in the release process.

@bcmills

This comment has been minimized.

Copy link
Member Author

commented Aug 21, 2019

@bcmills bcmills modified the milestones: Unreleased, Go1.14 Aug 21, 2019

@bcmills

This comment has been minimized.

Copy link
Member Author

commented Aug 21, 2019

Marking release-blocker for 1.14. We really ought to be running all of the tests before we cut a release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.