x/build: reduce ambiguity in long test of main Go repo

[dmitshur]

Taking into account the build infrastructure at build.golang.org, local development, and issues such as #26473, #34707, I understand we currently have the goal of having good support for two well-known testing configurations (for a given GOOS/GOARCH pair) for the main Go repository:

• Short, as implemented by all.bash
  • target of 3-4 minutes, acceptable to skip slow tests as implemented by go test -short
• Long, as implemented by -longtest post-submit builders (e.g., linux-amd64-longtest)
  • no goal of skipping tests for performance reasons

While investigating #29252, I found that there is some ambiguity in what it means to test the main Go repo in long mode. It's not easy to say "in order to run a long test, do this" and have a predictable outcome. We currently say "run all.bash and go test std cmd" in some places, but there's room for improvement.

We want to ensure long tests are passing for Go releases. To support that goal, I think it will helpful to reduce ambiguity in what it means to run a long test on the Go repo.

This is a high level tracking issue for improvements in this area, and for any discussion that may need to happen.

/cc @andybons @cagedmantis @toothrot @golang/osp-team @bradfitz @rsc

[gopherbot]

Change https://golang.org/cl/233898 mentions this issue: cmd/dist: use GO_TEST_SHORT value more consistently

[gopherbot]

Change https://golang.org/cl/233901 mentions this issue: dashboard: use more consistent definition of longtest builder

[gopherbot]

Change https://golang.org/cl/234531 mentions this issue: cmd/releasebot, cmd/release: include long tests in release process