Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
GitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
cmd/dist: failure of test in "Testing packages" phase causes remaining tests to be skipped #12670
If one of the tests during the "Testing packages" phase of "go tool dist test" fails, remaining test phases are skipped.
For example, on Solaris, due to issue #12115:
We see that it exits almost immediately after the "Testing Packages" phase completes.
Once I fixed issue #12115, suddenly, these additional tests were executed:
This seems undesirable, as it means that even a minor failure in the "Testing packages" phase could result in all remaining tests being skipped, allowing bitrot to set in while the failing test is still being resolved.
However, I can see how this might have been intentional in some cases with the logic being that if any of those tests failed, there wasn't any point in running the remaining. Since I was uncertain of the intent, I've opened this bug.
Personally, my desired alternatives, ranked by preference would be:
There is a keep going even if there are test failures option for cmd/dist. go tool dist test -k I think this issue is working as intended. If the std library tests are failing, there isn't much point in continuing further. And the platform maintainer should fix the test asap. (Even if the builders do continue after the first failure, i don't think it will prevent bitrot as people will see that the builder has always been failing and just won't bother to see if there is any new failures caused by their commit.) I also remember there are rejected proposal to make the builder use -k.
If the current behaviour is really intended, or still desirable, then I'd like to request the addition of a warning message on exit to dist, something like:
The current situation leaves contributors unfamiliar with the current subtle behaviour in a bad place, and sometimes I would imagine even experienced contributors might forget about it.
I'm happy to make that change myself if it's agreeable, but I feel like something should be done as the current situation can hide errors subtly.
This is working as intended. Once we find something wrong there's no need to continue enumerating things that are wrong. You're not supposed to stay in a broken state like that for long. You're certainly not supposed to send CLs unless all.bash passes. This is not hiding all.bash failures, because it is an all.bash failure.
@rsc respectfully, I disagree. While it may be obvious to long-time project members that there were other failures, it was not obvious to me, nor apparently anyone else testing on Solaris, since no one noticed the other test failures until I started to get inquisitive about them.
Most of the test suites I've interacted with in the past generally provide a summary at the end to make this clear, for example, they might say "10/100 tests passed" or "remaining tests skipped".