cmd/dist: per-developer scheduling of tests based on past failure rate

[mdempsky]

I don't regularly run go tool dist test (or run.bash or all.bash), because usually either (1) I've broken cmd/compile so badly even hello world will explode, or (2) I've broken something really subtle that's only going to be caught by a few packages / tests, which are really late in the test order. Instead, I keep a test script that I constantly and manually revise based on which failures I'm currently running into during development.

However, I think it would be nice if cmd/dist could track this automatically. E.g., I often break test/run.go and go/importer, so it would be nice if go tool dist test ran those tests earlier.

--

As a strawman proposal to kick things off, I'd suggest adding a -history filename flag to cmd/dist.

A cmd/dist history file is a list of (TestName string, FailureRate float64) pairs. Simplest format would be just line-oriented; e.g., go_test:go/importer 0.75.

If the -history flag is specified, at startup the history file is read in. After the list of tests to run are determined, they're sorted in order of decreasing failure rate.

After running tests is finished, the FailureRate is updated: the new score is 0.8*old if the test passed, and 0.8*old + 0.2 if the test failed. (I.e., exponentially-weighted moving average, with 80% as the weight.) If the test was skipped, then the score is unchanged.

Finally, the history file gets replaced with the new list of failure rates.

[rsc]

In general we've been trying to move complex logic out of cmd/dist. This seems like moving in the opposite direction.

[mdempsky]

I'm indifferent about where the logic lives. Currently it lives in cmd/dist. If we want to move it elsewhere first before making it smarter, that would be fine with me.