testing: limit the number of errors

[josharian]

Exhaustive tests (such as found in math/big and strconv) can generate a phenomenal number of errors when they fail. At some point, there is no value in additional errors.

I think that we should maintain an error counter in testing.T and call t.Fatal("too many errors") once we hit (say) 100 errors in a test.

cc @adg and @robpike for feedback. I'll implement if you're ok with it, since this is a pain point for SSA development.

[robpike]

go test | sed 100q seems good enough to me. If you add features like this, you need features to control the features, new features are justified because they are in the same style, and they need more features to control them, and it never stops.

[josharian]

Ok. FTR, on digging, a large part of the pain is that I'm running with GOGC=off (by necessity) and all the output is buffered (not fixed by sed), which ends up causing massive memory usage.

[robpike]

The place to fix that is certainly not with new flags in the testing package.

[josharian]

I didn't have in mind new flags, FWIW, just an unexported counter field in testing.T and something at the beginning of Error(f) like

t.nerr++
if t.nerr > 1000 {
  t.Fatal("too many errors")
}

But it's not really worth more words.

[robpike]

But you know someone will then ask to have a flag to change that 1000. Where to draw the line?

[josharian]

Yep. :/