Add a script to shrink test-wpt output to only unexpected results

[nox]

No idea if this issue should live in saltfs, wpt itself or here.

Cc @metajack.

[Ms2ger]

I imagine we'd want a separate kind of logger; cc @jgraham

[jgraham]

Yeah, you would want to implement a mozlog formatter.

Not having an easy way to tell if a test ran at all, or see log output, seems rather problematic though. My suspicion is that whatever the use case here is, you're trying to solve the wrong problem.

[nox]

@jgraham https://github.com/servo/servo/wiki/Meeting-2016-03-21#automating-failureintermittent-finding

[jgraham]

So it seems like you might just want --log-errorsummary=<file>. That produces a line-by-line json view of only the test failures and critical/error level log messages. It's not ideal for humans but is the right thing to use if you're implementing a bot (it's what treeherder uses for autoclassification, for example)

[nox]

Is that usable to update expectations too?

[jgraham]

No, you still need the raw log for that. Although conceivably someone could perhaps implement log updating based on the error summary in the future.

[nox]

And does it include unexpected PASS?

[jgraham]

Yes (this conversation would maybe be shorter on irc, or by pointing you at https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/mozlog/mozlog/formatters/errorsummary.py )

[jgraham]

(sorry "test failures" was very lax language. I meant "unexpected test results")

[nox]

Cc @mateon1 who is doing a lot of fuzzing and shrinking.

[nox]

Never mind, I can't read.

[nox]

This was more or less done, with @jdm's intermittent stuff. Closing.