Write a tool that extracts failures from WPT test logs

[jdm]

This will be easy to integrate with https://github.com/jdm/highfive to automatically report test failures to the PR when they occur.

Relevant prior art for dissecting logs: https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/mozlog/mozlog/scripts/unstable.py

[AgostonSzepessy]

I'd like to try this one. If I start working on this, where should I put the code in the source tree?

[wafflespeanut]

Let's put it inside python/ :)

[AgostonSzepessy]

I have a question: am I supposed to extract the failures from the raw log files generated by this command: ./mach test-wpt --log-raw=/path/to/some/logfile? If not, then which log file am I supposed to extract the failures from?

[wafflespeanut]

Yep, that's the one. If we don't pass --log-raw, then ./mach test-wpt command automatically outputs something like

▶ TEST-RESULT [expected RESULT]
// traceback

whenever it encounters a mismatch in the results^[1]. The argument makes mach write it to the log file. So, we have two choices. We could either generate the log file and let the tool analyze it, or (more easily) just pipe the output from stdout into this tool, and do stuff along the run. I'm not sure what @jdm had in mind.

^{[1]: The intermittent issues might give some insight on the different test results.}

[wafflespeanut]

No, we should probably go for analyzing the generated log file - piping doesn't make any sense here.

[jdm]

I think we should parse the default stdout output, since the raw logs are 50-100mb and will be slow to transfer.

[jdm]

And specifically I'm thinking of the stdio links from buildbot results in PRs.

[jdm]

Any progress @AgostonSzepessy? Any questions?

[AgostonSzepessy]

@jdm Yes, I've started writing the code. I've run the wpt-tests and saved the output to a file so I can just cat that and feed the output into the program by piping it in. Is that the way that I should do it? Also, should I format it any specific way?

[jdm]

That sounds like you're doing the right thing :) Formatting the output in structured JSON or something would be valuable.

[AgostonSzepessy]

And one more thing, do I need to extract the tests don't have status of FAIL or CRASH? So for example in this file: http://pastebin.com/5t5Na4Yk (the one I'm using), would I need to extract the tests that TIMEOUT?

[nox]

AFAIU, we want all unexpected results, be them FAIL, CRASH, TIMEOUT or even PASS.

[wafflespeanut]

@AgostonSzepessy Are you still working on this?

[AgostonSzepessy]

Yes I am. Sorry it's been so slow; I've been really busy with end of school projects and studying for exams. I need to check for a few more tests that I haven't checked for yet and clean up the code a bit, and then it should be ready. The program produces this output when I pipe in the tests:http://paste.pound-python.org/show/AGzKKwVddQqHjGTapm94/
Here's a cleaned up version (not from the program): http://paste.pound-python.org/show/OFD9jIeMeqnaRSjQDOb2/

Is this what's required?

[wafflespeanut]

No worries, just wanted to check :)

To clarify, if the script is exposed to an stdout like this, we'll get a JSON like the one you've shown, right?

[AgostonSzepessy]

Yes. When you run ./mach test-wpt | python/wpt_test_reader.py, it will look for lines that are similar to:

▶ TEST-RESULT [expected RESULT]
| // test info

And then parse them.

[wafflespeanut]

@jdm I guess it's time to take this to highfive?

[wafflespeanut]

@highfive can do this now even though it's intermittent.