Total subtests don't count manual tests

[zqzhang]

In http://w3c.github.io/test-results/vibration/all.html, there is a FAIL of a manual test for one of the two user agents; but it doesn't be counted into total subtests, so http://w3c.github.io/test-results/vibration/less-than-2.html fails to show it.

Actually, I am not sure if this is the proper place to fix this issue; just report here for reminder :)

[darobin]

wptreport just takes whatever data it is given in the JSON; if it does not include manual tests, if the data is wrong, or if there are discrepancies (e.g. some tests run in one UA but not in another) then that won't work.

I see where the problem is coming from in this case, though. Manual tests seem to report themselves as test files but not as test cases. Is that data you generated using the online test runner? If so, there's a bug somewhere in there. Even manual tests should report test cases, otherwise it is indeed as if they don't exist. You'll note that all.html already shows the problem there: both browsers have 100% pass despite the failure.

[zqzhang]

Yes, the JSON data was generated from the online test runner, with manually update the results (e.g. from TIMEOUT to PASS) by re-running the manual tests one-by-one, due to the runner is not so good at manual tests execution.

[darobin]

Ah, changing from TIMEOUT to PASS on a test file will not do the right thing. When there is a time out, there are no test cases listed, so that won't give correct information.

Can you please file bugs in WPT on the issues you've had with the manual running? That will be easier to address. I can look at it there, or possibly @jgraham.

I'm closing here since it is clearly not a wptreport issue.