-
Notifications
You must be signed in to change notification settings - Fork 199
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add WPT appendix listing all test results by section #2216
Comments
That would be excellent. Is there a way to choose whether the stable or experimental versions of browsers are tested? |
So the Shepherd tests have bitrotted and are no longer useful - tests added in the last year or so are not visible in the test report and also can't be run, so there are no results. Thus, the little test result icon at the top of the spec is currently actively misleading. However, the automatically formatted "by spec section" is exceptionally useful in terms of presenting a CR implementation report and also for checking spec coverage. The WPT tests require the extra step of adding <wpt> elements into the bikeshed source, which wasn't needed in Shepherd (the links were from the test to the spec); however having done so, there is now a nice per-section list of tests which looks readily harvestable to build an implementation report (appendix, or standalone spec; both would be useful). However, the WPT tests have the disadvantage that tests for all levels are in the one folder, which requires spec-editor effort to add <wpt-ignore> blocks for older and newer levels. |
I should also add that, based on a recent experience where WPT were re-organized, it is now harder to write test for just one specific section because they will get merged with others. The emerging pattern seems to be "one parsing test for all valid values of one property". However, I would still love to see this bikeshed-generated report because apparently fixing the Shepherd-generated ones will require additional rounds of future funding and future work so does not seem likely to happen in the near term. An example of how far apart the WPT and Shepherd views of test status are: Shepherd says zero tests; WPT says 3160 tests. |
See the Shepherd test results, nicely grouped by spec section, versus the "everything at once in alpha order" chaos of the wpt.fyi results.
I should add an Appendix that looks like the shepherd results.
The text was updated successfully, but these errors were encountered: