Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add WPT appendix listing all test results by section #2216

Open
tabatkins opened this issue Jan 14, 2022 · 3 comments
Open

Add WPT appendix listing all test results by section #2216

tabatkins opened this issue Jan 14, 2022 · 3 comments

Comments

@tabatkins
Copy link
Collaborator

See the Shepherd test results, nicely grouped by spec section, versus the "everything at once in alpha order" chaos of the wpt.fyi results.

I should add an Appendix that looks like the shepherd results.

@svgeesus
Copy link
Contributor

That would be excellent.

Is there a way to choose whether the stable or experimental versions of browsers are tested?

@svgeesus
Copy link
Contributor

So the Shepherd tests have bitrotted and are no longer useful - tests added in the last year or so are not visible in the test report and also can't be run, so there are no results. Thus, the little test result icon at the top of the spec is currently actively misleading.

However, the automatically formatted "by spec section" is exceptionally useful in terms of presenting a CR implementation report and also for checking spec coverage.

The WPT tests require the extra step of adding <wpt> elements into the bikeshed source, which wasn't needed in Shepherd (the links were from the test to the spec); however having done so, there is now a nice per-section list of tests which looks readily harvestable to build an implementation report (appendix, or standalone spec; both would be useful).

However, the WPT tests have the disadvantage that tests for all levels are in the one folder, which requires spec-editor effort to add <wpt-ignore> blocks for older and newer levels.

@svgeesus
Copy link
Contributor

svgeesus commented Nov 1, 2022

I should also add that, based on a recent experience where WPT were re-organized, it is now harder to write test for just one specific section because they will get merged with others. The emerging pattern seems to be "one parsing test for all valid values of one property".

However, I would still love to see this bikeshed-generated report because apparently fixing the Shepherd-generated ones will require additional rounds of future funding and future work so does not seem likely to happen in the near term.

An example of how far apart the WPT and Shepherd views of test status are:

image

Shepherd says zero tests; WPT says 3160 tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants