Enhancement: Add external feed source for non-browser tests.

[faceless2]

This is an idea based on comment by @svgeesus at w3c/csswg-drafts#6435 (comment)

We now have on WPT automated running of all tests - but in four browsers, three of which use different engines, and with no way to inject non-browser test results.

This issue is to see if there any interest in adding an "External Feed" source to the list of sources? This would allow vendors running the WPT tests in an environment that is not browser based to supply results. This kind of thing:

I don't know how the WPT works internally but I presume this would involve coming up with a XML/JSON format that described the product, version, and for each test run:

• name of test
• pass or fail
• URLs of bitmap image that was generated and URL of reference image it was compared to (if applicable to the test)

Interested vendors could then publish their results to well-known URL, and the WPT could reference that URL as an optional source. We're already running these tests (link) and would do this. I believe Weasyprint run them too (link), pinging @liZe in case it's of interest.

The reason I think this is useful is that there are some aspects of the CSS tests that cannot be tested in a browser - anything to do with paged media is the most obvious example, but also things like Lab colors (native to PDF), and I presume once the structure is in place it could be useful for specifications like https://www.w3.org/TR/css-round-display-1/ that may require specialised environments. Testing against the WPT is a good thing and is should be encouraged, regardless of platform.

The reason I'm suggesting this approach rather than supplying some sort of docker image (or similar) to run locally is that there's a bit more manual intervention required. For example, lots of tests involve drawing a green rectangle over a red background and claiming there should be no red. The only way to verify this in a PDF is to convert it to bitmap, and anti-aliasing (PDF coords are all floating point) means that there's often a hint of red, so that test has to be approved manually. Some tests also make assumptions about the WPT environment that don't hold up - for example if they use absolute paths or make assumptions about which fonts are available on a platform. So we have to patch some tests before running. And finally, all the commercial CSS-to-PDF vendors put some sort of stamp on the output created with their demo versions, which would ruin the image comparisons.

If this is of interest I'm more than happy to discuss requirements for the feed, but I wouldn't know how to add this to WPT.

[svgeesus]

Thanks for linking to your WPT test results,@faceless2

I agree that this would be of interest. The old Shepherd-based CSS test suite, for example, had this feature. Here are the test results for CSS Color 4 which include the results from your product, BFO Publisher in addition to the major browsers. (Safari, and BFO Publisher, were the first two implementations to takle the majority of that specification).

However, the Shepherd import of WPT tests has been broken for over a year so I no longer refer to results from that source.

As you say, certain features are a higher priority for non-browser contexts and it would be good to get earlier spec feedback by way of early implementations.

[liZe]

Thanks for pinging me!

We would definitely be interested in a solution to test and include the results of WeasyPrint.

The reason I think this is useful is that there are some aspects of the CSS tests that cannot be tested in a browser - anything to do with paged media is the most obvious example, but also things like Lab colors (native to PDF), and I presume once the structure is in place it could be useful for specifications like https://www.w3.org/TR/css-round-display-1/ that may require specialised environments. Testing against the WPT is a good thing and is should be encouraged, regardless of platform.

I agree. There are many differences that make testing on paged media difficult, and I would be happy to talk about the ways we could use the WPT more easily on specialized environments. We have tried to automatize these tests for WeasyPrint but we face small problems that prevent us from using the platform reliably.

[liZe]

I have played with wpt today and it worked quite well once everything was set up correctly. That’s impressive!

I then tried to read the code, and even if Python is my favorite language, even if there’s a lot of useful documentation, I’m a bit lost. There are some questions I’d like to ask before trying to go further, I’m sorry if the answers are obvious:

• wpt run launches automatic tests according to the documentation, but I don’t understand how it can be automatic. I know that some tests have a reference, but in CSS at least it wasn’t the case for many tests. Is there another way to get automatic test results? How are the results for non-automatic tests gathered?
• Would it be useful to add the support of a non-browser solution in wptrunner? If so, could it be possible to have a short introduction about what’s required to add a new "browser"?
• I don’t understand how tests and their metadata (such as their tags) are discovered. Is there a file hidden somewhere listing all the tests, or are they automatically found by listing the folders and their children recursively?
• Which format could we use to store tests results?
• I have the impression that more and more tests to rely on JavaScript support. Is this just an impression, or is it a common trend for newer test suites?

I’ll probably have some time during the next weeks, don’t hesitate to ping me if there’s anything you’d like to discuss or if there’s anything useful I can do.

[foolip]

@faceless2 @liZe if you can provide results in the same wptreport.json format that ./wpt run produces for browsers, then it should be pretty easy to upload results to wpt.fyi. There's an API for this described here:
https://github.com/web-platform-tests/wpt.fyi/blob/main/api/README.md#results-creation

You'll also need credentials to upload, I suggest reaching out to @past to arrange that.

[liZe]

Our work on a platform dedicated to CSS-Print is going well. It’s now able to both launch automatic tests and let users review and correct errors. It will also soon be able to automatically test new versions of the testing suite and of the tested libraries by comparing results to older versions, so that we won’t always have to check everything manually.

It should now generate the same files as WPT. Here is a sample: weasyprint-57.2.zip

We would like to release the platform as open source software in the future, so that other tools can use it to test and generate reports too. It’s been designed in such a way that it is quite easy to be interfaced with other CSS-Print solutions.

So… Hi @past! We’d like to ask you a few questions:

• Is there a way to test that our file has the right structure? We have a sample for CSS tests attached if you’re interested.
• Could it be possible to have credentials to upload the results for future versions, when everything works well?
• What’s the screenshot_file parameter? We’ve tried to generate a screenshot database with wpt run --log-wptscreenshot=filename but it only generates an empty file for us.

Thanks a lot!

(And if anyone is interested in discussing about the tool itself, don’t hesitate to contact us!)