Export or move the WebGPU CTS into WPT

[kainino0x]

WPT (Web Platform Tests) is the standard repository for... web platform tests. It would theoretically be nice to move the WebGPU CTS into WPT someday. This issue tracks reasons we haven't been able to do so. (Thanks @greggman for suggesting I file this issue.)

Here's a copy of an email I wrote to blink-dev a few days ago:

• If we just export without moving it, then browser-to-wpt auto-export will not be usable, so I would want to move it.
• TypeScript meaningfully enables our test development and we would not want to strip it from the code, so we would want to somehow support that inside WPT. (We actually had an idea involving running babel in a service worker that could make this work now, but it's something we wouldn't want to do just for WebGPU.)
• Each browser's wpt-to-browser auto-import will need to run the WebGPU tests on all hardware/software configurations used on that browser's CQ equivalent, and be able to collect the test results and update expectations files automatically. Without this, auto-imports will frequently be unable to import without breaking the build.
• Chromium doesn't actually run most of the WebGPU CTS through WPT/web_tests anymore - only reftests still use web_tests. For numerous reasons, it was more practical to run under Chromium's GPU integration test framework. Other browsers are integrating the tests in whatever way is most practical for them (for example WebKit can't run it under their WPT runner using the WPT "variants" feature - they need it split up into files or somehow baked into the WPT manifest).
  • The test tree is very large and deep and test runtimes are highly variable, we needed a heartbeat mechanism for timeouts, which WPT does not have.
  • We needed expectations to depend on the GPU hardware and various configurations of Chromium's graphics and WebGPU stack, which we already had in that framework.
  • It uses the real browser instead of content_shell: tests what we ship, and we also had difficult-to-debug flakiness somehow relating to GPU initialization in content_shell.

Mozilla (@jimblandy) noted they have so far been successful running in WPT (although, so was Chromium, for a good while).

Here's a more complete design document from 2021 on what we wanted to improve in WPT if we kept using it:
https://docs.google.com/document/d/1R_acOB_V0xY2GvsLkePkbTLqTvsdBninVwrNA3-gu5A/edit?usp=sharing&resourcekey=0-UlefddeGCk3KrpkumYz7lQ