Hardware and Testing capabilities for mobile environments

[karlcow]

Hi https://github.com/orgs/web-platform-tests/teams/wpt-core-team!

(This is related to a discussion started by @jgraham in web-platform-tests/interop-2022-viewport#23 (comment) related to viewport testing)

Context

The availability of appropriate hardware for testing on mobile is key to be able to get meaningful results for some areas such as viewport. If the WPT project wants to explore further, there exist a number of issues.

Issues

• The current hardware setup (taskcluster or Azure) doesn't make it possible to test properly for viewport tests.
• Android and iOS have different hardware and infrastructure requirements.
• What would be the yearly budget for these infrastructures? (Ballpark)
• Are there other blockers, or at least difficulties to identify ?
• Would it be possible to gain resources and/or budgets to be able to run these tests.

[jgraham]

Some more details about what Gecko/Firefox does for Android testing:

We're running wpt in CI in the Android emulator. wptrunner runs on the host machine, and the frontend launches the emulator and ensures that the browser is installed (in practice we use a "geckoview test runner" application rather than actual Mobile Firefox, but in principle either could work).

The emulator requires KVM for reasonable performance, which in turn means that it needs to be run on bare-metal type cloud instances, or those with support for nested virtualisation, rather than standard VM-based cloud instances. Typically these are more expensive (but I don't know actual numbers).

Generally I think this setup works well for regression testing; compared to previous setups involving specific hardware it's very reliable and catches a large number of Android-only issues. I'm not aware of any concern that we might be missing significant problems due to the use of the emulator (obviously perf testing, as well as regression testing very hardware-specific pieces of code e.g. the JS JIT, have different considerations, but I don't think those are relevant to general wpt tests).

[gsnedders]

On the iOS/iPadOS side:

For on device testing, the only possible version of WebKit that can be tested is the system-provided version. This requires a macOS host machine to run safaridriver and communicate with the embedded device (and minimising latency is probably beneficial there).

For simulators, we can rely on Azure Pipelines or similar for this: there's no similar complexity as there is for Android. There definitely are differences between the simulator and real devices, but the vast majority of behaviour should be accurate. (And on Apple Silicon hosts, you of course have very little hardware difference.)

However, while you can run WebKitTestRunner on simulators, there are differences in WebKit between builds with the Apple internal SDK and the public SDK, along with any differences between WebKitTestRunner's usage of WebKit versus Safari. And, inevitably, this still reflects behaviour of tip-of-tree WebKit on the current shipping iOS, which isn't a configuration we actually ship—hence any changes in libraries used by WebKit won't be reflected in the results.

[jgraham]

So it sounds like setting up WebKit/iOS testing is the most tractable to get running at all (we can just set up the simulator on existing infrastructure, which is work, but just software work), and the hardest to get completely right (because we can't use something that will behave just like actual Safari). So from an Interop point of view, a question is whether Apple would be content with what we can achieve being used as the WebKit/Safari score, or whether there's an alternate proposal if not.

[jgraham]

I made an Interop 2023 proposal for this work. I think it's helpful to have set out the reasoning for making this a priority, and the roadmap for the work, even if we eventually decide to approach it in a different venue.

[gsnedders]

So from an Interop point of view, a question is whether Apple would be content with what we can achieve being used as the WebKit/Safari score, or whether there's an alternate proposal if not.

It being used as a score for WebKit is likely acceptable; it being used as a score for Safari won't be.

That said, I don't think it has the same behaviours as Safari when it comes to collapsing UI on scroll, nor does it have touch events (or pointer events, etc.) enabled. Given some of the interest prior to Interop 2022 was for Pointer Events, that does somewhat limit usefulness.

[WeizhongX]

For Chrome Android:
We are now running WPT on chrome android in Chromium CI, then upload WPT results to wpt.fyi. We are running the tests with Android Emulator. With 36 linux bots, each run takes about 2 hours to finish (with --no-restart-on-unexpected enabled). At this point we are uploading results once each day. We plan to make a change to make this once per three hours, to generate more comparable results with desktop browsers.

The chrome android binary is compiled from ToT. We are not running against a stable chrome android release. To do that we need to download the binary from google play store.

For Chrome iOS, we are working on this now. We will use simulator to run the tests. Once we have set up the builders, we will use similar mechanism as Android side to upload results to wpt.fyi.