Allow Firefox & Chrome to use fake media stream

[Hexcles]

This is yet another difference in results I observed between BuildBot and Taskcluster.

This was added to wpt.fyi in web-platform-tests/results-collection#504 .

[foolip]

Thanks! Perhaps @jugglinmike would also like to review?

[jgraham]

Honestly I think this should be in https://searchfox.org/mozilla-central/source/testing/profiles/unittest/user.js or the wpt-specific equivalent that @whimboo is adding. But I guess here is OK temporarily if it's inconvenient to make an upstream patch.

[Hexcles]

I'd be happy to make a patch once @whimboo creates the wpt-specific profile, and I'll remove this afterwards. In the interim, let's land this to make Taskcluster results more useful.

[whimboo]

My patch has already been landed, and was merged upstream via #13050.

[Hexcles]

Sent a patch to m-c: https://phabricator.services.mozilla.com/D6579

[Hexcles]

Darn, meant to squash-merge but clicked the wrong button. Sorry.

[jugglinmike]

I'm a little late, but I'm still interested in hearing your perspective.

This makes the browser behave less realistically, so I'm concerned about making it the default behavior for users of the WPT CLI. It might be confusing for folks to understand discrepancies in results reported by the CLI and by running manually or with a different automation tool. And if they wanted to learn more, they'd have to research the source code (rather than read a log file).

What do you think about maintaining this at the edges? Specifically, that would mean adding the browser-specific flags to the TaskCluster script.

[Hexcles]

Sorry for the rush, @jugglinmike .

What do you think about maintaining this at the edges? Specifically, that would mean adding the browser-specific flags to the TaskCluster script.

IMHO, this merely shuffles the confusion around, from discrepancy between manual runs (or other runners) and wpt run, to discrepancy between wpt.fyi and wpt run.

Our current setup until very recently (i.e. on BuildBot) is more like the latter, but I'd argue that the former is slightly better, for the following reasons:

• wpt run is the canonical way to run tests. Running tests "manually" (without automation) isn't trivial, as one needs to at least change settings like popup blockers, certificates, etc. And it's the other runners' responsibilities to replicate upstream's behaviours as they made the tradeoff not to use the upstream runner (as a maintainer of the Blink test runner, I can say this).
• Firefox already has some special prefs for testing, e.g. with a few experimental features enabled. In other words, browsers started by wpt run have always been behaving somewhat differently from the defaults.
• However, the difference between wpt run and wpt.fyi is more surprising. One would expect a dashboard for WPT should be an honest representation of results from the wpt runner. Yet we have some magic flags specific to wpt.fyi that aren't documented and would require digging into another repo to find out.
• The lower level an engineer goes, the more they'd tend to know. And I'd say running the test manually is one level lower than using the wpt frontend, and is most likely for debugging. For someone debugging WebRTC tests, they have a better chance to know fake devices aren't enabled by default (and how to enable it).

Feel free to respond to my reasoning above. This is certainly something worth discussing!

[jgraham]

I fully agree with what @Hexcles says.

I agree that there's a concern about making the test configuration too different from the real browser, and I would like to reduce the number of differences we have for Firefox by cutting down the profile settings we apply (removing ones that just enable features that aren't on by default in nightly for example). However, enabling fake input streams so that media tests can work seems pretty clearly on the side of acceptable test configuration.

[jugglinmike]

Yet we have some magic flags specific to wpt.fyi that aren't documented and would require digging into another repo to find out.

We've been talking about providing a link to log files on wpt.fyi. That's the feature I've had in mind when comparing the discoverability of these two solutions. To me, logs are a more intentional way to communicate details about the runtime (compared with program source code, however well documented).

That's the third time we've mentioned documentation. I appreciate your perspective as a downstream consumer, and docs seems particularly relevant in that case. The concerns I have about result veracity would be at least partially mitigated if we were more forward about the things we do to support automation.

Before we can get there, I think a more general restructuring is in order--see gh-13125.

In the immediate term, we were previously recommending the use of these arguments for Chrome, so that should be removed: gh-13126