Pass browser-specific command line flags when running webrtc/

[foolip]

http://wpt.fyi/webrtc is almost entirely red, often because getUserMedia() needs permissions and user interaction. Example: http://wpt.fyi/webrtc/simplecall.html

This is obscuring the “real” status of the tests, and makes it impossible for @guidou, @henbos and other implementers to effectively use the dashboard to prioritize their work.

The "proper" fix for this is:

• WebDriver extension for permissions (in review, @JKereliuk will implement in ChromeDriver in Q4)
• WebDriver extension for device mocking (@agouaillard to send PR, ChromeDriver implementation possibly in Q1)
• Wrapping both of these in the infrastructure that @gsnedders is working on
• Rewriting the tests

However, this will at earliest be done some time in early 2018. We should get useful results for the webrtc/ directory ASAP, and then work on paying back that technical debt.

@jgraham FYI. If this will require some changes in wpt runner I hope you think the above approach makes sense.

@patrickkettner @jan-ivar @youennf, we may need your help in picking the right set of command line flags for Edge, Firefox and Safari. Or @agouaillard, you know them, can paste them here?

[agouaillard]

*chrome*: copied from https://webrtc.org/testing/ I think we should update that page and keep it current """ For manual development and testing, here are some command line flags that are useful for WebRTC-related testing:

…

--allow-file-access-from-files allows getUserMedia() to be called from file:// URLs.

--disable-gesture-requirement-for-media-playback removes the need to tap a <video> element to start it playing on Android.

--use-fake-ui-for-media-stream avoids the need to grant camera/microphone permissions.

--use-fake-device-for-media-stream feeds a test pattern to getUserMedia() instead of live camera input.

--use-file-for-fake-video-capture=path/to/file.y4m feeds a Y4M <http://wiki.multimedia.cx/index.php?title=Y4M> test file to getUserMedia() instead of live camera input. """ missing args we use in KITE: - enable-features=WebRTC-H264WithOpenH264FFmpeg - allow-file-access-from-files - use-fake-device-for-media-stream - disable-translate - no-process-singleton-dialog - mute-audio - enable-experimental-web-platform-features - headless - disable-gpu - no-sandbox - use-fake-ui-for-media-stream - auto-select-desktop-capture-source= - load-extension= Note that exceptions can be used to bypass security prompt already ....... https://bugs.chromium.org/p/chromium/issues/detail?id=459532 *Firefox:* all through profiles, the main options are: - media.navigator.streams.fake - media.navigator.permission.disabled there are also a few options for H264. *Safari:* most of the settings needs to be done through the command line on the host system, and not as command line arguments to the safari binary. the tokbox team in Sydney got most of them in a shell script: https://github.com/aullman/travis-multirunner/blob/26838e9c6e1765ad31e0b1e118178ac786cca14a/start-safari.sh from memory all but one are accessible from the shell command line, but one is still only accessible from the dev menus: Screen-Shot-2017-06-16-at-3.20.30-PM.png <https://i1.wp.com/webrtcbydralex.com/wp-content/uploads/2017/06/Screen-Shot-2017-06-16-at-3.20.30-PM.png> if you want to use safaritechpreview, there is a flag in selenium: setTechnologyPreview( bool ) @youennf must know, and we can speak about it in spain next week.

[youennf]

Safari version would need to be upgraded first to Safari 11 or STP.
The script @agouaillard mentioned should work for Safari11.
For STP, it might be good to use some recently added WebDriver options.

[agouaillard]

can you point us to the recently added WebDriver options?

…

On Wed, Sep 27, 2017 at 10:51 AM, youennf ***@***.***> wrote: Safari version would need to be upgraded first to Safari 11 or STP. The script @agouaillard <https://github.com/agouaillard> mentioned should work for Safari11. For STP, it might be good to use some recently added WebDriver options. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#125 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAT1nse4h9Z7Oq4kzjO57U4_SsRPwip2ks5smm7-gaJpZM4Plkbi> .

-- Alex. Gouaillard, PhD, PhD, MBA ------------------------------------------------------------------------------------ President - CoSMo Software Consulting, Singapore ------------------------------------------------------------------------------------ sg.linkedin.com/agouaillard -

[foolip]

Thanks @agouaillard, that's very helpful.

@JKereliuk, I think the minimum set of flags for Chrome would be --use-fake-ui-for-media-stream and --use-fake-device-for-media-stream, corresponding to the two WebDriver extensions that are needed.

@youennf, we'd love to have Safari 11 or STP as part of the dashboard. Having them available on Sauce would fix that. I've reached out to y'all over email again.

[rwaldron]

This is excellent—I will do whatever I can to be helpful and offer myself as a resource in achieving these goals.

[kereliuk]

@agouaillard @youennf I'm not sure which options you are talking about?
You can pass capabilities to ChromeDriver, or you can pass browser options to the chromeOptions object. Which (if either) do you mean?

[jgraham]

I wonder if it is possible to add the firefox prefs to http://searchfox.org/mozilla-central/source/testing/profiles/prefs_general.js; we already apply those testing prefs when running wptrunner.

Otherwise, if you only want to apply these prefs to the WebRTC tests, the easiest way is to set up the dashboard with some metadata ini files. We already support setting Fx prefs like [1], and I feel like I might have implemented passing binary args at some point for servo, and could certainly make that work for other browsers.

[1] http://searchfox.org/mozilla-central/source/testing/web-platform/meta/webrtc/__dir__.ini#1

[agouaillard-cosmo]

@JKereliuk , @youennf is speaking about safari. I'm speaking about chrome.

We use chromeOptions -> addArgument( ) for each command line flag:

switch (browser.getBrowserName()) {
      case "chrome":
        ChromeOptions chromeOptions = new ChromeOptions();
        chromeOptions.addArguments("use-fake-ui-for-media-stream");
        chromeOptions.addArguments("use-fake-device-for-media-stream");
        capabilities.setCapability(ChromeOptions.CAPABILITY, chromeOptions);
        break;
      case "firefox":
        FirefoxProfile firefoxProfile = new FirefoxProfile();
        firefoxProfile.setPreference("media.navigator.streams.fake", true);
        FirefoxOptions firefoxOptions = new FirefoxOptions();
        firefoxOptions.setProfile(firefoxProfile);
        capabilities.merge(firefoxOptions.toCapabilities());
        break;
      case "MicrosoftEdge":
        capabilities.setCapability("avoidProxy", true);
        break;
      case "safari":
        SafariOptions options = new SafariOptions();
        options.setUseTechnologyPreview(true);
        capabilities.setCapability(SafariOptions.CAPABILITY, options);
        break;
    }

[foolip]

web-platform-tests/wpt#7424 is relevant to this as well.

[foolip]

We should probably include https://wpt.fyi/mediacapture-streams as well, since it would be trivial.

[foolip]

@kereliuk, this is now done for Chrome, but not Firefox, right?

[jugglinmike]

@kereliuk, this is now done for Chrome, but not Firefox, right?

The results collector currently specifies the parameters mentioned above, and, Chrome is reported as passing some WebRTC tests on wpt.fyi. Chrome continues to fail most WebRTC tests, but this matches my experience when manually executing tests hosted on w3c-test.org. Still, I'm not sure if this is an accurate reflection of the implementation status or another issue with the testing environment.

@foolip: to close this issue, I'd like to get confirmation from a Chromium implementer. Could you introduce me to someone who could sign off on the accuracy of the results?

Firefox:

all through profiles, the main options are:

• media.navigator.streams.fake
• media.navigator.permission.disabled

I've attempted to set these commands on a "headless" Ubuntu system running Firefox 59.0.2:

xvfb-run --auto-servernum \
  ./wpt run --include webrtc/RTCDTMFSender-ontonechange-long.https.html \
    firefox \
    --setpref 'media.navigator.streams.fake=true' \
    --setpref 'media.navigator.permission.disabled=true'

But that test continues to time out in the environment. This conflicts with the results I receive when running the same browser from my development system and manually visiting https://w3c-test.org/webrtc/RTCDTMFSender-ontonechange-long.https.html, so I suspect there is something else I am missing. @agouaillard do you have any thoughts on this? Can you suggest a Firefox implementer that I could speak with on the topic?