Skip to content

Conversation

@fippo
Copy link
Collaborator

@fippo fippo commented Oct 11, 2022

Description

adds interoperability tests between Chrome unstable and Firefox Nightly. These tests are designed to run nightly as github actions. Github provides a "free" budget of 2000 minutes per month. In the current configuration with four tests (a matrix of Chrome/Firefox) that each take around a minute the nightly runs have a "cost" of 120 minutes per month.

The tests do not rely on a signaling server, instead use the mocha-based test that is controlling the individual selenium webdriver instances to act as a signaling channel which exchanges offers and answers with candidates.
This pattern is described in the testbed repository
https://github.com/fippo/testbed
from 2016, this is a "more modern" take on the same subject.

To run the tests locally,

  • npm run mocha test/interop/connection

Running them with Chrome/Firefox unstable requires Linux for the integration with travis-multirunner and setting the

  • BROWSER_A
  • BROWSER_B
  • BVER environment variables.

Purpose
Getting rid of the dependency on apprtc in tests while keeping a minimum level of interoperability tests

adds interoperability tests between Chrome unstable and Firefox
Nightly. These tests are designed to run nightly as github actions.

They do not rely on a signaling server, instead use the mocha-based
test that is controlling the individual selenium webdriver instances
to act as a signaling channel which exchanges offers and answers with
candidates.
This pattern is described in the testbed repository
  https://github.com/fippo/testbed
from 2016, this is a "more modern" take on the same subject.

To run the tests locally,
* npm install --no-save chromedriver geckodriver
* npm run mocha test/interop/connection
@fippo
Copy link
Collaborator Author

fippo commented Oct 11, 2022

I've run those tests for ~2 weeks now and they were surprisingly reliable (unlike the samples tests...)
Adding Safari to the mix is possible (but that has a minute multiplier of 10) too but I'd rather run the basic case that we want to replace apprtc first for a month or two before getting more ambitious.

One of the tricky things is that chromedriver normally has a version check which allow it to operate +- 1 release of the chrome version. This broke when Chrome Unstable went to M108 and required adding the disable-build-check flag.
It might actually be useful to extend the chromedriver flag to specify a range of 2 as an argument instead of completly disabling the check

Copy link
Contributor

@KaptenJansson KaptenJansson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall but with a disclaimer: it's been a while since I worked on this but I trust you that it works.

NIT: Please add copyright headers on each file.

Copy link
Contributor

@alvestrand alvestrand left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do test failures show up?

@fippo
Copy link
Collaborator Author

fippo commented Oct 19, 2022

They will show up on https://github.com/webrtc/samples/actions which can be filtered.
I plan to add a "send email" functionality once running this for a month shows it is solid enough that a failure justifies nagging people ;-)

@fippo fippo merged commit 7a68420 into webrtc:gh-pages Oct 19, 2022
@fippo fippo deleted the interop-tests branch October 19, 2022 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants