The current state of interoperability testing for WebRTC is still mostly as described by two blog posts written by Google’s test engineer, Patrik Höglund in mid2014 here and here.
Basically, the testing process is using the apprtc sample application in a hermetic environment on Linux to test interoperability with Firefox as part of the Chrome release process.
Other notable examples include the work done by NTTs Yoshimasa Iwase who is running full factorial tests including different NAT configurations (described here).
Yet, some breakages happen which are not detected by the testing process mentioned above. For example, an upgrade to Chrome's DTLS library broke the interoperability with the Jitsi Videobridge in January 2015. This was noticed only very shortly before rolling out to all Chrome users.
Similar issues happened when Mozilla Firefox started to require Perfect Forward Secrecy for DTLS without announcing this change widely enough. This broke interoperability for several mobile applications based on older versions of the webrtc.org library, including Facebook Messenger which forced Mozilla to postpone the upgrade for several weeks.
More recently, SRTP between Firefox 49 and Edge was broken (in one direction) which went unnoticed for two weeks after Firefox 49 rolled out. And Video interop between Chrome Canary (M56) and Edge was broken in one direction as well due to stricter validation of H264 profile level ids.
Most of these issues have been noticed by people testing manually.
Support for this work was provided in part by the International Multimedia Telecommunications Consortium (IMTC) - http://www.imtc.org Many thanks to Bernard Aboba for making me start this!
The testing process is based on the process used in adapter.js and the samples. It uses selenium and webdriverjs and tests are written using tape.
H264 tests currently require Chrome 50 which adds H264 behind a flag.
The tests for Microsoft Edge only run on Windows currently. The video tests require either the insider version of Edge (which has H264 enabled by default or the TH2 release of Windows where H264 is available behind a flag).
Firefox uses a binary module from the OpenH264 project to provide H264 support. Typically, this module is downloaded by Firefox shortly after the creation of a new profile. Since Selenium creates a new profile for each test, the binary needs to be provided in a template profile. See this README for details.
VP9 can be enabled with a flag in Firefox. However, this is compatbile with Chrome only in Firefox 45+.
apprtc.js shows how to test the AppRTC example provided by Google. It uses a number of URL parameters to tweak application behaviour, e.g. to force the VP9 or H264 video codec. Both ICE connectivity as well as video interoperability is tested. For the latter, the frame checker from testRTC is used.
There is a second set of tests which use plain HTML pages and adapter.js. Those tests emulate the PeerConnection API to some degree which makes them look very similar to some of the tests in adapter.js or the JSFiddles written by Mozilla's Jan-Ivar.
Tests currently include
- audio interoperability tests working in Chrome, Microsoft Edge and Firefox
- video interoperability tests in Chrome and Firefox, using VP8, VP9 and H264.
- upgrade tests which upgrade an audio-only call to an audio-video call.