Any plans to contribute stuff back to Unity's WebRTC implementation?

[StephenHodgson]

https://github.com/Unity-Technologies/com.unity.webrtc

What your guys thoughts on this?

Any plans to contribute stuff back to Unity and roll this into a single package?

Sibling Issue:
Unity-Technologies/com.unity.webrtc#24

[djee-ms]

Hi @StephenHodgson,

We are having a look at the Unity project and discussing internally about it, and I will come back to you when we have more information.

[djee-ms]

Here's an answer. Apology for the delay, as we were discussing with other teams to align.
Ping @StephenHodgson and @chrisse27 for visibility.

---

Unity recently released com.unity.webrtc, a package to use WebRTC in Unity. The question of interoperability with MixedReality-WebRTC has rightfully came up several times. We have not yet engaged with Unity on possible integration. Here are the current differences making integration difficult at this time given our current knowledge about the Unity project.

Features

Parity of core features

First, the Unity project is new and missing support for features that Mixed Reality developers are currently getting with MixedReality-WebRTC.

	MixedReality-WebRTC	com.unity.webrtc
Windows Desktop	x86, x64	x64
Windows UWP	x86, x64, ARM	❌
IL2CPP	✔	❌
Hardware encoding/decoding	UWP only	NVIDIA GPUs only
Unity	2018.3+	2019.1+

Some of the items missing in com.unity.webrtc are highlighted in the roadmap of the Unity project, but in the absence of a timeline associated with that roadmap it is unclear when we could get parity with the current codebase if we were to switch to the Unity implementation of WebRTC. As of today, switching to that project would cause major regressions in MixedReality-WebRTC.

Mixed Reality features

Second, MixedReality-WebRTC provides support for features specific to Mixed Reality, like Mixed Reality Capture (MRC) allowing developers to render holograms on top of the head-mounted HoloLens camera, or video profile selection allowing developers to select camera resolutions better suited for video conferencing (lower resolution, less CPU consuming, better thermals). Those features are not currently available on the source repository maintained by Google, which MixedReality-WebRTC indirectly builds upon.

Runtime Interoperability

WebRTC is standardized, and any compliant implementation shall be interoperable with any other one, provided a common signaling solution is employed.

MixedReality-WebRTC currently builds upon the M71 milestone of Google's WebRTC implementation, which is not strictly compliant with WebRTC 1.0 yet. However we have plans to move to a later milestone compliant with the 1.0 version of the standard. The com.unity.webrtc package also uses the Google implementation, but at its M72 milestone, which is also not strictly compliant. As we understand, the first compliant milestone is M73. That being said, the fact that both projects build upon the same codebase reduces the risk of incompatibility. Therefore we expect runtime interoperability for the two projects.

The signaling solution is as always outside of the scope of the WebRTC standard, and MixedReality-WebRTC offers an extensible mechanism supporting any protocol/implementation.

Contributions

Sometimes we require modifications to parts of the core WebRTC implementation from Google. Although we aim to keep those modifications as small as possible to stay close to the Google codebase, some are still needed, in particular to support Mixed Reality features. This is the reason why MixedReality-WebRTC builds upon the WebRTC UWP project, which in turn consumes the Google repository.

With the Unity project we do not have access to the core implementation Unity is using (the native plugin appears to be a DLL checked in inside the repository, without its sources). As the Unity project seems focused on render streaming, we assume that Unity also needs access to the WebRTC core for its own customizations. As a first step towards collaboration, Unity would need to make their fork public. But beyond that, having separate repositories renders collaboration very difficult. Moving to a shared repository is also problematic, with all the administrative and legal issues that would surround such a cross-company collaboration. For example, NuGet packages for MixedReality-WebRTC are signed with a Microsoft certificate, which mandates restricted push access to the repository to Microsoft employees only for obvious security reasons.

As both libraries use the Google implementation of WebRTC under the hood, a longer term path forward to unify them could be to push all the necessary changes on each side into the Google repository, provided this is something that the maintainers of that repository agree on. Although we have no concrete information about any engagement on Google's side at the current time, the WebRTC UWP team and the MixedReality-WebRTC are determined to follow that path, and we are pleased to announce that we are taking steps to evaluate rebasing both of those Microsoft projects on the HEAD of the Google repository, and try to contribute our changes back to it, while being able to benefit from the latest features introduced by the Google's WebRTC team since the M71 milestone. We hope to see a path where Unity could do the same, which would provide a natural way to join forces and ensure runtime interoperability.

[djee-ms]

We will update this question if there is any news in the future, but closing it in the meantime as I believe the question was answered.