Experimental WebGL implementation

[nvanofwegen]

First and foremost, thanking @gtk2k for making the fundamental code which was used to make this contribution.

Summary

This PR adds experimental functionality for the WebGL platform, including:

• Data channels
• Receiving audio/video tracks
• Sending video with limited performance
• Sending audio not implemented yet
• RTCParameters and RTCStats are using complete JSON (de)serialization instead of pointers/marshalling, so the performance is low. For now, jillejr.newtonsoft.json-for-unity is required
• getUserMedia to send webcam/microphone over WebRTC (added to Samples)
• adapter.js to improve browser support

A demo can be accessed here: https://d1ddjbgn3ezoae.cloudfront.net

Known issues/limitations

Audio

• Audio in WebGL is limited. Currently the AudioStreamTracks are not sending any data over WebRTC, as OnAudioFilterRead is not available

Video

• Using Camera to generate VideoStreamTracks has limited performance and a lot of WebGL warnings appear
• Receiving video has a colorspace issue (rendering slightly too white)
• (Samples) Using the framebuffer causes the UICamera to crash and the UI turns black
• When receiving multiple VideoStreamTracks OnTrack simultaneously events, some tracks can remain grey and are not initialized properly

Firefox

• The RTCStatsReport generated by Firefox contains less information
• getCapabilities() is not implemented in Firefox, so setting the codec is not available.
• Firefox has a loopback error when the hostname is http://localhost:port (which is the hostname when you Build and Run and Unity hosts the files). However, the hostname http://localhost:port/path works. So serving the files with your own toolkit can be used as a workaround (eg jetbrains/vscode/apache/nginx)

Improvements/todo

• Find a way to generate and send audio sources. Alternatively, incoming audio could be implemented using the Web Audio API to send the audio data to Unity, or the DOM could be responsible for the audio and Unity can control the HTMLAudioElement
• Improve Camera video rendering performance
• Update RTCStats and RTCParameters to use pointers/marshalling
• Some RTCErrors are ignored
• Numerous TODO tags in the cs/jslib files
• The implementation is not fully tested, so expect unknown issues/bugs

[karasusan]

When pushing the Back button in DataChannel sample, we received the error below.

[karasusan]

In Stats sample, we got error below.

1. Push Call button.
2. Push Hangup button.

[nvanofwegen]

For audio: Not part of this PR, so errors are to be expected.

For data channels: as stated in this comment the 'error' is in this case, not an actual error.
Because the datachannel itself is already disposed, the datachannel that gets disconnected doesn't exist anymore in the reference table. It's basically a false error, which occurs by only destroying the context without cleanup first. However, I don't want to remove the comment, as the datachannel can be disconnected from the other side of the peer connection. It's simply prevented by hanging up first, then destroy the context.
Also, this scenario only occurs when you are connected to yourself, which is not a very likely scenario (but a valid scenario nonetheless)

For stats: I'll take a look, hangup should clean properly, so that needs to be fixed.

[mukaschultze]

There's a small inconsistency for data channels between the web and standalone. On the web the OnOpen callback is called for channels received via RTCPeerConnection.OnDataChannel, but on standalone it's not.

I suspect the web behaviour is the correct one, but I'm not sure.

[karasusan]

@mukaschultze
Yes, web browser behavior is correct, so It might be the bug.
I will be so glad if you make the issue page and write more detail of that.

[kannan-xiao4]

In MultiPeers sample,
On push Start button, console logging below error.

OverflowException: Arithmetic operation resulted in an overflow

after push Call button, this log continue to be output.

And push HangUp , error pop below

Uncaught RuntimeError: null function or function signature mismatch
    at build.wasm.gz:0x62735d
    at build.wasm.gz:0x165dea2
    at build.wasm.gz:0x147a5ab
    at build.wasm.gz:0xe6c456
    at invoke_vii (build.framework.js.gz:3425)
...
    at invoke_viiii (build.framework.js.gz:3425)
...
    at invoke_iiii (build.framework.js.gz:3425)
...
    at browserIterationFunc (build.framework.js.gz:3425)
    at callUserCallback (build.framework.js.gz:3425)
    at Object.runIter (build.framework.js.gz:3425)
    at Browser_mainLoop_runner (build.framework.js.gz:3425)

enviroment

• GoogleChrome on Windows10
• Build by Unity 2021.2

[kannan-xiao4]

In Bandwidth sample
On change parameter using dropdown menu, error happened.

but this error only occurred in app that build by Unity 2020.

[kannan-xiao4]

In PerfectNegotiation sample
The following error has occurred in the second (and from then) Swaps.

[jslib] SenderReplaceTrack: Unmanaged track. Ptr: 0

Sample works fine, but I think BAD_ACCESS happen.

[karasusan]

@nvanofwegen
Hi, how is the situation here?

[nvanofwegen]

Just got back from a long vacation, so I didn't have the time to look at it yet.

As for the stats issue and the issues @kannan-xiao4 mentioned, I will inspect those probably somewhere this week.

However, I have some things about the webgl implementation in general.

This PR is open for half a year now, and every couple of weeks things were added that had to be fixed. Meanwhile, the last rebase is also 5 months old, so I'm pretty sure there are again a lot of issues before this PR can be merged.
As this is done in my own free time, it's starting to feel a bit annoying that small issues are added every now and then and the PR stays open without a real plan and it feels more like a chore now.

So what is the roadmap for WebGL? For both this PR and the steps afterwards until it is ready to be merged in develop. The roadmap table in readme.md also hasn't been updated in a while.

I'm happy to look at the current issues found, but also patching the last 5 months of development where I'm the only one that can fix it is not something I'm looking forward to...

What is the downside of accepting the PR in the coming period, managing the issues here on github, work on the implementation and also let others be able to help?

[karasusan]

@nvanofwegen
I apologize for taking your free time. Our team are discussing about your reply.

FYI: I posted this thread for describing our direction.
https://forum.unity.com/threads/unity-render-streaming-status-update-q4-2021.1215801/

@gtk2k
@antoined73
@mukaschultze
If you have an opinion, I would like to hear it.

[nvanofwegen]

No need to apologize! I picked up the work from gtk2k myself! I actually had a lot of fun developing so far. I only think that my lack of a clear understanding of how to proceed took the fun out a bit.

About the forum thread, awesome, haven't seen it and already clarifies a bit!

With WebGL being a community-based development for the foreseeable future, I do think the proposed approach should be in place rather sooner than later, as this PR is far from a 'release'.

In the thread you mention that people can see the progress in this PR, but this PR has become a bit of a dragon (80+ replies). There are several decisions in this PR made about audio, marshalling, refactoring the api, etc, but there is no overview of that.

With those new managed issues (like refactoring, marshalling, audio, etc) discussions about the implementation could be better structured and organized and we could have a good overview in one main WebGL issue.

[mukaschultze]

I've been using mostly the DataChannels to implement a transport for https://github.com/vis2k/Mirror. From my tests it works as well as the native implementation.

With WebGL being a community-based development for the foreseeable future, I do think the proposed approach should be in place rather sooner than later, as this PR is far from a 'release'.

Totally agree with that

[gtk2k]

Thank you for your contribution to WebGL platform support over the long term to this day.

About the three points @nvanofwegen mentioned, audio, marshalling, and refactoring

audio

I think we are considering whether to implement this in the current PR. Since Microphone is not officially supported by Unity, it is possible to implement it by means such as using external assets, but I would like to avoid it as much as possible.

marshalling

Earlier, @karasusan drew a diagram and commented
#478 (comment)
I recognize that this is the main thing.
Currently, I've put together the Plugin API calls in WebRTC.cs (using a preprocessor), but I think it's easier to implement by creating a separate file for WebGL (eg WebRTC_WebGL.cs) and implementing a layer for WebGL there. increase.

refactoring

Refactor once all the tests pass with the current PR. After that, I think it would be better to consider audio and marshalling, but how about it?

If there is something I can help, I would like to help.

@karasusan
(日本語で失礼します)
私は、このリポジトリのフォルダ構成でUnityプロジェクトを開く方法がわかりません。WebRTC~フォルダにAssetsフォルダなどのUnityプロジェクトフォルダ構成となっており、このWebRTC~フォルダをUnityで開いてみたのですが、以下の画像のエラーが発生します。

package.jsonには相対アドレスが書かれていますが、これは本来Unity側で認識してくれて、実際のファイルを読み込んでくれるものなのでしょうか？

ちなみに Windows (11)です。

試しに、Windowsでのシンボリックリンクに置き換えてみたのですが(Runtime, Editor, TestsフォルダもWindowsシンボリックリンクに変更)、今度はプロジェクトを開くと以下の状態でずっとローディングのままとなってしまいます。(CPU使用率が上がりっぱなしで、一応何らかの処理は行われているようですが。。。)

どのようにすればプロジェクトを(Windows)Unityで開けるのでしょうか？

[karasusan]

Sorry for everyone but writing in Japanese is easier for me 🙂

@gtk2k
WebRTC~ 以下を Unity プロジェクトとして開くことができます。
git のチェックアウト時にシンボリックリンクが正しく反映されなかった可能性があります。以下の記事を確認してください。
https://isgs-lab.com/440/

指摘のあった package.json はこれのことだと思いますが、相対パスのように表示されているということは、git がシンボリックリンクをファイルとして扱っているということだと思います。

Windows11 が原因ではないとは思いますが、テスト環境が手元にないので確証はないです。

[gtk2k]

( Excuse me for continuing in Japanese. )
@karasusan
ありがとうございます。
Windowsの場合、git の config が複数存在し、また、アカウントを切り替えて使用しているため、Symlinks が false となっているのが原因で、これをtrueにすることでプロジェクトを開くことができました。

ですが、WebGLにプラットフォームを切り替えるときに Sampleの*.wavファイルで Import が進まなくなってしまいます。
現時点において、オーディオは実装されていないため、とりあえず*.wavファイルを削除することで回避してます。

[karasusan]

@nvanofwegen
@gtk2k
@antoined73
@mukaschultze

Hi, our team discussed about this topic.

• We are going to merge this pull request to the experimental feature branch experimental/webgl-platform in our repository soon.
• After merged that, please keep developing based on the feature branch.
• About known issues for WebGL support, we expect that you create issues on GitHub Issues, and add the link for the parent issue.

What our team do

• Maintain CI system (building and testing)
• Review pull requests from external developers
• Check issue tickets

If you have questions, please feel free to ask me.
I appreciate your efforts.