The series of tutorial apps for WebRTC using Angular (Typescript), Android (Java), SignalR (C#) and NodeJS (Typescript).
The tutorial apps include:
- basic peer-to-peer implementation (for web and mobile apps),
- multi-peer implementation (for web apps) via all four of the most known WebRTC topologies - MESH, STAR, SFU (Selective Forwarding Unit) and MCU (Multipoint Control Unit).
The signaling server is implemented using Microsoft SignalR. The media (streaming) server is implemented using NodeJS.
The audio/video track(s) mixing for MCU (Multipoint Control Unit) topology is implemented fully custom in NodeJS, using node-webrtc and its Nonstandard APIs. The implementation can be seen in the MediaStreamMixer class.
The face recognition (anonymization) in Android app is implemented using Google's MediaPipe algorithms. The implementation can be seen in the FaceAnonymizer class. To test it out, just set the "USE_FACE_ANONYMIZATION" flag to "true" in AndroidCameraCapturer class.
The WebRTC Android app now has full support for Android-based Augmented Reality smartglass devices (such as Vuzix M400, Google Glass EE2, RealWear Navigator 500, Almer Arc2 and similar). The implementation can be seen in WebRTCAndroidApp. To test it out, just set the "USE_SMARTGLASS_OPTIMIZATION" flag to "true" in WebRTCAndroidApp class.
The WebRTC web app has the full integration with OpenAI's Realtime API using .NET WebAPI as backend (to fetch the remote SDP).
This integration features:
- real-time audio chat with OpenAI agent
- agent's function calling in code (e.g. changing page title or background color)
- live transcription of agent's messages
To test it out, just select "Chat with OpenAI" option on the web app's home screen.
Note: For integration to work, OpenAI's API key is needed with some credits available on the account (since Realtime API isn't free).
Find more details on the implementation itself on my LinkedIn.
This repository uses MIT license. Copyright (c) Dalibor Kofjač.