Introduction

Infobip RTC extensions is a JavaScript library which provides extended functionality to Infobip RTC SDK.

Currently available functionalities are:

• audio filter implementations
• video filter implementations

Here you will find an overview, and a quick guide on how to include and use these extensions in your application. There is also in-depth reference documentation available.

Prerequisites

Infobip RTC Extensions requires ES6.

Getting the library

There are a few ways in which you can get our library. We publish it as an NPM package and as a standalone JS file hosted on a CDN.

If you want to add it as an NPM dependency, run the following:

npm install infobip-rtc-extensions --save

After which you would use it in your project like this:

let infobipRtcExtensions = require('infobip-rtc-extensions');

or as ES6 import:

import {VirtualBackgroundVideoFilter} from "infobip-rtc-extensions"

You can include our distribution file in your JavaScript from our CDN:

<script src="//rtc.cdn.infobip.com/0.0.10/infobip.rtc.extensions.js"></script>

The latest tag is also available:

<script src="//rtc.cdn.infobip.com/latest/infobip.rtc.extensions.js"></script>

Audio filters

The Infobip RTC supports user-defined audio filters which manipulate outgoing audio streams during a call. This library implements several audio filters which are easy to configure and use.

Filters which are currently available are:

• BackgroundMusicAudioFilter
• NoiseSuppressionFilter

BackgroundMusicAudioFilter

BackgroundMusicAudioFilter allows the user to specify audio to be played alongside their outgoing stream. This audio is heard by other participants of the call, but not by the user.

To use this filter, an instance of BackgroundMusicAudioFilter needs to be created. The only parameter of the class constructor is a string containing the URL of the audio file which is to be combined with the user's outgoing stream. The audio file loops until the filter is turned off or the call is terminated.

const musicURL = "path/to/desired/audio.mp3";
const backgroundMusicFilter = new BackgroundMusicAudioFilter(musicURL);

NoiseSuppressionFilter

The NoiseSuppressionFilter enhances speech by removing several types of background noise. This filter works in real-time. Currently, it is focused on removing background noises commonly encountered in call centers, such as babble, noise produced by different devices (e.g. air conditioner) and keyboard typing sounds. However, it performs well on a wider range of noise types. It is also independent of the language spoken.

Since noise suppression uses a neural network, a deep learning model needs to be loaded into the filter. The model consists of two files:

• model.json - a JSON which contains metadata describing the model
• group1-shard1of1.bin - a binary file containing model data

To use the noise suppression filter, an instance of the class NoisesuppressionFilter needs to be created. Its constructor accepts a single parameter - a string containing a URL of the model.json file. The binary file is then automatically loaded, which is possible only if it is available at the same location as the JSON.

Model files are available at our CDN:

• model.json at https://d217hhmiwkst9z.cloudfront.net/noise-suppression-tfjs-model/latest/model.json
• group1-shard1of1.bin at https://d217hhmiwkst9z.cloudfront.net/noise-suppression-tfjs-model/latest/group1-shard1of1.bin

If you wish to use our CDN directly, it suffices to pass the first URL to the constructor. Another possibility is to download both files and host them. When doing so, both files must be available at the same path (see example code).

Real-time noise suppression is a hardware-intensive process, so not all hardware will be able to sustain it. In case of insufficient performance, the filter is automatically disabled in order to not interfere with the call.

const modelURL = "https://www.example-cdn.com/noise-suppression-model/model.json";
// group1-shard1of1.bin must be available at https://www.example-cdn.com/noise-suppression-model/group1-shard1of1.bin
const noiseSupressionFilter = new NoiseSuppressionFilter(modelURL);

Video filters

The Infobip RTC supports user-defined video filters capable of manipulating outgoing video streams during calls. The library provides an extensive implementation of commonly used video filters, making configuration easier and enabling seamless integration.

Currently available implementations are:

• VirtualBackgroundVideoFilter

VirtualBackgroundVideoFilter

This filter allows users to modify their background during video calls.

Supported video filter modes include:

• Virtual background (VideoFilterMode.VIRTUAL_BACKGROUND) - Users can set a custom image to be displayed as their background
• Background blur (VideoFilterMode.BACKGROUND_BLUR) - Users can blur their background.
• Face track (VideoFilterMode.FACE_TRACK) - Automatically adjusts the video to keep the user's face centered and properly framed within the view.
• None (VideoFilterMode.NONE) - No video filtering is applied; video frames are passed through unchanged. This option is recommended over repeatedly reallocating video filter resources to avoid visible disruptions.

To utilize this feature, begin by creating an instance of the VirtualBackgroundVideoFilter object. The constructor accepts optional VirtualBackgroundVideoFilterOptions for customization.

const options = {
    mode: VideoFilterMode.VIRTUAL_BACKGROUND,
    image: sourceImage // can be an instance of ImageBitmap, ImageData, HTMLImageElement, …
};
const videoFilter = new VirtualBackgroundVideoFilter(options);

For optimal performance, it's recommended to avoid reallocating video filter instances solely for mode changes. Instead, pass the new options directly to the existing video filter instance. This approach minimizes resource overhead and enhances overall efficiency.

const options = {
    mode: VideoFilterMode.NONE
};
await videoFilter.setOptions(options);

Applying the video filter

Once you've created the video filter, you can utilize it during calls.

You can set it beforehand when initiating a new ApplicationCall using VideoOptions object within the ApplicationCallOptions object:

let token = obtainToken();
let infobipRTC = new InfobipRTC(token);
infobipRTC.connect();

let videoOptions = VideoOptions.builder().setVideoFilterFactory(() => videoFilter).build();
let applicationCallOptions = ApplicationCallOptions.builder().setVideo(true).setVideoOptions(videoOptions).build();
let applicationCall = infobipRTC.callApplication('45g2gql9ay4a2blu55uk1628', applicationCallOptions);

Alternatively, you can apply the filter to the existing ApplicationCall using the setVideoFilter method:

await applicationCall.setVideoFilter(videoFilter);