Skip to content
Chat Heads is a native sample that uses RealSense to overlay background segmented (BGS) player images on a 3D scene or video playback in a multiplayer scenario.
C++ C HLSL GLSL Objective-C Fortran Other
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

###Chat Heads Intel RealSense Technology can be used to improve the e-sports experience for both players and spectators by allowing them to see each other on-screen during the game. Using background segmented video (BGS), players’ “floating heads” can be overlaid on top of a game using less screen real-estate than full widescreen video and graphics can be displayed behind them.

Chat Heads is a sample that will help you understand the various pieces in the implementation (using the Intel RealSense SDK for background segmentation, networking, video compression and decompression), the social interaction, and the performance of this RealSense use case. The code in this sample is written in C++ and uses DirectX*.

For more info, please checkout

####Library dependencies:

  • Realsense SDK
  • Raknet for networking
  • Microsoft Windows Media Foundation (WMF) framework for encoding/decoding
  • Intel Instrumentation and Tracing Technology (ITT) for performance analysis
  • Libtheoraplayer/libtheora/libvorbis/libogg and OpenAL for converting video into texture data and for ogg playback
  • IMGui for UI

####Building the sample: Install the [RealSense SDK R5 or greater] ( prior to building the sample

####Running the sample: Install the OpenAL runtime and RealSense DCM (depth camera manager) before running the sample. There are no command line arguments. Just start the .exe and the UI should be self-explanatory.

The sample has been tested on Windows 8.1 and 10 with both embedded and external (usb) RealSense cameras.

####Project structure: *ChatheadsNativePOC* contains windowsMain.cpp, the sample (ChatHeads.cpp/h), RealSense wrapper (RealsenseMgr.cpp/h) and Raknet wrapper (NetworkLayer.cpp/h)

*CPUTDX* houses our in-house 3D game engine framework

*VideoStreaming* has the encoder and decoder logic, and uses Microsoft's WMF

*Imgui* is the UI implementation

*Media* contains the video clips, shader and other art assets

*TheoraPlayer* contains the header and library files for LibTheoraPlayer and associated libraries

*Raknet* contains the header and library files for Raknet

*itt* contains the header and library files for libittnotify

####Code browsing pointers: Application code is in ChatHeads.h/cpp

void RenderChatheads(CPUTRenderParameters& renderParams);
void UpdateRemoteChatheadBuffer(NetMsgVideoUpdate *pMsg);

RealSense stuff is in RealsenseMgr.h/cpp; Important functions include:

bool PreInit();
bool Init();
 	void Shutdown();
DWORD RSThreadUpdate();

Network wrapper over Raknet is in NetworkLayer.h/cpp

static DWORD WINAPI NetworkThread(LPVOID lpParam);
bool SendVideoData(NetMsgVideoUpdate& msg, bool broadcast);
static void ProcessVideoUpdateMsg(NetworkLayer *pNet, RakNet::Packet *pPacket);

Media code is in EncodeTransform.h/cpp and DecodeTransform.h/cpp

####Feedback For bugs reports, fixes & feedback, please send a pull request (or) email to

You can’t perform that action at this time.