MediaPipe_for_OBS

An Electron App to send MediaPipe data to OBS. Landmark positions from the MediaPipe pose model are sent to OBS Browser Source and the Advanced Scene Switcher Plugin.

How to set up MediaPipe for OBS

https://youtu.be/S-IjFVj6zyE

graph LR;
subgraph OBS
  direction LR
  A(Video Source)-->B(Video Projector Window);
  D[Advanced Scene Switcher];
  E[Browser]
end

B(Projector Window)--> C(MediaPipe for OBS);

C ---> |WebSocket|E;
C ---> |WebSocket|D;

OBS Setup

MediaPipe for OBS app requirements

1. Enable the OBS Web Socket Server ,
2. an OBS Video source with a projector window
3. A Browser or Advanced Scene Switcher to receive the landmark data

Turn on OBS WebSocket Server

• In OBS menu bar, click Tools --> WebSocket Server Settings
• Check "Enable WebSocket server"
• Press the "Show Connect Info" button.
  • Copy the Server Password

Create a Video Capture Device Source

• Create a "Video Capture Device" source to bring a camera into OBS. In the Video Capture Device Properties set the "Resolution Type" to Custom

• Open a Windowed Projector by Right clicking (control+click for Mac) on the Video Capture Device source.
  • click Windowed Projector
    

Create a Browser Source

Load a local html file with an event listener. The landmarks will be sent as an evnt named "pose-landmarks".
See the creative example folder for example HTML pages.

Each Landmark has an X, Y, Z and Visability value.

Messages are also sent the Advanced Scene Switcher Plugin with the values as the message

Using MediaPipe for OBS

Download the latest

release

**MacOS note: Before using the MediaPipe for OBS, Add permissions in Settings-->Privacy & Security-->Screen Recording & System Audio.

• Enter the OBS WebSocket details, and press the "Connect to OBS" button

• Choose a projector window

• press the "Start MediaPipe" Button

A log file of the landmarks is saved to the folder "~/Library/Logs/MediaPipe_for_OBS/"

Dev install

Steps to setup a developement environment.

npm install electron --save-dev
npm install --save-dev @electron-forge/cli
npm install --save-dev @electron-forge/plugin-fuses

npm run start

npx electron-forge import

npm run make

OBS Connections

MediaPipe for OBS is a part of a meeting and presentation automation project.

flowchart LR
PTZ[PTZ \nCamera Data]
GC[Game 🎮\n Controllers]
MP["MediaPipe\nfor OBS"]
GAS["Google \n Apps \n Scripts"]
GS["Google \n 🗃️"]
Midi["Midi 🎹 \n Devices"]
ZOSC[ZoomOSC]
spw["Source\nProjector\nWindow"]

PTZ-->MP
GC-->MP
Midi--"midi"-->MP
spw-->MP
ZOSC<--"osc"-->MP
MP <--"ws"-->wss
obs-b<--"js"-->GAS
GAS<--gs-->GS
obs-b<--"webRTC"-->MP

subgraph OBS[OBS]
    direction LR
    obs-b[browser]
    wss["WebSocket \nServer"]

    obs-b<--"ws"-->wss
    
    
end

linkStyle default stroke-width:4px,fill:none,stroke:green;
linkStyle 0,1,2,3 stroke-width:4px,fill:none,stroke:blue;