Unofficial library that handles the communication with the Quest 2 for mixed reality and provides an easy bridge to OpenCV.
Nearly ready for a first release. Already usable but the interface may change a little bit in future versions.
A graphical interface is available here.
The master branch is hosted at https://github.com/RandomPrototypes/libQuestMR
Already implemented :
- Communication with Oculus Mixed Reality Capture App (Headset and controller poses, trigger button, calibration load and upload)
- Calibration between external camera and headset
- Extraction and decoding of the frames from the headset
- Recording and playback of the frames from the headset
- Windows, Linux, Macos support
- Video matting without greenscreen
- Audio recording
Future plan :
- Moving camera
- Body tracking
- Android, iphone, maybe raspberry pi support (the code is portable but the makefiles probably need to be modified)
Networking is managed by BufferedSocket, a lighweight portable library I made.
XML loading/writing is managed by tinyxml2. It's automatically downloaded by cmake.
Video encoding/decoding is managed by FFMPEG. You need to install it and let cmake find it.
OpenCV is a recommended dependency. Some parts of the project can work without it but many functions including the calibration and visualization would be disabled if compiled without OpenCV.
RPCameraInterface is a recommended but optional dependency. It's a portable camera library I made to support some functions that were not available in OpenCV : camera enumeration, camera formats and resolution enumeration, MJPEG streaming, network camera (android phone with custom app,...).
onnx-runtime is a recommended but optional dependency. It's used to run the deep learning method for background subtraction without greenscreen.
To use the deep learning background subtraction method, you need to download rvm_mobilenetv3_fp32.onnx from RobustVideoMatting project and specify the folder with the command setBackgroundSubtractorResourceFolder().
Please check the license of this file before using it in your project.
A part of the code is based on the official OBS plugin for Quest 2.
Most of the rest of the code is based on wireshark captures and some reading of the code of RealityMixer