Skip to content

jmdurant/RealityMixer

 
 

Repository files navigation

Reality Mixer
Mixed Reality app for iOS

Reality Mixer lets you use your iPhone or iPad to capture your favorite VR games and applications in Mixed Reality without a PC and without a green screen.

Click on this link to get it from the App Store, or follow the instructions below to install it using alternative methods.

Follow us on Twitter for more updates!

Features

  • Mixed Reality capture without green screen (using ARKit's person segmentation).
  • Avatars (using ARKit's body tracking) and Ready Player Me avatars. Example.
  • "AR" mode. Example.
  • Moving camera. Example.
  • Spectator camera. Example.
  • Mixed Reality with green screen. Example.

Examples

Game YouTube video Photos
Beat Saber Link
SUPERHOT Link
The Thrill of the Fight Link
Space Pirate Trainer Link

How to use it?

You can find instructions and tutorials on this page.

Requirements

  • Oculus Quest 1 or 2 with the Oculus MRC Calibration app version 1.7 installed.
  • Virtual Green Screen and Avatars: iPhone or iPad with an A12 chip or newer, running iOS 14. The LiDAR sensor is optional but recommended for better results.
  • Green Screen and Spectator: iPhone or iPad runnning iOS 14.
  • 5 Ghz WiFi network (802.11ac).
  • A compatible Quest VR application/game.

Compatible Applications/Games

The app is currently only compatible with the Oculus Quest 1 and 2, check this link for a list of the compatible games/applications.

Installation

App Store

Link

AltStore

  • Follow these instructions to install and configure AltServer on your PC or Mac, then install the AltStore app on your iPhone/iPad.

  • Open the camera app on your iPhone/iPad and scan this QR code:

download

OR

  • Open this URL on your iPhone/iPad:

altstore://install?url=https://github.com/fabio914/RealityMixer/releases/download/0.6.0/RealityMixer.ipa

Xcode

To build the app using your own developer account, create a DeveloperSettings.xcconfig config file inside RealityMixer/ with your development team identifier and your organization identifier:

DEVELOPMENT_TEAM = YOUR_TEAM_ID
ORGANIZATION_IDENTIFIER = com.yourcompany

TO-DOs

  • Investigate the audio delay;

  • Improve the calibration screen and fix the positioning of the controller models;

Wishlist

These are features that could be implemented in the future:

  • Create a custom calibration app for the Quest to replace the Oculus MRC app, allowing the user to calibrate from VR itself;

  • Allow the user to adjust the FOV of the mixed reality capture;

  • Allow the app to cast the video output (although this should already be possible if you long press the "Screen recording" button on "Control Center" and then select a streaming app);

  • Android version;

Credits

This project is based on the Oculus MRC plugin for OBS.

It uses SwiftSocket to handle the TCP connection with the Oculus Quest, FFMPEG with Apple's VideoToolbox to decode the stream, and ARKit for the "virtual green screen".

The right and left controller models are modified versions of the oculus-touch-v3 models from the WebXR Input Profiles.

This project also uses sounds from Notification Sounds and avatars from Ready Player Me.

About

Mixed Reality app for iOS

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • C 91.7%
  • Swift 7.6%
  • Other 0.7%