Sample project for using RICOH THETA HTC Vive with SteamVR for telepresence applications.
Display the 4K video stream from a THETA V inside a HTC Vive headset.
The RICOH THETA is the world's most popular 360 camera. The THETA V model was released in late 2017. The "V" outputs a 4K equirectangular video stream with H264 compression 3840×1920/29.97fps as well as a 2K 1920x960/29.97 stream.
You can easily use a Unity webcam texture to wrap the THETA V 4K video stream onto a sphere to provide a telepresence experience for industrial or experiential applications.
This test application illustrates the following techniques:
- identify THETA V camera inside of Unity
- assign THETA V to a texture
- use flip normals on a Blender sphere to display the webcam texture to the inside of the Sphere. The Blender sphere is imported into Unity
- use negative scale values to solve mirroring problem of objects in sphere appearing in reverse
- display THETA V stream inside of a HTC Vive headset with SteamVR
- track head movements and controller movements with SteamVR
- use THETA V as a microphone and output audio to HTC Vive
- initial configuration tests to orient the headset and the camera to simulate real-world telepresence
These techniques are compiled from the theta360.guide community in the Unity Development discussion conference. If you have any questions or would like to tell us about your project please join this free and independent community of developers. We'd love to hear from you.
Communications gear for the next gen! Join us.
Tests were done with the following:
- Unity 2017.4.0f1 (earlier version should work)
- RICOH THETA V
- RICOH THETA V firmware 2.11.1 (or higher)
- RICOH THETA V live streaming driver 1.0.1 64 bit - You NEED 1.0.1 or higher to work with Unity
Thanks to the great community of contributors at theta360.guide.
Special thanks to Ricoh for working with the independent development community and updating the THETA V live streaming driver to work with Unity.
This is based on the project by @zimmermegan for the THETA S.
Additional credits:
-
DrustZ for the initial hack of the THETA S registry that led to ZimmerMegan getting Unity to work with the THETA V. This led to Ricoh updating the driver.
-
joshapplman for the technique of using negative scale to reverse the mirroring effect inside of the sphere
-
Nima for the technique to use the THETA V microphones with Unity.
This is a skeleton project for
developers using Unity and is not intended for use as a binary.
If you want to check it out quickly, I
have included a binary in the directory release. If you don't want to download the entire repo, it may be easier for you to download zipped package from the releases section of GitHub. The release file is 15.2MB.
Extract the file. There should be a Windows 10 64 bit executable with a green THETA logo.
To simulate the perspective a standing person, put the THETA V on a tripod at eye level. Connect the THETA V with a USB cable and put it into live streaming mode.
The HTC Vive default position is pointed in the same direction as the rear THETA V camera lense. The rear lense is the one that is facing away from the shutter button. A person talking directly to the rear lense will appear to be talking to the face of the person wearing the HTC Vive headset.
Double-click on theta_v_htc_vive_test.exe on your Windows 10 computer.
The app should load in less than 5 seconds.
The screen will turn black as the headset adjusts to the THETA V input.
With the default SteamVR features, you can do the following:
- track both controllers inside of the headset. Position is accurate
- take screenshots from inside of the headset using the controller buttons
I am not using the controllers to grab anything right now.
The script will use the first audio device it finds. Adjust this line in the code to properly identify the THETA V. The script will display the connected microphones to the debug console of Unity.
public const int THETA_V_AUDIO_NUMBER = 0;
As of April 13, 2018, the THETA V will only stream a single audio channel. You can save 4 channel spatial audio to a file, but right now only one audio channel will stream to your headset.
You need 1.0.1 or later. Earlier versions will not work. See this article for info. Note how Ricoh updated the driver is response to our community feedback. :-)
I am using a flip-normals sphere that I created in blender. This should be included in the package.
If you're interested in building your own sphere with Blender, this video will give you a step-by-step process.
The flip-normals sphere causing the scene to appear like a mirror image. To correct this problem, I am inverting the sphere with negative scale. -8, -8, -8. Original hack by @joshapplman
I needed to rotate the X axis of the SteamVR rig by 180 degrees in order to get it to work.
The script should automatically detect the webcams on your system. By default, the script
will look for RICOH THETA V 4K. If you want to use a different resolution or a different
driver, you will need to modify the script.
The script will also display all webcams attached to your computer for debugging.
Put the bottom of the THETA V at eye level for a telepresence experience that is close the real world. If the camera is at eye level, the person you are talking to should be face the lense that is opposite the shutter button.
Alternately, you can change the perspective to a surreal experience. Several people wearing the headset have mentioned that they like it when the THETA V is close to the ground as they feel like a small animal.
The cool thing about developing for the VR headset is that you are forced to stand away from your computer to test it. You may even need your friend to help you with the headset. :-)
Thanks to Ricoh for these cool THETA V t-shirts they gave out at CES.
