|
WebXR is now possible in Unity thanks to the SimpleWebXR library, which contains an example with the MRTK! So, this repo is deprecated, please have a look at this one: https://github.com/Rufus31415/Simple-WebXR-Unity |
|---|
This repository is a fork of Microsoft MRTK. It adds to the MRTK the WebXR target allowing to do augmented reality in the web browser. This project is a proof of concept and not an industrial implementation ! I developed this demonstration over a long weekend in confinement.
Live demo (please use WebXR Viewer browser app on iOS) : https://rufus31415.github.io/sandbox/webxr-hand-interaction/
Video YouTube: https://www.youtube.com/watch?v=msORgnO6R9U&cc_load_policy=1
Blog post : https://rufus31415.github.io/webxr-webgl-demo-app-built-with-unity-and-mrtk.html
Hacker News post:
 |
 |
 |
 |
 |
 |
MRTK-Unity is a Microsoft-driven project that provides a set of components and features, used to accelerate cross-platform MR app development in Unity. Here are some of its functions:
- Provides the basic building blocks for Unity development on HoloLens, Windows Mixed Reality, and OpenVR.
- Enables rapid prototyping via in-editor simulation that allows you to see changes immediately.
- Operates as an extensible framework that provides developers the ability to swap out core components.
- Supports a wide range of platforms, including
- Microsoft HoloLens
- Microsoft HoloLens 2
- Windows Mixed Reality headsets
- OpenVR headsets (HTC Vive / Oculus Rift) read more...
The WebXR Device API provides access to input and output capabilities commonly associated with Virtual Reality (VR) and Augmented Reality (AR) devices. It allows you develop and host VR and AR experiences on the web. Today (May 2020), few browsers are currently WebXR compatible. read more...
Unity is a 3D rendering engine and development environment to create video games, but also 3D experiences of augmented or virtual reality. It provides applications for Windows, MacOS, Linux, Playstation, Xbox, Hololens, WebGL...
WebGL is here the export format used.
WebGL allows to dynamically display, create and manage complex 3D graphical elements in a client's web browser. Most desktop and mobile browsers are now compatible with WebGL.
MRTK is developed in Unity. I used here the example scene Hand Interaction. I compiled it into WebGL with Unity, so I got javascript (as WebAssembly). Since Safari is not yet compatible with WebXR under iOS (I don't have an Android phone), I had to use the XRViewer browser developed by Mozilla. However, the implementation of WebXR is not standard because it was a test application. (read more about XRViewer app.... I was inspired by the examples provided by Mozilla. I added the generated WebGL component on top of it. I use the WebXR javascript API to transmit the position of the smartphone to the Unity Main Camera.
Moreover, according to the Mozilla examples, I use Three.js to display the world mapping mesh (in green).
A new Input System Profile has been created to support multitouch as well as the mouse pointer. See MyMixedRealityInputSystemProfile.asset and MyMixedRealityHandTrackingProfile.asset.
The WebGL canvas is placed on top of the video canvas. The black background normally generated by Unity must be made transparent in order to view the video behind the holograms.
For this, a new Camera Profile has been created with a solid color (black background) and 100% transparency.
In addition, a jslib plugin has been written to apply this transparency which is not supported by default in WebGL.
var LibraryGLClear = {
glClear: function(mask) {
if (mask == 0x00004000) {
var v = GLctx.getParameter(GLctx.COLOR_WRITEMASK);
if (!v[0] && !v[1] && !v[2] && v[3]) return;
}
GLctx.clear(mask);
}
};
mergeInto(LibraryManager.library, LibraryGLClear);Finally, after generation, the Build/build.json file must be modified to replace "backgroundColor": "#000000" by "backgroundColor": "transparent". This operation could be done automatically in post-build.
A WebGL export template is used to generate the web page to get the final application. This template was inspired by the World Sensing example from Mozilla. It manages the display of the video, the start of the WebXR session and the display in green of the mesh from the world scanning. I added on the video the WebGL canvas generated by Unity.
The position of the smartphone is in a Camera Three.js object. It is recovered in loop and transmitted to the Unity camera with the following code:
var pose = new THREE.Vector3();
var quaternion = new THREE.Quaternion();
var scale = new THREE.Vector3();
engine.camera.matrix.decompose(pose, quaternion, scale);
var coef = 1/0.4; // Three.js to Unity length scale
var msg = {x: pose.x * coef, y: pose.y * coef, z: -pose.z * coef, _x: quaternion._x, _y: quaternion._y, _z: quaternion._z, _w: quaternion._w, crx: -1, cry: -1, crz: 1, fov:40};
var msgStr = JSON.stringify(msg);
unityInstance.SendMessage("JSConnector", "setCameraTransform", msgStr); // send message to Unity (only 1 argument possible)public void setCameraTransform(string msgStr) {
var msg = JsonUtility.FromJson<JSCameraTransformMessage>(msgStr);
Camera.main.transform.position = new Vector3(msg.x, msg.y, msg.z);
var quat = new Quaternion(msg._x, msg._y, msg._z, msg._w);
var euler = quat.eulerAngles;
Camera.main.transform.rotation = Quaternion.Euler(msg.crx * euler.x, msg.cry * euler.y, msg.crz * euler.z);
Camera.main.fieldOfView = msg.fov;
}Some settings, such as scale change and FOV are hard written and adapted to my iPhone 7 for this POC.
To be more efficient, the access to the WebXR API should be done in a jslib plugin so that the code is integrated to the WebGL build, this would avoid the heaviness of Three.js, the lags of the "sendMessage" function, and the decoding/encoding of json in string. Also, it would be necessary to think about how to transmit the good parameters of FOV and scaling to Unity. The current code is clearly inefficient, but it is a quick working proof of concept.
WebGL compilation is performed by Unity using the IL2CPP scripting backend then the Emscripten WebAssembly compiler. Note that the ARSessionOrigin.cs file in the AR Foundation package was voluntarily committed because it includes modifications to be compilable in WebGL.
As mentioned above, after compilation, the Builb/build.json file must be modified for supporting transparency.
Finally, to avoid warnings at runtime about browser incompatibility, the code generated in UnityLoader.js should be modified by replacing: UnityLoader.SystemInfo.mobile by false and ["Edge", "Firefox", "Chrome", "Safari"].indexOf(UnityLoader.SystemInfo.browser)==-1 by false.
To date (May 2020), the XRViewer application is the only browser that offers a WebXR implementation on iOS (Safari is not yet compatible). But this implementation is a draft as explained here. It would not be complicated to make this example compatible with Chrome for Android, or Chrome or Firefox for desktop. This would allow to generate Unity scenes compatible with Android and virtual reality devices like WMR, Oculus, OpenVR, or Vive. Maybe I'll work on it if needed 😀.
If you are using a browser other than XRViewer, you will get an error popup indicating that WebXR is not supported. But you can then enjoy the WebGL scene with the mouse.
With another browser :
With XRViewer :
 Getting Started |
 MRTK Overview |
 Feature Guides |
 API Reference |
|---|
| Branch | CI Status | Docs Status |
|---|---|---|
mrtk_development |
 Windows SDK 18362+ |
 Unity 2018.4.x |
 Visual Studio 2019 |
 Emulators (optional) |
|---|---|---|---|
| To build apps with MRTK v2, you need the Windows 10 May 2019 Update SDK. To run apps for immersive headsets, you need the Windows 10 Fall Creators Update. |
The Unity 3D engine provides support for building mixed reality projects in Windows 10 | Visual Studio is used for code editing, deploying and building UWP app packages | The Emulators allow you to test your app without the device in a simulated environment |
 Button |
 Bounding Box |
 Manipulation Handler |
|---|---|---|
| A button control which supports various input methods, including HoloLens 2's articulated hand | Standard UI for manipulating objects in 3D space | Script for manipulating objects with one or two hands |
 Slate |
 System Keyboard |
 Interactable |
| 2D style plane which supports scrolling with articulated hand input | Example script of using the system keyboard in Unity | A script for making objects interactable with visual states and theme support |
 Solver |
 Object Collection |
 Tooltip |
| Various object positioning behaviors such as tag-along, body-lock, constant view size and surface magnetism | Script for laying out an array of objects in a three-dimensional shape | Annotation UI with a flexible anchor/pivot system, which can be used for labeling motion controllers and objects |
 Slider |
 MRTK Standard Shader |
 Hand Menu |
| Slider UI for adjusting values supporting direct hand tracking interaction | MRTK's Standard shader supports various Fluent design elements with performance | Hand-locked UI for quick access, using the Hand Constraint Solver |
 App Bar |
 Pointers |
 Fingertip Visualization |
| UI for Bounding Box's manual activation | Learn about various types of pointers | Visual affordance on the fingertip which improves the confidence for the direct interaction |
 Near Menu |
 Spatial Awareness |
 Voice Command |
| Demonstrates how to use Solver to attach objects to the hand joints | Make your holographic objects interact with the physical environments | Scripts and examples for integrating speech input |
 Eye Tracking: Target Selection |
 Eye Tracking: Navigation |
 Eye Tracking: Heat Map |
| Combine eyes, voice and hand input to quickly and effortlessly select holograms across your scene | Learn how to auto-scroll text or fluently zoom into focused content based on what you are looking at | Examples for logging, loading and visualizing what users have been looking at in your app |
 Optimize Window |
 Dependency Window |
 Build Window |
 Input recording |
|---|---|---|---|
| Automate configuration of Mixed Reality projects for performance optimizations | Analyze dependencies between assets and identify unused assets | Configure and execute an end-to-end build process for Mixed Reality applications | Record and playback head movement and hand tracking data in editor |
Explore MRTK's various types of interactions and UI controls in this example scene.
You can find other example scenes under Assets/MixedRealityToolkit.Examples/Demos folder.
With the MRTK Examples Hub, you can try various example scenes in MRTK. You can find pre-built app packages for HoloLens(x86), HoloLens 2(ARM), and Windows Mixed Reality immersive headsets(x64) under Release Assets folder. Use the Windows Device Portal to install apps on HoloLens.
See Examples Hub README page to learn about the details on creating a multi-scene hub with MRTK's scene system and scene transition service.
 |
 |
|---|---|
| Periodic Table of the Elements is an open-source sample app which demonstrates how to use MRTK's input system and building blocks to create an app experience for HoloLens and Immersive headsets. Read the porting story: Bringing the Periodic Table of the Elements app to HoloLens 2 with MRTK v2 | Galaxy Explorer is an open-source sample app that was originally developed in March 2016 as part of the HoloLens 'Share Your Idea' campaign. Galaxy Explorer has been updated with new features for HoloLens 2, using MRTK v2. Read the story: The Making of Galaxy Explorer for HoloLens 2 |
-
Join the conversation around MRTK on Slack. You can join the Slack community via the automatic invitation sender.
-
Ask questions about using MRTK on Stack Overflow using the MRTK tag.
-
Search for known issues or file a new issue if you find something broken in MRTK code.
-
For questions about contributing to MRTK, go to the mixed-reality-toolkit channel on slack.
This project has adopted the Microsoft Open Source Code of Conduct. For more information, see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.
 Discover |
 Design |
 Develop |
 Distribute |
|---|---|---|---|
| Learn to build mixed reality experiences for HoloLens and immersive headsets (VR). | Get design guides. Build user interface. Learn interactions and input. | Get development guides. Learn the technology. Understand the science. | Get your app ready for others and consider creating a 3D launcher. |
 Spatial Anchors |
 Speech Services |
 Vision Services |
|---|---|---|
| Spatial Anchors is a cross-platform service that allows you to create Mixed Reality experiences using objects that persist their location across devices over time. | Discover and integrate Azure powered speech capabilities like speech to text, speaker recognition or speech translation into your application. | Identify and analyze your image or video content using Vision Services like computer vision, face detection, emotion recognition or video indexer. |
You can find our planning material on our wiki under the Project Management Section. You can always see the items the team is actively working on in the Iteration Plan issue.
Learn how you can contribute to MRTK at Contributing.
For details on the different branches used in the Mixed Reality Toolkit repositories, check this Branch Guide here.