HoloTouch Unity package/project creates custom augmented visualizations over 3D printed models using Vuforia and Microsoft HoloLens.
- Augmented visual on top of 3D printed model using the Hololens. Tracking of the model handled by Vuforia, which tracks 4 image trackers located on the corners of the 3D model.
The HoloLens website includes a checklist of recommended installed for HoloLens Development. This project uses Unity 2017.2.1p2 as it is recommended version for developing with Vuforia at the moment. This project repository will include Vuforia, so downloading it will be unnecessary.
- Unity 2017.2.1p2
- Vuforia 6.5.25
- Visual Studio 2017
- Windows 10 Fall Creators Update
Create free accounts for Unity and Vuforia. Unity Personal Version is a free beginner friendly development platform for 2D, 3D, VR and AR applications.
Vuforia License Key
- Click on Add License Key to create a free key enabling your Unity application to use Vuforia's image recognition services up to a specified limit.
Vuforia Target Image Database
Go back to Vuforia License Manager page, click on Target Manager link
Click on Add Database button to set up a set of images your HoloLens application will recognize. Provide a name for the database of images that will be recognized by the app and make sure Type:Devices is selected.
Vuforia can recognize different types of targets ranging from 2D images, 3D objects, and shapes. Further information and suggestions for improving image recognition and tracking stability can be found here.
- Included in this repository is a set 2"x2" and 3"x3" fiduciary markers that can be uploaded for image tracking.
When uploading the image target, the width must be entered meters. So 0.0508m if you are uploading the 2x2" markers and 0.0762m for the 3x3" markers.
Back on the Target Manager page, the uploaded image target will appear with a star rating indicating whether the target image will be reliably recognized and tracked by the Vuforia software. Continue uploading all of the image targets with at least 4-5 star ratings which will be used for tracking the 3D Model.
After uploading your image targets, click on the Download Database(All) button and select Unity Editor to download the image database files into a Unity package.
3D Model Preparation
- Find a 3D Model you would like to print and be augmented using the HoloTouch toolkit, or create your own using websites like https://craftml.io/ or https://www.tinkercad.com.
- For the time being, it is recommended to use a model that are rectangular or square dimensions (Equal widths and equal length dimensions).
- Unity can import .obj file formats, but if your model is a .stl format, you can convert it using http://www.greentoken.de/onlineconv/ or open it with Microsoft's 3D Builder and save it an .obj file.
- To improve the performance of the application, it is recommended to minimize and reduce the number of meshes of the model as much as possible.
- It is also recommended that the 3D Model has a flat base allowing it to be easier affixed to a board or flat surface with the image tracking markers.
- NOTE: Keep in mind the measurement scale of the selected 3D model and the axis the model is oriented in. Typical 3D modelling tools align the z-axis to be "UP", however Unity's coordinate system treats y-axis to be UP. If possible, in a 3D modelling software program
- position the origin of the coordinate system to be located to the model's lower left corner
- reorient the model so the up direction (height) of the model is aligned with the modelling software's y axis. The model's length will be in the modelling software's -x axis, and the width will be along the modelling software's z axis.
Download the HoloTouch Toolkit from this github repo
Open the HoloTouch project with Unity 2017.2.1p2 editor
With the template scene selected, press Ctrl+D to make a copy of the scene and rename it.
Double click to open the scene that you just created which will be your HoloLens application.
Configure Project Settings
To start configuring the Unity application to build to the HoloLens.
- Select File/ Build Settings / and underneath Platform, if Universal Windows Platform is not selected, select it and click on Switch Platform to build for the HoloLens.
- Click on Add Open Scene, which should include your current scene with a checked box when you are building your application to the HoloLens.
- Next click on Players Settings, next to the Switch Platform button
- The right side in the inspector panel of the Players Settings. Click to drop down the XR Settings tab. Check box the Virtual Reality Supported and Vuforia Augmented Reality options.
- At the top of the Players Settings is a text field for Product Name, which you can change to reflect the name of your application in the HoloLens.
Configuring Vuforia Target Database
The template scene will start with an AR Camera, Target-Placement-Manager, and Background Instruction Plane, shown in the Scene tab and Hierarchy Tab. The Target-Placement-Manager contains the ImgTarget1,2,3,4, which will be tracked by Vuforia.
- Import the Vuforia Image Database file that was downloaded from Vuforia website by navigating to Assets/Import Packages/Custom Packages and open the downloaded Image Database Unity Package.
After importing all the database image Unity package, navigate to Windows/Vuforia Configuration to show the Vuforia Configuration settings in the Inspector window.
In the Vuforia Configuration, paste your Vuforia App License key from your Vuforia Account into the App License Key text input field.
Underneath the Digital Eyewear header, select Optical See-Through for Eyewear Type and HoloLens for See Through Config
Underneath Databases header the name of your image database should show up. (In my case I named my database DataViz) Select the check boxes to load and activate your databases and enable tracking with Vuforia
Setting up Target Images in Unity
With the Vuforia Configurations loaded with the image database set in your Vuforia account, we'll set up a virtual scene for our objects to appear when an images are recognized.
- For each imgTarget1,2,3,4 set each associated image target. ImgTargets will be located sequentially left to right at each corner of the 3D model. (By default the image targets will be overlapping since an image has not been specified. The positioning and ordering of the targets will be apparent when we define the image targets)
- imgTarget1 located at top left corner of model
- imgTarget2 located at top right corner of model
- imgTarget3 located at bottom left corner of model
- imgTarget4 located at bottom right corner of model
- Select an imgTarget in the hierarchy tab
In the inspector, the Image Target Behaviour component handles the image target settings
In the Database drop down field (currently labelled as ---EMPTY---) , select your image database for the project
Next in the Image Target drop down field, select the appropriate image target file name associated to the image you will place in the corners of your actual 3D printed model. (In my case my database was named DataVis, and I will use the 2x2" img target)
- After selecting the Image Target, the imgTarget gameobject will be re-scaled in the Scene tab to the actual size in meters of your uploaded image targets. Continue setting the image targets for the remaining imgTarget gameobjects.
Import 3D Model
Next import your 3D obj model into Unity.
- Navigate to Assets/Import New Asset and find the .obj file of the 3D model for your application. Depending on the size of the model, it may take a while to complete the import.
- In the Project tab, navigate to Assets/ folder and select on the imported .obj model file.
- The Inspector tab will show the Import Settings, which includes a field for Scale Factor. Unity's base units are in meters, so if the model was created in mm scaling, then scale down the model by .001.
- Drag the model in to the scene to check whether the import scaling is correct.
- As mentioned earlier, a caveat about Unity is that the Y-Axis is in the "UP" direction. So if the imported 3D model looks rotated then the its original coordinates were oriented differently. Fix this by dragging the .obj file into the scene and reorient the rotation of the model changing the model's GameObject's transform rotations in the Inspector tab. Once the GameObject's is oriented properly, drag the model GameObject's from the hierarchy into Asset/HoloTouch/Models folder to save it as a new Prefab. Rename the new prefab as "model-reoriented" and delete the model GameObject from the Hierarchy tab.
Configuration HoloTouch Editor
The next step is to orient and position the image targets and 3D model in the scene using the HoloToch Editor.
- Navigate to HoloTouch/Settings in the menu bar, which will open the HoloTouch Editor window. This window can be expanded larger or attached into the Unity Editor like any other window.
- From the Assets/HoloTouch/Models folder in the Project tab, drag the "model-reoriented" prefab into the HoloTouch Editor's 3D model field. This sets the viewed model that will be displayed when Vuforia tracks the markers.
Next enter the 3D model's width and length (in meters) into the HoloTouch Editor.
Next enter the image targets width and length (in meters) into the HoloTouch Editor.
With all of the fields filled out in the HoloTouch Editor, press the Apply Settings button. This should automatically orient the 3D model in center, with imgTarget1,2,3,4 placed in the appropriate corners. If the scaling of the model and image target dimensions set correctly in the HoloTouch Editor, then the corners of the image targets should only touch the corner of the 3D model. Click and inspect the Target-Placement-Manager GameObject to make sure that the TargetModelManager script is referencing the created 3D model in the scene.
Building Project for Visual Studio
Once the image targets and model are placed in the scene, its time to build the Unity project to the Hololens.
- Navigate to File/Build Settings and make sure that the current scene is checked, and Universal Windows Platform is already selected under Platform.
- Click Build, which will ask to save the build in a new folder. Try saving this new folder outside of your Unity project folder.
- After Unity finishes saving the build, navigate to this folder and locate the Visual Studio Solution file (.sln) , which should be the same name as the one set in the Players Setting step.
Preparing the Hololens
Start by enabling Developer Mode on your device so Visual Studio can connect to it.
- Turn on your HoloLens and put on the device.
- Perform the bloom gesture to launch the main menu.
- Gaze at the Settings tile and perform the air-tap gesture. Perform a second air tap to place the app in your environment. The Settings app will launch after you place it.
- Select the Update menu item.
- Select the For developers menu item.
- Enable Developer Mode. This will allow you to deploy apps from Visual Studio to your HoloLens.
Deploying in Visual Studio
- In the folder that the Unity project was built and saved into, double click on the .sln file, which will open up Visual Studio 2017.
- Connect your HoloLens to your laptop/computer with a microUSB cable, and make sure the HoloLens is on.
- Select Release and x86 build configuration for your app
- Select Device in the deployment target drop-down menu
- Select green arrow icon next to Device to deploy your app
- The first time you deploy an app to your HoloLens from your PC, you will be prompted for a PIN. Follow the Pairing your device instructions in https://docs.microsoft.com/en-us/windows/mixed-reality/using-visual-studio.
- Once Visual Studio finishes deploying the application, you can unplug your HoloLens from your laptop and find your application in your Applications folder.
Physical 3D Model Print Preparation
- Affix the 3D printed model in the center of a flat surface
- Print and cut out the 4 image targets, making sure that the dimensions and scaling are as specified when uploading to the Vuforia image database as Vuforia will scale the augmented projections based on the size of the printed image targets
- Affix the 4 image targets on the corners of the 3D printed model, ensuring that the orientation and placement are exactly reflect the placement in the Unity editor.
- In the Target-Tracking-Manager script there is a boolean field for Debug Mode. Check this true if you want an additional visual cue to indicate which and when the image targets are recognized by the Hololens. When this is set True and deployed on the Hololens, a capsule will appear above each of the image targets are recognized. When the HoloLens loses tracking on an image target, the capsule will disappear.
- The recorded HoloLens video of the project shows that the augmented model is offset from the actual 3D model, but in the headset they do overlap. Tracking performance is much better when deployed on a cell phone compared to the HoloLens
- Add voice control to toggle Debug Mode outside of the editor mode
- Additional information embedded into augmented overlay.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.