No description, website, or topics provided.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
Assets
Documentation
ProjectSettings
QCAR
UWP
UnityPackageManager
.gitignore
README.md

README.md

HoloTouch

3x3Testing 2x2Testing

HoloTouch Unity package/project creates custom augmented visualizations over 3D printed models using Vuforia and Microsoft HoloLens.

Features

  • Vuforia
  • Hololens
  • Augmented visual on top of 3D printed model using the Hololens. Tracking of the model handled by Vuforia, which tracks 4 image trackers located on the corners of the 3D model.

System Requirements

The HoloLens website includes a checklist of recommended installed for HoloLens Development. This project uses Unity 2017.2.1p2 as it is recommended version for developing with Vuforia at the moment. This project repository will include Vuforia, so downloading it will be unnecessary.

  • Unity 2017.2.1p2
  • Vuforia 6.5.25
  • Visual Studio 2017
  • Windows 10 Fall Creators Update

Account Registration

Create free accounts for Unity and Vuforia. Unity Personal Version is a free beginner friendly development platform for 2D, 3D, VR and AR applications.

Vuforia Setup

Vuforia License Key

The first step is to log into the Vuforia License Manager page with the Vuforia Account you created.

  1. Click on Add License Key to create a free key enabling your Unity application to use Vuforia's image recognition services up to a specified limit.

Vuforia License Key

  1. Select Project Type: Development, create a name for your application, and click next to continue. Vuforia License Key 2

  2. Clicking on your Application Name in the License Manager will show the Vuforia license key, which will be later copied and pasted into Unity Editor. Vuforia License Key 3

Vuforia Target Image Database

  1. Go back to Vuforia License Manager page, click on Target Manager link

  2. Click on Add Database button to set up a set of images your HoloLens application will recognize. Provide a name for the database of images that will be recognized by the app and make sure Type:Devices is selected. Vuforia Target Images2

  3. When the database is created, click on the database link and click on Add Target Button to start adding the images to be recognized.
    Vuforia Target Images2

  4. Vuforia can recognize different types of targets ranging from 2D images, 3D objects, and shapes. Further information and suggestions for improving image recognition and tracking stability can be found here.

  • Included in this repository is a set 2"x2" and 3"x3" fiduciary markers that can be uploaded for image tracking.
  1. When uploading the image target, the width must be entered meters. So 0.0508m if you are uploading the 2x2" markers and 0.0762m for the 3x3" markers.

  2. Lastly give the image target a unique name that will be saved in the database, before clicking on the Add Button Vuforia Target Images 3

  3. Back on the Target Manager page, the uploaded image target will appear with a star rating indicating whether the target image will be reliably recognized and tracked by the Vuforia software. Continue uploading all of the image targets with at least 4-5 star ratings which will be used for tracking the 3D Model. Vuforia Target Images 6

  4. After uploading your image targets, click on the Download Database(All) button and select Unity Editor to download the image database files into a Unity package.

3D Model Preparation

  1. Find a 3D Model you would like to print and be augmented using the HoloTouch toolkit, or create your own using websites like https://craftml.io/ or https://www.tinkercad.com.
  • For the time being, it is recommended to use a model that are rectangular or square dimensions (Equal widths and equal length dimensions).
  1. Unity can import .obj file formats, but if your model is a .stl format, you can convert it using http://www.greentoken.de/onlineconv/ or open it with Microsoft's 3D Builder and save it an .obj file.
  • To improve the performance of the application, it is recommended to minimize and reduce the number of meshes of the model as much as possible.
  • It is also recommended that the 3D Model has a flat base allowing it to be easier affixed to a board or flat surface with the image tracking markers.
  • NOTE: Keep in mind the measurement scale of the selected 3D model and the axis the model is oriented in. Typical 3D modelling tools align the z-axis to be "UP", however Unity's coordinate system treats y-axis to be UP. If possible, in a 3D modelling software program
    • position the origin of the coordinate system to be located to the model's lower left corner
    • reorient the model so the up direction (height) of the model is aligned with the modelling software's y axis. The model's length will be in the modelling software's -x axis, and the width will be along the modelling software's z axis. Vuforia Target Images2

Configuring Project

  1. Download the HoloTouch Toolkit from this github repo

  2. Open the HoloTouch project with Unity 2017.2.1p2 editor

  3. In the Project folder tab, navigate to Assets/HoloTouch/Scenes/ folder and select on the Template.scene file. Vuforia Template Scene

  4. With the template scene selected, press Ctrl+D to make a copy of the scene and rename it.

  5. Double click to open the scene that you just created which will be your HoloLens application.

Configure Project Settings

To start configuring the Unity application to build to the HoloLens.

  1. Select File/ Build Settings / and underneath Platform, if Universal Windows Platform is not selected, select it and click on Switch Platform to build for the HoloLens. Vuforia Build Settings
  2. Click on Add Open Scene, which should include your current scene with a checked box when you are building your application to the HoloLens.
  3. Next click on Players Settings, next to the Switch Platform button
  4. The right side in the inspector panel of the Players Settings. Click to drop down the XR Settings tab. Check box the Virtual Reality Supported and Vuforia Augmented Reality options. Vuforia XR Settings
  5. At the top of the Players Settings is a text field for Product Name, which you can change to reflect the name of your application in the HoloLens.

Configuring Vuforia Target Database

The template scene will start with an AR Camera, Target-Placement-Manager, and Background Instruction Plane, shown in the Scene tab and Hierarchy Tab. The Target-Placement-Manager contains the ImgTarget1,2,3,4, which will be tracked by Vuforia.

  1. Import the Vuforia Image Database file that was downloaded from Vuforia website by navigating to Assets/Import Packages/Custom Packages and open the downloaded Image Database Unity Package.

Vuforia Target Images 7

  1. After importing all the database image Unity package, navigate to Windows/Vuforia Configuration to show the Vuforia Configuration settings in the Inspector window.

  2. In the Vuforia Configuration, paste your Vuforia App License key from your Vuforia Account into the App License Key text input field.

Vuforia Config App Key

  1. Underneath the Digital Eyewear header, select Optical See-Through for Eyewear Type and HoloLens for See Through Config

  2. Underneath Databases header the name of your image database should show up. (In my case I named my database DataViz) Select the check boxes to load and activate your databases and enable tracking with Vuforia

Vuforia Config App Key

Setting up Target Images in Unity

With the Vuforia Configurations loaded with the image database set in your Vuforia account, we'll set up a virtual scene for our objects to appear when an images are recognized.

  1. For each imgTarget1,2,3,4 set each associated image target. ImgTargets will be located sequentially left to right at each corner of the 3D model. (By default the image targets will be overlapping since an image has not been specified. The positioning and ordering of the targets will be apparent when we define the image targets)
  • imgTarget1 located at top left corner of model
  • imgTarget2 located at top right corner of model
  • imgTarget3 located at bottom left corner of model
  • imgTarget4 located at bottom right corner of model
  1. Select an imgTarget in the hierarchy tab

Selected Image Target

  1. In the inspector, the Image Target Behaviour component handles the image target settings

  2. In the Database drop down field (currently labelled as ---EMPTY---) , select your image database for the project

  3. Next in the Image Target drop down field, select the appropriate image target file name associated to the image you will place in the corners of your actual 3D printed model. (In my case my database was named DataVis, and I will use the 2x2" img target)

Selected Image Target

  1. After selecting the Image Target, the imgTarget gameobject will be re-scaled in the Scene tab to the actual size in meters of your uploaded image targets. Continue setting the image targets for the remaining imgTarget gameobjects. All Images Set

Import 3D Model

Next import your 3D obj model into Unity.

  1. Navigate to Assets/Import New Asset and find the .obj file of the 3D model for your application. Depending on the size of the model, it may take a while to complete the import.
  2. In the Project tab, navigate to Assets/ folder and select on the imported .obj model file.
  3. The Inspector tab will show the Import Settings, which includes a field for Scale Factor. Unity's base units are in meters, so if the model was created in mm scaling, then scale down the model by .001.
  4. Drag the model in to the scene to check whether the import scaling is correct.
  5. As mentioned earlier, a caveat about Unity is that the Y-Axis is in the "UP" direction. So if the imported 3D model looks rotated then the its original coordinates were oriented differently. Fix this by dragging the .obj file into the scene and reorient the rotation of the model changing the model's GameObject's transform rotations in the Inspector tab. Once the GameObject's is oriented properly, drag the model GameObject's from the hierarchy into Asset/HoloTouch/Models folder to save it as a new Prefab. Rename the new prefab as "model-reoriented" and delete the model GameObject from the Hierarchy tab. All Images Set

Configuration HoloTouch Editor

The next step is to orient and position the image targets and 3D model in the scene using the HoloToch Editor.

  1. Navigate to HoloTouch/Settings in the menu bar, which will open the HoloTouch Editor window. This window can be expanded larger or attached into the Unity Editor like any other window.Holotouch Settings
  2. From the Assets/HoloTouch/Models folder in the Project tab, drag the "model-reoriented" prefab into the HoloTouch Editor's 3D model field. This sets the viewed model that will be displayed when Vuforia tracks the markers.

Set Model

  1. Next enter the 3D model's width and length (in meters) into the HoloTouch Editor.

  2. Drag the Target-Placement-Manager GameObject from the hiearchy into the HoloTouch Editor's Target Manager field. Set Target Manager

  3. Under the Image Target Info header, if the Image Target 1,2,3,4 inputs show up as None (Game Object) , drag each imgTarget1 gameobject from the Hierarchy tab into each appropriate field. Setting Image Targets

  4. Next enter the image targets width and length (in meters) into the HoloTouch Editor.

  5. With all of the fields filled out in the HoloTouch Editor, press the Apply Settings button. This should automatically orient the 3D model in center, with imgTarget1,2,3,4 placed in the appropriate corners. If the scaling of the model and image target dimensions set correctly in the HoloTouch Editor, then the corners of the image targets should only touch the corner of the 3D model. Click and inspect the Target-Placement-Manager GameObject to make sure that the TargetModelManager script is referencing the created 3D model in the scene. Applying HoloTouchEditor settings

Building Project for Visual Studio

Once the image targets and model are placed in the scene, its time to build the Unity project to the Hololens.

  1. Navigate to File/Build Settings and make sure that the current scene is checked, and Universal Windows Platform is already selected under Platform.
  2. Click Build, which will ask to save the build in a new folder. Try saving this new folder outside of your Unity project folder.
  3. After Unity finishes saving the build, navigate to this folder and locate the Visual Studio Solution file (.sln) , which should be the same name as the one set in the Players Setting step. Building Project

Preparing the Hololens

Instructions from https://docs.microsoft.com/en-us/windows/mixed-reality/using-visual-studio

Start by enabling Developer Mode on your device so Visual Studio can connect to it.

  1. Turn on your HoloLens and put on the device.
  2. Perform the bloom gesture to launch the main menu.
  3. Gaze at the Settings tile and perform the air-tap gesture. Perform a second air tap to place the app in your environment. The Settings app will launch after you place it.
  4. Select the Update menu item.
  5. Select the For developers menu item.
  6. Enable Developer Mode. This will allow you to deploy apps from Visual Studio to your HoloLens.

Deploying in Visual Studio

  1. In the folder that the Unity project was built and saved into, double click on the .sln file, which will open up Visual Studio 2017.
  2. Connect your HoloLens to your laptop/computer with a microUSB cable, and make sure the HoloLens is on.
  3. Select Release and x86 build configuration for your app
  4. Select Device in the deployment target drop-down menuVisual Studio Settings
  5. Select green arrow icon next to Device to deploy your app
  6. The first time you deploy an app to your HoloLens from your PC, you will be prompted for a PIN. Follow the Pairing your device instructions in https://docs.microsoft.com/en-us/windows/mixed-reality/using-visual-studio.
  7. Once Visual Studio finishes deploying the application, you can unplug your HoloLens from your laptop and find your application in your Applications folder.

Physical 3D Model Print Preparation

  • Affix the 3D printed model in the center of a flat surface
  • Print and cut out the 4 image targets, making sure that the dimensions and scaling are as specified when uploading to the Vuforia image database as Vuforia will scale the augmented projections based on the size of the printed image targets
  • Affix the 4 image targets on the corners of the 3D printed model, ensuring that the orientation and placement are exactly reflect the placement in the Unity editor.

Extra Functionalities

  • In the Target-Tracking-Manager script there is a boolean field for Debug Mode. Check this true if you want an additional visual cue to indicate which and when the image targets are recognized by the Hololens. When this is set True and deployed on the Hololens, a capsule will appear above each of the image targets are recognized. When the HoloLens loses tracking on an image target, the capsule will disappear.

Suggested Improvements

  1. The recorded HoloLens video of the project shows that the augmented model is offset from the actual 3D model, but in the headset they do overlap. Tracking performance is much better when deployed on a cell phone compared to the HoloLens
  2. Add voice control to toggle Debug Mode outside of the editor mode
  3. Additional information embedded into augmented overlay.

License

Copyright <2018>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.