Hololens 2 Unity tutorial (Windows)

A collection of some of the setups I had to go trough, and things I had to learn in order to develop applications for Hololens 2 in unity.

1. Setup and first build

2. Run and debug your Hololens 2 with Unity

3. Some tips to follow while developing to Hololens

4. If you need to build to Hololens 1

5. How to get GPS coordinates from Hololens

6. Contribute to this repository

Setup and first build

In this section, I'll be teaching you how I got my first project up and running in Hololens 2, from an application built in Unity.

First step - installing packages and applications

1. Let's start by installing Visual Studio Community 2022, in case you don't have it yet. Here you can find the microsoft page to install it. Whether you downloaded it just now or already had VS on your machine, be sure to add the following workloads on your installation (You can have access to that if you go to windows Settings -> System -> Storage -> Apps & Features, then search for Microsoft Visual Studio Installer, click Modify and then Modify again):
   • .NET desktop development (or download manually here).
   • Desktop development with C++.
   • Universal Windows Platform (UWP) development.
   • Game development with Unity.
2. Now, if you don't have the Windows 10 SDK, you can find the installer here.
3. Download the Emulator here (optional, just in case you don't have the Hololens in hand).
4. Open Unity Hub and install the Universal Windows Platform Build Support module for the unity version that you want to use (unity 2019.3 not recommended due to compile errors in ARM build architecture). You can do that by opening unity hub -> Installs -> clicking the gear button for the version you want -> Add Modules -> Universal Windows Platform Build Support.
5. Go to your windows Settings -> Update & Security -> For developers -> Developer Mode -> On
6. Last but not least, install the Mixed Reality Feature Tool from Microsoft here.

We're all set, now it's time to build our first application!

Second step - creating first unity project

1. Open Unity Hub.
2. In the Projects tab, click New Project.
3. In the dropdown underneath the New project text, select the unity version that you have installed Universal Windows Platform Build Support.
4. The template must be 3D Core.
5. Click Create Project.

Third step - configuring build for Hololens 2

1. Go to File -> Build Settings
2. Select Universal Windows Platform.
3. Click Switch Platform if needed.
4. Now make sure to set the following settings: (if any error occurs, check if all the installations are properly working)
   • Target device: HoloLens
   • Architecture: ARM64
   • Build Type: D3D Project
   • Target SDK Version: Latest Installed
   • Minimum Platform Version: 10.0.10240.0
   • Visual Studio Version: Latest installed
   • Build and Run on: Local Machine
   • Build configuration: Release (there are known performance issues with Debug)

Forth step - importing and configuring MRTK in Unity

1. Open the previously installed Mixed Reality Feature Tool, and run MixedRealityFeatureTool.exe (unzip the folder if necessary).
2. In the Mixed Reality Feature Tool, select Start.
3. Select the project folder that we've created by clicking the three dots in Project Path.
4. Click Discover Features.
5. On the Discover Features page click the "+" button to the left of Mixed Reality Toolkit (0 of 10) and select the latest version of Mixed Reality Toolkit Foundation.
6. On the Discover Features page Click the "+" button to the left of Platform Support (0 of 5) and select the latest version of Mixed Reality OpenXR Plugin.
7. Click Get Features -> Import -> Approve -> Exit.
8. Go back to Unity.
9. Restart Unity if asked to do so.
10. The MRTK Project Configurator should appear. If it doesn't, on the menu bar select Mixed Reality -> Toolkit -> Utilities -> Configure Project for MRTK.
11. Click Unity OpenXR Plugin (recommended).
12. On the Welcome to MRTK! screen, click Show XR Plug-In Management Settings.
13. On the Project Settings window now open be sure that you're on the XR Plug-in Management page with the Universal Windows Platform settings (Windows logo tab) displayed.
14. Select Initialize XR on Startup if not selected, and then, under Plugin Providers, click Open XR.
15. Two items will appear underneath OpenXR. Select Microsoft HoloLens feature group.
16. Now a yellow warning triangle next to OpenXR. Hover your cursor over the triangle, and then select it.
17. Click Fix All.
18. One issue still remains "At least one interaction profile must be added ..."
19. Click Edit in that warning. You're now in the settings for the OpenXR plugin in the Project Settings window.
20. First click the Depth Submission Mode drop down and then select Depth 16 Bit.
21. Underneath Interaction Profiles, there is a plus sign (+) button.
22. Click the button three times, with the following profiles each time:
    • Eye Gaze Interaction Profile
    • Microsoft Hand Interaction Profile
    • Microsoft Motion Controller Profile
23. If any profile added appear with a yellow triangle next to it, fix the warnings by clicking the triangle and then Fix All button.
24. In the Project Settings window under OpenXR Feature Groups, make sure that the following are selected:
    • Microsoft HoloLens
    • Hand Tracking
    • Motion Controller Model
25. Close the Project Settings window.
26. Back at the MRTK Project Configurator window, select Apply Settings.
27. Click Next, then Apply.

Now you're ready to start working with unity and Hololens. The next step is not necessary, so you can jump to sixth step if you want to.

Fifth step - example scene with many interactions implemented (optional)

1. Go to the official MRTK github repository here
2. Go to tags next to branches and select the latest release. For me it's Microsoft Mixed Reality Toolkit v2.7.3.
3. Scroll down to Assets and download the package with examples in the version you chose, in my case, "Microsoft.MixedReality.Toolkit.Unity.Examples.2.7.3.unitypackage".
4. Open unity and on the menu bar, select Assets -> Import package -> Custom package, and click on the example package that you've just downloaded from github.
5. Just click Import when prompted with the package contents.
6. If it worked, there should be a folder named "Examples" with the path Assets/MRTK/Examples.
7. Now open the example scene in Assets/MRTK/Examples/Demos/HandTracking/Scenes/HandInteractionExamples.unity.
8. If asked to import Text Mesh Pro (TMP), click Import TMP Essentials. Close the TMP window.
9. Last but not least, go to File -> Build Settings on the menu bar, and click Add Open Scenes.
10. Click Build and choose a folder to save it.
11. Wait until it's finished.

Sixth step - enable developer mode in Hololens and setting Visual Studio to build remotely thought wifi or cable

1. Turn on your Hololens 2.
2. Enter the menu and Select Settings -> Update and Security -> For developers and enable developer mode.
3. Now on your pc, go to the folder that you have built the unity app and click on the file with the sln extension, in my case, "HoloTutorial.sln".
4. That should open Visual Studio. If not, open it manually by selecting in the menu bar File -> Open -> Project/Solution and choose the file with sln extension.
5. If the startup project in your Visual Studio is not set properly, just right mouse click above the desired solution (all of them are available in the right) and select Set as Startup Project. The startup project must be the one with the same name of the solution you've built in Unity.
6. Now on the menu bar, go to Project -> Properties.
7. Configure Visual Studio for HoloLens by selecting the Master or Release configuration and the ARM64
8. Under Configuration Properties click General and ensure that you have the correct installed Visual Studio in the Platform Toolset box, in my case, Visual Studio 2022, then click Apply and close this window.architecture. In my case, release.
9. Click the deployment target drop-down and then do one of the following:
   • If you're building and deploying via USB, select Device
   • If you're building and deploying via Wi-Fi connect to the same network as the Hololens, select Remote Machine and do the following:
     • On the menu bar, go to Project -> Properties, and then under Configuration Properties click Debugging.
     • You must enter the machine name of the Hololens which is the local IP address of the device (to find the IP of the hololens, open the Settings again on the device, and go to Network & Internet -> Wi-Fi, scroll all the way down and click Hardware properties. It should be displayed as IPv4 address)
     • Click Ok and close the window
10. Now build and run the application by clicking the green play button.
11. If Visual Studio asks for a PIN, open the menu in Hololens 2 and Select Settings -> Update and Security -> For developers, then click Pair. There should appear a PIN.

If all worked, the application should appear at any seconds on your device as soon as it finish building! That's all!

Debug your Hololens

Now I'm going to show you some tricks to help you debug your Hololens application in Unity.

Debug memory and CPU usage

This one is pretty simple, all you need to do is create a GameObject and add the MixedRealityToolkit script to it (if that doesn't exist) and go to Diagnostics -> Enable Diagnostic System.

Use Holographic Remoting Player to run your project in Hololens from your Unity editor wirelessly

Great way to quickly debug your app without building and deploying a full project.

Unity uses Holographic Remoting to enter "Play Mode" in Unity to preview your content on a real HoloLens device. Play Mode can also be used with a Windows Mixed Reality headset attached to your development PC.

1. Install Holographic Remoting Player on your Hololens 2.
2. Open Holographic Remoting Player app on the HoloLens 2 and make sure to press the "Play" button on the app's interface.
3. Read the IP adress that appears in yellow.
4. Now in Unity, on the menu bar, go to Mixed Reality -> Remoting -> Holographic remoting for play mode.
5. In the Holographic Remoting for Play Mode window, in the Remote Host Name box, enter the IP address of your HoloLens 2.
6. Select Enable Holographic Remoting for Play Mode.
7. Click the Play button to enter Play Mode and voila!

Ps: Keep in mind that Vuforia is not going to work properly in Hololens with Holographic Remoting Player because of the webcam recognition (if you managed to make it work feel free to make a pull request in this repo).

Tips and tricks for Hololens

Some things I had to learn to overcome Hololens limitations (there're a lot of them).

Optimizing

The tips for optimizing your project and code for Hololens 2 are pretty much the same to any Unity project. Some good tips can be found here and here. Remember to always set the flag Master if the build is the final one, that way all the debug code is removed from the app and performance is saved.

Screen recording

There are various ways to capture Hololens screen, but from my experience, record screen directly from the device without connecting it to a PC is the best one.

App in background

Once you close the app from the menu button in Hololens, the app continues to run in background, therefore, you need to close it again in the space you're in.

Display

Avoid dark colors, because they're just transparent in AR.

Build

Hololens 2 apparently does not accept multiple pc's deploying to the same device, so if that's the case, you'll always have to delete the current app from the Hololens if the build was made with a different pc from last time.

Side load and install App in Hololens 2

To remotely install and distribute your app built in Unity into Hololens 2 (not yet in microsoft store), without the need of having the device in hands, follow this link and this link.

If you need to build to Hololens 1

Everything that was presented in this document should work for Hololens 1 except for the Architecture. From what I know, it should be x86. The rest should work.

How to get GPS coordinates from Hololens

Much thanks to this guy, that gave me the insights to solve this issue.

Hololens doesn't have GPS sensors in it, because it was made to be used indoors. But that doesn't mean that it's not possible to find out where it is located. One possible solution is to connect the Hololens with bluetooth devices and get their location instead (if you want to do this solution, this links can be useful: 1, 2, 3).

For the ones that don't have other devices available or just want to get the location without having to connect and depend on external devices, I'm going to show you how. This topic is extremely useful because I have not seen anyone solve this issue without using bluetooth devices. The solution is to use all listed Wifi to estimate a possible latitude and longitude of the device. To do this, you need to use UWP Windows.Devices.Geolocation Namespace. The accuracy of the location information depends on the available source (Wifi for Hololens). The latitude and longitude may vary within the following ranges:

• GPS : within approximately 10 meters
• Wi-Fi : between approximately 30 meters and 500 meters <- The one we're using for Hololens
• Cell towers : between approximately 300 meters and 3,000 meters
• IP address : between approximately 1,000 meters and 5,000 meters

This is a pretty straightforward process, as we don't need to worry about estimating the location or checking witch source is available, because the UWP library does that for us. To do this, we need to create a script, that I've called UWPGeolocation. The script can be copied below:

using System.Collections;
using System;
using System.Collections.Generic;
using UnityEngine;

#if WINDOWS_UWP
using Windows.Devices.Geolocation;
#endif

public class UWPGeolocation
{
#if WINDOWS_UWP
    private static uint _desireAccuracyInMetersValue = 0;
    
    public static async void GetLocation(Action<Geoposition> callbackLocationData, Action<string> callbackFailed)
    {
        var accessStatus = await Geolocator.RequestAccessAsync();
        switch (accessStatus)
        {
            case GeolocationAccessStatus.Allowed:
                // If DesiredAccuracy or DesiredAccuracyInMeters are not set (or value is 0), DesiredAccuracy.Default is used.
                Geolocator geolocator = new Geolocator { DesiredAccuracyInMeters = _desireAccuracyInMetersValue };

                // Subscribe to StatusChanged event to get updates of location status changes
                geolocator.StatusChanged += OnStatusChanged;

                // Carry out the operation
                Geoposition pos = await geolocator.GetGeopositionAsync();

                callbackLocationData(pos);
                // notify user: Location updated
                break;

            case GeolocationAccessStatus.Denied:
                // notify user: Access to location is denied
                callbackFailed("Access to location is denied");
                break;

            case GeolocationAccessStatus.Unspecified:
                // notify user: Unspecified error
                callbackFailed("Unspecified error");
                break;
        }
    }

    private static void OnStatusChanged(Geolocator sender, StatusChangedEventArgs args)
    {
        switch (args.Status)
        {
            case PositionStatus.Ready:
                // Location platform is providing valid data.
                break;

            case PositionStatus.Initializing:
                // Location platform is attempting to acquire a fix. 
                break;

            case PositionStatus.NoData:
                // Location platform could not obtain location data.
                break;

            case PositionStatus.Disabled:
                // The permission to access location data is denied by the user or other policies.
                break;

            case PositionStatus.NotInitialized:
                // The location platform is not initialized. This indicates that the application 
                // has not made a request for location data.
                break;

            case PositionStatus.NotAvailable:
                // The location platform is not available on this version of the OS.
                break;
        }
    }
#endif
}

To call the function and find out the location you can use the code below wherever you want to:

#if WINDOWS_UWP
var latitude, longitude;
UWPGeolocation.GetLocation(pos => {
   latitude = pos.Coordinate.Point.Position.Latitude;
   longitude = pos.Coordinate.Point.Position.Longitude;
}, err => {
   Debug.LogError(err);
});
#endif

Notice how we first need to check if GeolocationAccessStatus is Allowed get the GPS coordinates. But Hololens needs to know that you want to access the device location in order to ask user permission. Luckily, this is a simple fix. In the Unity menu bar, go to Edit -> Project Settings -> Player -> Publishing Settings -> Capabilities -> Look for Location and check the toggle button. Another way you can do this if your app is already built, is to go to the Visual Studio project and do the following: Open your project -> Look for Package.appxmanifest -> Capabilities -> Look for Location and check the toggle button. This way, when your app opens and you request the user location for the first time, Hololens will pop up a dialog box asking fot the user permission.

And that's it!

Just an observation, the #if WINDOWS_UWP is super important because the Unity editor will point you an error that it does not recognize Windows.Devices.Geolocation Namespace. So you can't test this while in Editor, you need to build the app for Hololens and test this there. Everything you do that relies on this location must also have the #if WINDOWS_UWP branch. Keep in mind that this part of the code is only compiled when built, so the syntax errors will only appear once you ask Unity to build.

More info here

Contribute to this repository

This repository was created with the purpose of helping people that are struggling with Hololens. But I am by no means an expert in this device, so any help, tips or personal experience are welcome as a pull request. Thanks!

References

https://docs.microsoft.com/en-us/windows/mixed-reality/develop/advanced-concepts/using-visual-studio?tabs=hl2

https://docs.microsoft.com/en-us/windows/mixed-reality/develop/install-the-tools?tabs=unity

https://www.youtube.com/watch?v=wSPXTRYxq9A

https://github.com/microsoft/MixedRealityToolkit-Unity/releases/tag/2.7.3

https://docs.microsoft.com/en-us/learn/modules/learn-mrtk-tutorials/

https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unity/unity-play-mode?tabs=openxr

https://learn.microsoft.com/en-us/hololens/app-deploy-app-installer

https://learn.microsoft.com/en-us/answers/questions/576485/hololens-2-gps-location.html

https://learn.microsoft.com/en-us/answers/questions/576485/hololens-2-gps-location.html

https://learn.microsoft.com/en-us/uwp/api/windows.devices.geolocation?view=winrt-22621

https://learn.microsoft.com/en-us/windows/uwp/maps-and-location/get-location

Hope I helped some of you guys!