Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time


HoloLens App for detecting, reading and displaying text in the environment

Projects using this code should acknowledge:

An Augmented Reality Sign-reading Assistant for Users with Reduced Vision. J. Huang, M. Kinateder, M.J. Dunn, W. Jarosz, X. Yang and E.A. Cooper PLOS One, 14(1), e0210630, 2019


This application is based of the MS HoloToolkit and in particular it uses the spatial mapping abilities. Basic familiarity with developing and deploying for MS HoloLens is assumed.

To get started with HoloLens, follow tutorials on the HoloAcademy website (Holograms 101, 212, and 230 in particular).

For ease of use, the necessary assets from the HoloToolkit are included.

This application uses the Google Vision API. The application does not run as is and you need an account for the Google Vision API. For more information see Google Vision.


make sure project is set up to work with Github


  • MS HoloLens enabled for developer mode
  • Bluetooth clicker (toggles augmentation on and off)

Deploying the app

  1. Clone repo to your computer
  2. Open project in Unity (make sure that the SignAR scene is loaded)
  3. Open ApiManager.cs and add your Google vision api account information into line 26
  4. Build and Deploy (see HoloAcademy for tutorials)

Using the app

General App description

The SignAR application will detect text in the direction the user is looking at and place spherical icons wherever text exists. The number of icons to display is restricted to 5 (can be adjusted in Unity).

Every icon represents a sign containing text in the real world. These icons are color-coded (green for confident, orange for semi-confident, and red for doubtful) and can be selected. Once selected, the application will read and display the text stored at that icon, and the icon will disappear. See below for a list of modes and voice commands.

The clicker can be used with the application. If the user is gazing at an object, a click will select the object. If the user is not gazing at an object and the user previously selected an icon, then the click will deselect the previous icon. Otherwise, a click will tell the HoloLens to detect text if it is manual mode.


  • Audio Only Mode (default): when the application detects text, the application will read all of the text in front of the user without showing any icons
  • Icon Mode: when the application detects text, the application will first show icons in the scene. The user can tap on any of the icons, and the application will read/display the text to the user

Voice Commands

  • What's here: Tell the application to search for text in the scene
  • Icon mode: Switch to mode to show icons
  • Audio only mode: Switch to audio-only mode
  • Clear Icons: Delete all icons in the scene (Icon mode only)
  • Show me: Reads and displays the text of the icon that is currently gazed at (Icon mode only)
  • Read all here: Reads text of all icons that are currently in front of the user
  • Hide words: Hides the text of the icon that is currently selected (Icon mode only)


source code for SignAR



Unknown and 2 other licenses found

Licenses found






No releases published


No packages published