A sample Unity project for Microsoft HoloLens that lets you interact with a floating robot head using speech recognition, and getting answers spoken back by the "bot" using Text-to-Speech. The commands sent to the bot are powered by the Microsoft Bot Framework and LUIS from Microsoft Cognitive Services.
To get started with HoloLens, visit the Windows Holographic Dev Center.
- Hovering bot head (aka HoloBot) with looping sound
- Tap the HoloBot to move it to a different location, tap again to place
- Gaze at the HoloBot to trigger the speech recognizer, you will hear a "Ping" sound
- Speak "commands" using natural language
- The HoloBot speaks back to you using Speech Synthesis (aka Text-to-Speech, or TTS)
- All sounds and speech use spatial audio that originate from the HoloBot
- The "brain" of the HoloBot can be any public bot built with the Microsoft Bot Framework
The bot demonstrated in this video is The Maker Show Bot, also found here on GitHub.
- The HoloBot model and sounds come from the Holographic Academy tutorial: Holograms 240: Sharing Holograms
- Uses Gaze Manager & Gesture Manager from HoloLens Unity Toolkit (Input scripts)
- Uses Text to Speech Manager from HoloLens Unity Toolkit (Utilities scripts)
- Make sure to copy a UWP build of Newtonsoft.Json.dll in the /Plugins folder of the HoloBot Unity project
- Edit the Inspector settings for the Newtonsoft.Json.dll plugin as follows:
- Create and register your bot as per the intructions at https://dev.botframework.com. Since HoloBot uses free natural language dictation, it is highly recommended that your bot support NLP via Natural Language Understanding Service (LUIS) from Microsoft Cognitive Services
- Enable the Direct Line channel on your bot
- Generate and copy your Direct Line secret (aka API key)
- Open BotService.cs in the /Scripts folder of the HoloBot Unity project and paste your Direct Line secret in the _APIKEY private string
I want to offer special thanks to the following people who have helped me in building this sample app:
- Jarez Bienz, for the Text-to-Speech component that he wrote the the HoloLens Unity Toolkit, and for his help in integrating my UWP Bot Framework code into Unity
- Kat Haris, for her awesome Unity skills and helping me with audio sources triggers
- Vanessa Arnauld & Sara Nagy, for being incredible holographic "enablers" :)
- The whole Microsoft Holographic Academy team & mentors - especially Pat - for their awesome training, resources, patience and help
- Twitter: @ActiveNick
- Blog: AgeofMobility.com
- SlideShare: http://www.slideshare.net/ActiveNick