WARNING: This repository is no longer maintained ⚠️

This repository will not be updated. The repository will be kept available in read-only mode.

Create voice commands for VR experiences with Watson services

Read this in other languages: 한국어, 中国.

This is a version of the Speech Sandbox that uses Watson Translator to translate from any language (Japanese in this example) to English, and then passing to Speech-to-Text and Watson Assistant to extract intents and entities.

Change Scripts/SpeechSandboxStreaming.cs line #51 to use a different language translation model:

private string _translationModel = "ja-en";

For all translation models

Watson Conversation is now Watson Assistant. Although some images in this code pattern may show the service as Watson Conversation, the steps and processes will still work.

In this Code Pattern we will create a Virtual Reality game based on Watson's Speech-to-Text and Watson's Assistant services.

In Virtual Reality, where you truly "inhabit" the space, speech can feel like a more natural interface than other methods. Providing speech controls allows developers to create more immersive experiences. Google Cardboard is by far the most popular VR headset platform, with 84 million sold in 2016 (http://www.hypergridbusiness.com/2016/11/report-98-of-vr-headsets-sold-this-year-are-for-mobile-phones) in 2016.

When the reader has completed this Code Pattern, they will understand how to:

• Add IBM Watson Speech-to-Text and Assistant to a Virtual Reality environment build in Unity.

Flow

1. User interacts in virtual reality and gives voice commands such as "Create a large black box".
2. The Android phone microphone picks up the voice command and the running application sends it to Watson Speech-to-Text.
3. Watson Speech-to-Text converts the audio to text and returns it to the running Application on the Android phone.
4. The application sends the text to Watson Assistant. Watson conversation returns the recognized intent "Create" and the entities "large", "black", and "box". The virtual reality application then displays the large black box (which falls from the sky).

Watch the Video

Included components

• IBM Watson Assistant: Create a chatbot with a program that conducts a conversation via auditory or textual methods.
• IBM Watson Speech-to-Text: Converts audio voice into written text.

Featured technologies

• Unity: A cross-platform game engine used to develop video games for PC, consoles, mobile devices and websites.
• Google Cardboard: An inexpensive viewer that mounts an Android phone to view VR apps.

Steps

1. Before you begin
2. Create IBM Cloud services
3. Building and Running

1. Before You Begin

• IBM Cloud Account
• Unity
• Android Unity SDK

2. Create IBM Cloud services

On your local machine:

1. git clone https://github.com/IBM/vr-speech-sandbox-cardboard.git
2. cd vr-speech-sandbox-cardboard

In IBM Cloud:

1. Create a Speech-To-Text service instance.
2. Create an Assistant service instance.
3. Create a Language Translator service instance.
4. Once you see the services in the Dashboard, select the Assistant service you created and click the button.
5. After logging into the Assistant Tool, click the button.
6. Import the Assistant workspace.json file located in your clone of this repository.

3. Building and Running

Note: This has been compiled and tested using Unity 2018.2.0f2 and Watson Unity SDK from the Unity asset Store as of Sept 2, 2018 and tested with the develop branch of the github unity-sdk as of commit 9b439ca57bdb May2.

1. Download the Watson SDK for Unity or perform the following:

git clone https://github.com/watson-developer-cloud/unity-sdk.git

Make sure you are on the develop branch.

1. Open Unity and inside the project launcher select the button.
2. Navigate to where you cloned this repository and open the Creation Sandbox directory.
3. If prompted to upgrade the project to a newer Unity version, do so.
4. Follow these instructions to add the Watson Unity SDK downloaded in step 1 to the project.
5. Follow these instructions to create your Speech To Text, Language Translator, and Watson Assistant services and find their credentials (using IBM Cloud). You can find your workspace ID by selecting the expansion menu on your conversation workspace and selecting View details.
6. In the Unity editor project tab, select Assets->Scenes->Playground and double click to load the scene.
7. In the Hierarchy of your Unity app, go to Playground->Player->StreamingSpeech object and fill in the credentials for Speech to Text, Language Translator, and Conversation, and the Conversation workspace id:

1. Press Play

2. To Build an android .apk file and deploy to your phone, you can File -> Build Settings (Ctrl + Shift +B) and click Build.

3. When prompted you can name your build and then move it to your phone.

4. Alternately, connect the phone via USB and File-> Build and Run (or Ctrl+B).

   Make sure you have enabled USB Debugging:

• Open Settings-> About-> Software Information-> More

• Then tap “Build number” seven times to enable Developer options.

• Go back to Settings menu and now you'll be able to see “Developer options” there.

• Tap it and turn on USB Debugging from the menu on the next screen.

  Once the app is deployed to your phone it will start, but you'll need to set permissions for the app before it will work correctly:

  • Open Settings-> Apps-> SpeechSandboxCardboard-> Permissions and enable Microphone and Storage.

Links

• Demo on YouTube
• Watson Unity SDK

Troubleshooting

• If you see:

Assets/Scripts/CreatableObject.cs(162,43): error CS0117: `GvrController' does not contain a definition for `Gyro'

This is because you have your Build Settings -> Platform set to PC, Mac, or Linux Standalone. Solve this problem by changing the Build Settings -> Platform to Android and clicking Switch Platform.

• If you see:

Assets/unity-sdk/Scripts/Utilities/Credentials.cs(399,51): error CS1061: Type `System.DateTimeOffset' does not contain a definition for `ToUnixTimeSeconds' and no extension method `ToUnixTimeSeconds' of type `System.DateTimeOffset' could be found. Are you missing an assembly reference?

The solution is to go to Build Settings->Player Settings-> Other Settings->Scripting Runtime Version and change to: Experimental(.NET 4.6 Equivalent).

Learn more

• Artificial Intelligence Code Patterns: Enjoyed this Code Pattern? Check out our other AI Code Patterns.
• AI and Data Code Pattern Playlist: Bookmark our playlist with all of our Code Pattern videos
• With Watson: Want to take your Watson app to the next level? Looking to utilize Watson Brand assets? Join the With Watson program to leverage exclusive brand, marketing, and tech resources to amplify and accelerate your Watson embedded commercial solution.

License

This code pattern is licensed under the Apache Software License, Version 2. Separate third party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 (DCO) and the Apache Software License, Version 2.

Apache Software License (ASL) FAQ