Talk-to-the-Bot

A WebGL demo that combines virtual reality, speech recognition and synthetic voice from the web browser.

It shows a panorama picture that contains an interactive 3D model that allows the user to interact with voice commands and a VR interaction pointer. The app is configured to work in English and Spanish detecting the browser language automatically.

This demo currently works 100% only in Google Chrome and Google Chrome for Android. You'll find errors and issues with speech recognition in other browsers, and synthetic voice only works in Chrome as an experimental Javascript API.

I showed this demo in one of my three.js workshops in Spain ( IT STARTS ON 31:40 !!!! )

Install and run

• Download or clone the source files and run it in a local web server using Google Chrome or Google Chrome for Android.
  
• When it runs from Android, the app will setup automatically for VR mode. Then tap the screen and the app will setup the browser window in fullscreen mode. You can use a Google Cardboard or any other third-party VR glasses to enjoy the experience. The screen will split in stereoscopic mode and you'll notice that the camera moves using the device's accelerometer.
• Use the VR pointer ( using the mouse in desktop ) to interact with the small floating 3D robot. It will show you small text boxes at her side and the sythetic voice API will reproduce the text in Google Chrome. The speech sequence is started and stopped by pointing in or out of the robot.
• When the robot asks for your name, Chrome will ask you about allow the microphone for the speech recognition. You must allow it in order to talk to the app.
• Then... say your name! The speech commands are configured to say it in different ways ( "I am..." or "My name is..." ), and the synthetic voice will salute you using your given name.
• Finally, if you say "Thanks" or "Thank you" after the salute, the robot will answer again.

Practical cases

This is a very simple implementation of a sort of "VR bot". It can be taken to the next level using conversational platforms such as API.ai or any other information API using AJAX requests in Javascript. Replacing the synthetic voice with recorded audio would increase the quality of the final product.

Enabling Helper Mode

Just modify this line in js/main.js:

enableHelperMode(false);

If you set it to true, it changes the three.js Perspective Camera properties and it shows a wireframed box that represents a "reference area" to place a 3D object in order to make it "credible" combined with the panorama sphere. It also shows a transparent box overlapping with the robot model. That's because the pointer of the Reticulum plugin only works pointing to three.js basic geometries. So any imported 3D object in the scene needs to be "wrapped" within an invisible box geometry that interacts with the pointer.

Implementation

A list of the libraries and assets used for this demo:

• WebGL implementation : three.js r83 by Mr. Doob
• Stereoscopic view : StereoEffect.js for three.js by Mr. Doob
• Device Orientation Controls : DeviceOrientationControls.js for three.js by richt & WestLangley
• VR Pointer : Reticulum 2.0.2 by Skezo
• Speech recognition : Annyang 2.4.0 by Tal Ater
• SpeechSynthesis API in MDN
• three.js OBJLoader.js
• THREEX dynamic texture extension by Jerome Etienne
• 3D Model: "Drone" by Renafox