Skip to content

Commit

Permalink
use new tjbot library
Browse files Browse the repository at this point in the history
  • Loading branch information
Damian Cummins committed Apr 1, 2017
0 parents commit 30a0c39
Show file tree
Hide file tree
Showing 16 changed files with 2,195 additions and 0 deletions.
53 changes: 53 additions & 0 deletions .gitignore
@@ -0,0 +1,53 @@
# Logs
logs
*.log
npm-debug.log*

# Runtime data
pids
*.pid
*.seed
*.pid.lock

# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov

# Coverage directory used by tools like istanbul
coverage

# nyc test coverage
.nyc_output

# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
.grunt

# node-waf configuration
.lock-wscript

# Compiled binary addons (http://nodejs.org/api/addons.html)
build/Release

# Dependency directories
node_modules
jspm_packages

# Optional npm cache directory
.npm

# Optional eslint cache
.eslintcache

# Optional REPL history
.node_repl_history

# Output of 'npm pack'
*.tgz

# .DS_Store files
.DS_Store

#output.wav
output.wav

#config
config.js
119 changes: 119 additions & 0 deletions README.md
@@ -0,0 +1,119 @@
# Tell the Time

> Build a robot that can tell the time in different cities with [Watson](https://www.ibm.com/watson/developercloud/conversation.html)
### "Watson, what time is it in Berlin?"

This module provides Node.js code to get your Raspberry Pi to tell the time in any city you choose to tell Watson about. It uses [Watson Speech to Text](https://www.ibm.com/watson/developercloud/speech-to-text.html) to parse audio from the microphone, uses [Watson Conversation](https://www.ibm.com/watson/developercloud/conversation.html) to generate a response, and uses [Watson Text to Speech](https://www.ibm.com/watson/developercloud/text-to-speech.html) to "read" out this response!

**This will only run on the Raspberry Pi.**


## How It Works
- Listens for voice commands
- Sends audio from the microphone to the Watson Speech to Text Service - STT to transcribe [Watson Speech to Text](https://www.ibm.com/watson/developercloud/speech-to-text.html)
- Parses the text looking for the attention word
- Once the attention word is recognized, the text is sent to [Watson Conversation](https://www.ibm.com/watson/developercloud/conversation.html) to generate the response.
- The response is sent to [Watson Text to Speech](https://www.ibm.com/watson/developercloud/text-to-speech.html) to generate the audio file.
- The robot speaks the response via the Alsa audio playback tools

## Hardware
Check out [this instructable] (http://www.instructables.com/id/Build-a-Talking-Robot-With-Watson-and-Raspberry-Pi/) to prepare your system. You will need a Raspberry Pi 3, Microphone, Speaker, and [the TJBot cardboard](https://ibmtjbot.github.io/#gettj).

## Build
> We recommend starting with our [step by step instructions] (http://www.instructables.com/id/Build-a-Talking-Robot-With-Watson-and-Raspberry-Pi/) to build this recipe.
Get the sample code and go to the application folder. Please see this [instruction on how to clone](https://help.github.com/articles/cloning-a-repository/) a repository.


Install ALSA tools (required for recording audio on Raspberry Pi)

sudo apt-get install alsa-base alsa-utils

Install Dependencies

npm install

Set the audio output to your audio jack. For more audio channels, check the [config guide. ](https://www.raspberrypi.org/documentation/configuration/audio-config.md)

amixer cset numid=3 1
// This sets the audio output to option 1 which is your Pi's Audio Jack. Option 0 = Auto, Option 2 = HDMI. An alternative is to type sudo raspi-config and change the audio to 3.5mm audio jack.

Update the Config file with your Bluemix credentials for all three Watson services.

edit config.js
enter your watson usernames, passwords and versions.

## Creating a Conversation Flow
You need to train your robot with what to say and when to say it. For that, we use [Watson Conversation] (https://www.ibm.com/watson/developercloud/conversation.html). Open a browser and go to [IBM Watson Conversation link](http://www.ibmwatsonconversation.com)
From the top right corner, select the name of your conversation service and click 'create' to create a new workspace for your robot. You can create intents and dialogs there. [Here](http://www.instructables.com/id/Build-a-Talking-Robot-With-Watson-and-Raspberry-Pi/#step6) is a step-by-step instructions to create a conversation flow.

### Creating intents

An intent for telling the time might look like this:

![tellTheTime intent](https://github.com/DamianCummins/tell_the_time/blob/master/images/tellTheTimeIntent.PNG)


### Creating dialogs

A Dialog node for telling the time for different timezones might look like this:

![tellTheTime dialog](https://github.com/DamianCummins/tell_the_time/blob/master/images/tellTheTimeDialog.PNG)

The response should contain a placeholder (`todays_date`) that the node application can replace with a javascript Date. When a specific city entity is found, the response for each city should return the timezone offset in the output context:

![timezone in offset output](https://github.com/DamianCummins/tell_the_time/blob/master/images/tellTheTimeDialogDetails.PNG)


## Running

Start the application

node app.js

Then you should be able to speak to the microphone.
The robot gets better with training. You can go to your [Watson conversation module](http://www.ibmwatsonconversation.com) to train the robot with more intents and responses.

## Monitoring

The app serves up a Monitoring web interface locally on port `3000`. You should be able to point to this from another device on the same network as your Rasberry Pi. The Monitoring UI allows you to stop and resume listening on TJBot's microphone. You can also tell whether the TJBot module has stopped due to an unexpected error.

![tellTheTime monitor app](https://github.com/DamianCummins/tell_the_time/blob/master/images/monitorApp.gif)

The logs show the REST calls made from the Monitoring UI to control TJBot:

```
TJBot is listening, you may speak now.
TJBot monitor app listening on port 3000!
Fri, 24 Mar 2017 22:38:53 GMT LOG Router request GET /tjbot
Fri, 24 Mar 2017 22:38:54 GMT LOG Router request GET /tjbot/status
Fri, 24 Mar 2017 22:39:00 GMT LOG Router request POST /tjbot/pause
Fri, 24 Mar 2017 22:39:19 GMT LOG Router request POST /tjbot/resume
TJBot is listening, you may speak now.
```

## Customization
The attention word is the word you say to get the attention of the robot.
The default attention word is set to 'Watson' but you can change it from config.js. Some words are easier for the robot to recognize. If decided to change the attention word, experiment with multiple words and pick the one that is easier for the robot to recognize.

The default voice of TJBot is set to a male voice (`en-US_MichaelVoice`) but you can change it from config.js. Two female voices are available for TJBot (`en-US_AllisonVoice` and `en-US_LisaVoice`).

// The attention word to wake up the robot.
exports.attentionWord ='watson';

// You can change the voice of the robot to your favorite voice.
exports.voice = 'en-US_MichaelVoice';
// Some of the available options are:
// en-US_AllisonVoice
// en-US_LisaVoice
// en-US_MichaelVoice (the default)

# Dependencies List

- Watson Developer Cloud - [Watson Speech to Text](https://www.ibm.com/watson/developercloud/speech-to-text.html), [Watson Conversation](https://www.ibm.com/watson/developercloud/conversation.html), and [Watson Text to Speech](https://www.ibm.com/watson/developercloud/text-to-speech.html).
- mic npm package : for reading audio input


## Contributing
See [CONTRIBUTING.md](../../CONTRIBUTING.md).
71 changes: 71 additions & 0 deletions app.js
@@ -0,0 +1,71 @@
"use strict";

var express = require("express");
var app = express();
var bodyParser = require("body-parser");
var path = require("path");

var tellTheTime = require("./tellTheTime.js")();

var port = process.env.VCAP_APP_PORT || process.env.PORT || 3000;

var router = express.Router();

tellTheTime.start();

app.use(bodyParser.urlencoded({"extended": true}));
app.use(bodyParser.json());

router.use(function(req, res, next) {
console.log(new Date().toUTCString() + " LOG Router request " + req.method + " " + req.originalUrl);
next();
});

router.get("/", function(req, res) {
res.sendFile(path.join(__dirname + "/index.html"));
});

router.route("/status")
.get(function(req, res) {
tellTheTime.isMicStopped(function(status) {
if (status) {
res.json({'message':'SNOOZING'});
} else {
res.json({'message':'AWAKE'});
}
});
});


router.route("/pause")
.post(function(req, res) {
tellTheTime.pause(function(err) {
if (!err) {
res.json({'message':'SNOOZING'});
}
});
});

router.route("/resume")
.post(function(req, res) {
tellTheTime.start();
tellTheTime.isMicStopped(function(status) {
if (status) {
res.json({'message':'SNOOZING'});
} else {
res.json({'message':'AWAKE'});
}
});
});

app.use("/tjbot", router);

app.use(function(req, res, next) {
res.status(404);
console.error("ERROR: Page not found");
res.json({"code": 404, "message": "Page not found."});
});

app.listen(port, function() {
console.log("TJBot monitor app listening on port "+port+"!");
});
Binary file added audio/notUnderstood.wav
Binary file not shown.
Binary file added audio/understood.wav
Binary file not shown.
79 changes: 79 additions & 0 deletions csvToWorkspaceJson.js
@@ -0,0 +1,79 @@
"use strict";
var fs = require('fs');
var csv = require('fast-csv');

var fs = require('fs');
var workspace = JSON.parse(fs.readFileSync('tellTheTime_workspace.json', 'utf8'));

var stream = fs.createReadStream("timezones.csv");

var entityValues = [];

var dialogNodeCounter = 0;

var dialogs = [];



var csvStream = csv()
.on("data", function(data){
var zone = data[0];
var zoneTokens = zone.split("/");
var city = zoneTokens[zoneTokens.length -1].replace("_", " ");

var entityValue = {
value: city,
synonyms: [],
metadata: null
};

entityValues.push(entityValue);

if (city.indexOf(" ") > -1) {
city = "(" + city + ")";
}
var offset = parseFloat(data[1]);
var dst = data[2];
var dialogResponse = {
"type":"response_condition",
"go_to":null,
"output":{
"text":{
"values":[
"The time in @city is todays_date"
],
"selection_policy":"sequential"
},
"context":{
"timezoneOffset": offset,
"dst": dst
}
},
"parent":"Tell the Time",
"context":null,
"metadata":null,
"conditions":" @city:"+city,
"description":null,
"dialog_node":"dialog_node_" + dialogNodeCounter,
"previous_sibling":"dialog_node_" + (dialogNodeCounter - 1)
};
if(dialogNodeCounter === 0) {
dialogResponse["previous_sibling"] = "node_3_1490294223086";
}
dialogNodeCounter++;
dialogs.push(dialogResponse);

console.log(data);
console.log(JSON.stringify(entityValue));
console.log(JSON.stringify(dialogResponse));

})
.on("end", function(){
console.log("done");
console.log(dialogs.length);
console.log(entityValues.length);
workspace.entities[0].values = workspace.entities[0].values.concat(entityValues);
workspace.dialog_nodes = workspace.dialog_nodes.concat(dialogs);
fs.writeFileSync('tellTheTime_workspace_new.json', JSON.stringify(workspace, null, 3));
});
stream.pipe(csvStream);
Binary file added images/monitorApp.gif
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/tellTheTimeDialog.PNG
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/tellTheTimeDialogDetails.PNG
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/tellTheTimeIntent.PNG
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
55 changes: 55 additions & 0 deletions index.html
@@ -0,0 +1,55 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>TJBot Monitor</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css">
<style>
body { padding-top:50px; }
</style>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
<script>
function refresh() {
$.getJSON('/tjbot/status',
function(response) {
$('#status').empty().append(response.message);
});
};

function pause() {
// Send the request
$.post('/tjbot/pause', {}, function(response) {
$('#status').empty().append(response.message);
alert("TJBot is snoozing... Zzz");
}, 'json');
};

function resume() {
// Send the request
$.post('/tjbot/resume', {}, function(response) {
$('#status').empty().append(response.message);
alert("TJBot woke up!");
}, 'json');
};

refresh();
</script>
</head>
<body>

<div class="container">
<div class="jumbotron">
<h1>TJBot Monitor</h1>
<p>Current status of TJBot:</p><p id="status"></p>
</div>
<div class="container text-center col-sm-12">
<div type="button" class="btn btn-block btn-info" onclick="refresh()">Refresh</div><br/>
<div type="button" class="btn btn-block btn-warning" onclick="pause()">Let TJBot snooze</div><br/>
<div type="button" class="btn btn-block btn-success" onclick="resume()">Wake TJBot up</div>
</div>
</div>

</body>
</html>

0 comments on commit 30a0c39

Please sign in to comment.