Feedback

Give us a feedback after the meetup here. It is anonymous. feedback formular

Tutorial

Our short tutorial https://github.com/falent/GoogleAssistantTutorial

How to build your first Google Assistant Action?

In this short tutorial we will show you how to write a simple Google assistant action and develop it on your local machine. We are going to use Docker to avoid any additional installation steps and ensure that all of us start with the same code in the same environment (less problems for all of us. For those who don't know how Docker works it is a good introduction) Don't worry although we use Docker you will edit your code only locally because we implemented a mount option (shared folder).

When you finish your developing you can easily upload your code to any https node.js server (Heroku, Firebase, AWS, Azure or Openshift). At the end of this tutorial we will show you how to do it with Heroku and Firebase. Of course our solution will also work with any other Https node.js server provider

Table of Contents

1. Requirements

   1.1 Docker installation

   1.2 Google Account

   1.3 Node.js IDE

2. "Interaction Model" Configuration

   2.1 Google Actions

   2.2 Dialogflow

   2.3 Create a new agent

3. Docker environment for our app

   3.1 Docker Containers

   3.2 Project configuration

   3.3 Output

4. Optional Steps

   4.1 Restart Containers

   4.2 Additional docker containers

   4.3 Debugging your app

5. Dialogflow fullfilment

6. How to test your skill?

7. Deployment

   7.1 Heroku

   7.2 Firebase functions

8. References for Google Assistant actions developing

1. Requirements

We will work with Docker. So you need Docker and a Google account.

Check permissions in your google account: google permissions The following have to be activated:

• Web & App Activity
• Device Information
• Voice & Audio Activity

1.1 Docker installation

1.1.1 Linux

https://docs.docker.com/install/linux/docker-ce/ubuntu/

1.1.2 Windows

For Windows we recommend to use Docker Toolbox, because you do not need to activate Hyper-V for it ¹ https://download.docker.com/win/stable/DockerToolbox.exe

See requirements to install Docker Toolbox: https://docs.docker.com/toolbox/toolbox_install_windows/

1.2 Google Account

Please use an existing one or create a new Google account. https://accounts.google.com/signup/v2/webcreateaccount?hl=en-GB&flowName=GlifWebSignIn&flowEntry=SignUp

1.3 Node.js IDE

We will use webstorm https://www.jetbrains.com/webstorm/download/ but you can use any of your favourite JS/Node.js IDE like visual studio or atom or notepad.

2. "Interaction Model" Configuration

We will use Dialogflow and Google Actions to configure our interaction model and publish it for the Google assistant. The interaction model is a voice logic which needs to be defined in Google Actions and Dialogflow.

2.1 Google Actions

You can imagine an action as a part of a voice programm that extends the functionality of the Google Assistant. To start your voice action you need an invocation name. (if you already programmed Alexa, action is like an alexa's skill).

In our case we will create a change rate action and we can start it by saying "OK Google, talk to [your defined invocation name]". After that Google will call your external fulfillment to start a conversation with the user. The fulfilment is based upon your deployed application.

The voice interactions are transformed from audio to text and backward till a conversation between the user and Google assistant reaches its end (for example by saying stop).

Source: https://codelabs.developers.google.com/codelabs/actions-1/img/dbd725edb3a93e79.png

2.1.1 Create a new project

Open Google Actions page https://developers.google.com/actions/ and create a new project. You need a Google account to log in. Click a plus button (add / import project) and after that create a project and give it a name.

At the next page choose "skip".

2.1.2 Define the invocation name

In Actions Invocation define the invocation name for your action. Click on save.

Till now there are no interactions defined between Google assistant and the end user of your action. To create the interaction logic we will go to Dialogflow.

2.2 Dialogflow

Dialogflow (formerly Api.ai, Speaktoit) is a Google-owned developer of human–computer interaction technologies based on natural language conversations. We need it to translate human voice calls into JSON objects that can be consumed by our backend code and backwards to tranlate the responses from our backend to voice phrases. You define how all this works within a Dialogflow agent.

2.2.1 How does Dialogflow work?

1. A user starts voice interaction or text interaction saying/texting OK Google, talk to [your definded invocation name]

2. Google calls Dialogflow using google actions and looks for a defined agent for the invocation name.

3. Dialoglow checks if user intention is correponding to one of the training phrases. If Dialogflow detects a suitable intent there are two ways to call the logic behind the intent:

   a) If the whole logic is defined in Dialogflow, Dialogflow will send a response to the user

   or

   b) Dialogflow calls your external code and returns an answer processed from your code to Google Actions which returns it to your end user

In the next step open the Dialoflow interface: https://console.dialogflow.com

2.3 Create a new agent

Create a new agent where you configure the interaction model for your action. ** Please keep in mind to import your created project (Google PROJECT) from Google actions to DialogFlow**. The name how you call your agent in dialogflow is irrelevant

https://console.dialogflow.com/api-client/#/newAgent

2.3.1 Add your first intent

Intents are anticipated user intentions about what users might want to talk with the voice assistant in an action. For an exchange rate action possible intents would be questions about the exchange rate from a source currency like Euro to a target currency like Dollar. Intents are triggered by predefined training phrases and key words from the user. Possible questions about exchange rates could be "What is 50 Euro in Dollar?". To see how its configured click on Intentions in Dialogflow. Select the default welcome intent. Here you have listed several training phrases that invoke the default welcome intent. Under responses you see what the Google assistant could answer to the user utterances that triggered the default welcome intent.

To add a new intent click on the plus button. Then define its name, training phrases and entities. Entities are variables for common key words the user might say. For example in our exchange rate voice app these are the currencies. To see how it works, add a new intent, name it getExChangeRateIntent, add a training phrase and type a currency name for example Euro in the training phrase field. Click enter and you see the currency will be textmarked and is resolved to the entity currency. Now every currency the user might say will be resolved to the currency type. Type in Change 50 Euro to Dollar. Now edit the parameter names to adress the currencies as fields in your app. Set 50 as @sys.number entity, name it amount then name the first currency currencyBase and the second currencyTarget.

2.3.2 Add traning phrases

You can define responses to the traning phrases in an intent in Dialogflow. But we use our own code so we will define the responses in our web application code. For that we need to enable one option on the buttom of our intent. Swipe „Enable webhook call for this intent“ and click the „save“ button.

2.3.3 Add Follow-Up Intents

Add a Follow-Up Intent. This will trigger when the user answers Yes to the Question "Do you like to change more?" in the assitants response of the getExchangeRateIntent. To do so, click on Intents, hover over the getExchangeRateIntent and click on follow-up intent, choose yes. Create a getExchangeRateIntent - yes Intent. Add just one response. "What do you want to change?". Do not enable webhook call for this intent. This intent will be resolved only in dialogflow. Click save. Add a second Follow-Up Intent for the getExchangeRateIntent. This will be triggered, when the User rejects the assitants question. It will mark the end of the conversation and Google Assistant will left the conversation. Add a response like "Bye, till next time". Like the follow-up yes intent it will be resolved only in dialogflow, so do not enable webhook call for this intent.

2.3.4 Add the starting point of the conversation

Define a starting point of the conversation. Create a new intent and name it welcomeIntent. Add Google Assitant Welcome to the Events. This defines our welcomeIntent as the starting point of the conversation. Under Fulfilment enable webhook call for this intent. Every content of this intent is defined in our web application. So we do not need training phrases or responses in dialogflow. Do not forget to enable webhook call for this intent.

2.3.5

Below you see a dialog sample for our action:

3. Docker environment for our app

If you use Windows install Docker Tools on your machine or if you use docker under linux Docker CE (Community Edition) to test your Docker installation execute the following command:

linux:

$ sudo docker run hello-world

windows:

docker run hello-world

If you don't see his output in windows:

Hello from Docker! This message shows that your installation appears to be working correctly.

Please be sure that you started "Docker quickstart terminal". You can execute all Docker commands in a windows cmd terminal.

3.1 Docker Containers

Please do all steps in the order they are described below :) We will create 2 containers which will be connected to each other in one Docker virtual network.

Our simplified architecture looks like that:

Ngrok is a gateway that communicates with Google Assistants requests/responses via https. Our node.js action (skill) app is deployed in a Docker container and we share app code from our localhost with it. Thus we can change the code locally and the deployed app reacts instantly to the changes thanks to the npm package nodemon.

3.2 Project configuration

3.2.1 Linux

Open your terminal and execute the following commands:

1. Clone our repository from github $ git clone https://github.com/falent/googleHomeAssistantExpressNodeJS.git ~/Desktop/Template/Google_Assistant_universal_skill_template

2. Create a new Docker network in a terminal tab: $ sudo docker network create myNetwork

3. Run the ngrok Docker container in your terminal and do not close this tab! $ sudo docker run --rm -it --network myNetwork wernight/ngrok ngrok http myAssistant:5000

4. Open a new tab and run our Google Assistant Docker container. The Docker image is downloaded from my Docker Hub account falent: $ sudo docker run -v ~/Desktop/Template/Google_Assistant_universal_skill_template:/skill -it --rm --network myNetwork --name myAssistant falent/google_home_assistant_express_node_js_server

3.2.2 Windows

You can use a batch script in your command line to do all steps automatically.

If you put this curl command to your cmd terminal, batch script will execute all commands that you could do manually ;)

curl https://raw.githubusercontent.com/falent/googleHomeAssistantExpressNodeJS/master/scripts_for_meetup/automatedInstallationWindows.bat > automatedInstallationWindows.bat && automatedInstallationWindows.bat

Please keep in mind that you should execute this script only once. If you close the window or you want to reload npm modules please use these commands

You can see the content of the batch script file here. I put comments to explain what they are doing. https://github.com/falent/googleHomeAssistantExpressNodeJS/blob/master/scripts_for_meetup/automatedInstallationWindows.bat

OR

We encourage you to do all steps by yourself in a windows cmd console because you can understand better what we are going to do :)

1. Before we start you need to ensure that Git is properly configured to handle line endings. git config --global core.autocrlf true

2. Clone our repository. Your repository will be saved automatically in your Documents in the directory googleHomeAssistantExpressNodeJS. git clone https://github.com/falent/googleHomeAssistantExpressNodeJS.git C:\Users\%username%\Documents\googleHomeAssistantExpressNodeJS

3. Create a new Docker network: docker network create myNetwork

4. Run the ngrok Docker container in your cmd terminal and do not close this tab! docker run --rm -it --network myNetwork wernight/ngrok ngrok http myAssistant:5000

5. Open a new cmd window and run an Google Assistant Docker container in your created network. docker run -v //c/Users/%username%/Documents/googleHomeAssistantExpressNodeJS:/skill -it --rm --network myNetwork --name myAssistant falent/google_home_assistant_express_node_js_server

To build the container named myAssistant a docker image is downloaded from the Docker Hub account falent/google_home_assistant_express_node_js_server.

(https://hub.docker.com/r/falent/google_home_assistant_express_node_js_server/)

The Docker command -v creates a shared directory, the flag -v means: Create a directory in the docker container and share my local directory googleHomeAssistantExpressNodeJS with it. With a shared directory we can edit our files locally and share it automatically with our directory in the docker container.

3.3 Output

You will see the following output if your created myAssistant container works properly.

4 OPTIONAL STEPS

In this section you will find optional steps if you wish to restart your Docker or rebuild everything. You can go directly to the 5th step.

4.1 Restart (in case of new module installations or if you want to start your container app)

Our solution is based on nodemon which is kind of a watcher and it reloads your skill everytime you made code changes. You save a lot of time because of that ;) However it can happen that you wish to add more npm modules https://www.npmjs.com/ to your skill. In that case you need to restart your container. Npm modules are installed only at the start of your container. To restart your container type the following in a cmd terminal:

4.1.1 Linux

$ sudo docker run -v ~/Desktop/Template/Google_Assistant_universal_skill_template:/skill -it --rm --network myNetwork --name myAssistant falent/google_home_assistant_express_node_js_server

4.1.2 Windows

docker run -v //c/Users/%username%/Documents/googleHomeAssistantExpressNodeJS:/skill -it --rm --network myNetwork --name myAssistant falent/google_home_assistant_express_node_js_server

4.2 Additional docker containers

Imagine you would like to add a database to your Docker environment or any microservice app. With our solution it is very simple! Just start any database Docker container and place it in our Docker network. For example:

4.2.1 MongoDB

You only need to start a new container:

sudo docker run -it --name mongo_database --network myNetwork bitnami/mongodb:latest

Your database is listening on port 27017. The name of the container will be "mongo_database". To request your database from node.js you need to call this address: mongodb://mongo_database:27017

4.2.2 MySQL

You only need to start a new container:

sudo docker run -it --name=mysql1 --network myNetwork -d mysql/mysql-server:5.7

check what generated password is:

docker logs mysql1 2>&1

and login to shell to create a new table

docker exec -it mysql1 mysql -uroot -p

4.3 Debugging

You can dubug our node.js app.

4.3.2 Linux

Execute in terminal:

sudo docker run -v ~/Desktop/Template/Google_Assistant_universal_skill_template:/skill -it -p 9229:9229 -w /skill/functions --rm --network myNetwork --name myAssistant falent/google_home_assistant_express_node_js_server /bin/sh -c 'npm i && nodemon --inspect-brk=0.0.0.0:9229 start.js'

and connect your IDE with debug mode with localhost port 9229

4.3.1 Windows

In cmd terminal write:

docker-machine ip default

You will get an ip address. In my case it was: 192.168.99.101

open your docker container using:

docker run -v //c/Users/%username%/Documents/googleHomeAssistantExpressNodeJS:/skill -it -p 9229:9229 -w /skill/functions --rm --network myNetwork --name myAssistant falent/google_home_assistant_express_node_js_server /bin/sh -c "npm i && nodemon --inspect-brk=0.0.0.0:9229 start.js"

you should see that debugger is listening

Open for example WebStorm or any JS IDE supporting debug mode and connect to the debugger

In Webstorm click in the buttom toolbar Run-->Edit Configurations Please choose from templates: "Attach to Node.js/Chrome" In the host add ip address of your docker-machine. In my case 192.168.99.101

and click apply and + button in the left button corner

Now if you click Run-->You can run your defined debug process

You can add your first breakpoint clicking to a choosen line and start conversation from DialogFlow. You can see your debugged code in the WebStorm console.

5. Dialogflow fullfilment

We need to add a ngrok address so that our local server can communicate with Dialogflow

5.1 Ngrok address

Please copy https address from ngrok Docker.

5.2 Dialogflow endpoint

Go to Fulfilment and enable Webhook in Dialogflow, put your copied https address and click „save“.

At the end go to Integrations in Dialogflow and choose Google Assistant Integrations. Set up explicit invocation for the welcomeIntent and add "implicit invocation" for your created getExChangeRateIntent.

Now you can test your app clicking on the buttom "Test". You will be taken to Google Actions simulator.

6. How to test your skill?

In the Test Simulator of Google Actions just click on your suggested input. It will be "Talk to {your defined Google Actions invocation name}"

Your action will answer: Welcome to Change Rate. What do you like to change? Add the predefined training phrase Change 50 Euro to Dollar for the getChangeRateIntent in the text field.

If everything works, you will get the following answer to your request. (amount is up to actual change rate ;)) You will get 57.105 USD

In your ngrok container in your ngrok container you will see a POST request with 200 OK Code:

and in your Google assistant container you will see first logs

Beside the test simulator you can test your action with your private Google home or Smartphone Google assistant app. You do not need further installation or activation. Just test it by saying: Ok Google, talk to {your defined in Google actions skill invocation name}

7. Instructions for a quick deployment

We describe in this section how you can deploy your app in Heroku and Firebase functions. To deploy you can use a new instance of our image which will install Heroku and Firebase clients.

sudo docker run -v ~/Desktop/Template/Google_Assistant_universal_skill_template:/skill -itd --network myNetwork --name myAssistantDeployment falent/google_home_assistant_express_node_js_server:deploying

The following commands for Heroku and Firebase describe how to deploy your programmed action.

7.1 Heroku

Sign up for Heroku (it's for free).

7.1.1 Linux

1. Open shell in your Docker container

  $ sudo docker exec -it  myAssistantDeployment /bin/sh

2. Login to the client with your account

  $ heroku login --interactive

3. Create your app in Heroku server. The app name will be showed in a pink color. Save it in an editor of your choice to use it later

  $ heroku apps:create --region eu

4. Set npm config to false

  $ heroku config:set NPM_CONFIG_PRODUCTION=false -a <your app name>

5. In your myAssistantDeployment container go to functions catalog

  $ cd /skill/functions

6. Clear all git repositories files

  $ rm -rf .git

7. Init a clean git repository

  $ git init && git config --global user.email "you@example.com" && git config --global user.name "Your Name"

8. Add and commit all files

  $ touch readme.md && git add . && git commit -m "my first commit"

9. Link your app to git

  $ heroku git:remote -a <your app name>

10. Push your files to Heroku

  $ git push heroku master

After pushing you will get your Heroku app https address https://yourAppName.herokuapp.com/ , for example I got https://secret-reef-17554.herokuapp.com/. Please copy it and paste it to the endpoint

11. Start the webapplication in heroku

  $ heroku ps:scale web=1

12. You can see logs with

  $ heroku logs --tail

7.2 Firebase as functions

1. Open the shell in your Docker container

  $ sudo docker exec -it  myAssistantDeployment /bin/sh

2. Login to the client with your account

  $ firebase login --no-localhost

3. Please copy URL from the terminal and log into your google account where you want to host your app. After login paste the authorization code.

4. Run Firebase init functions

  $ firebase init functions

5. Choose Javascript and answer to all questions about overwriting files n but y for npm install

6. Deploy your functions

  $ firebase deploy --only functions

7. Go to https://console.firebase.google.com/project/[yourAppName]/functions/list to see your app address. In my case it was https://us-central1-fir-5c548.cloudfunctions.net/myFirstAction

Please keep in mind that if you call an extern API you need to activate Blaze Plan in some condition it will remain for free https://firebase.google.com/pricing/

8. References for Google Assistant actions developing

https://www.npmjs.com/ - npm is the package manager for JavaScript and the world’s largest software registry

https://codelabs.developers.google.com/?cat=Assistant – Google Labs how to build actions (skills) in Node.js for google assistant

https://github.com/falent/googleHomeAssistantExpressNodeJS - Our tutorial for google assistant

https://github.com/falent/GoogleAssistantTutorial - our tutorial and code snippets

https://www.tutorialspoint.com/nodejs/ - Node.js Tutorial

https://dialogflow.com/docs/getting-started - Official DialogFlow documentation

_{1. Hyper-V is not compatible with virtual box or vmware. So in case you use VM in these it prevents you from switching Hyper-V on and off.}