Nuance Mix Demo Client

Develop and Troubleshoot Nuance Mix AI Powered Chat Bots locally and on Microsoft Azure.

Table of contents

• Overview
• About the Client
• Setup
• Functionality
• Pre-Requisites
• Quick Start 🚀
• Getting Started
• Conventions for Rich UI
• Conventions for Link Handling
• Data Access Integrations
• Themes/Branding
• Publishing to Azure
• License

Overview

This demo client offers a sample integration to Nuance Mix's Conversational AI Runtime Services, specifically

• Dialog (DLGaaS)
• Speech Recognition (ASRaaS)
• Natural Language Understanding (NLUaaS)
• Text to Speech (TTSaaS) and the
• Runtime Event Log API

The client uses the HTTP/1.1 transcoded, or WebSocket transport version of the APIs (rather than the native HTTP/2 gRPC versions).

The prime purpose of this tool is to assist in Bot development and troubleshooting, including offering controlled hosted availability using Azure StaticWebApps.

Azure Functions with Core Tools are used throughout the development and deployment lifecycle.

For more information on how to leverage the client across various environments, see networking options, and be sure to secure your functions. Monitoring is made available through Application Insights.

Disclaimer: This sample client is not intended to illustrate a production-ready implementation; it is intended to aid development, demonstration, troubleshooting, and other such use cases.

About the Client

• Gatsby, React and React-Bootstrap for the Client/Frontend
• Azure Functions Python for the API
  • Acts as a Proxy for Nuance Services, with Data Integrations for Mix.dialog/DLGaaS

Setup

Please see the Exchanging data section in the Mix documentation. It is crucial to understand the concepts as you design your Mix projects and leverage this client -- it acts more like a gateway.

Functionality

• Bot engagements using Nuance Mix's DLGaaS HTTP/1.1 or WebSockets API (web-gRPC not HTTP/2 gRPC)
  • Audio or Text Input, Audio or Text Output
  • PRESENTATION LAYER: Rich UI through conventions
  • DATA: Functions for development through Client Fetch, and when deployed, enables External Fetch usage
  • DATA: Client Fetch handler for Location Data supplied in userData
  • LOGS: Log Events, Filtering and Timeline Visualization
  • CHANNEL SIMULATION: Visual VA, IVR, SmartSpeaker, SMS (audio in/out, text in/out, or any permutation)
  • CHAT: Standalone mode for launching without logs and for data collection (/chat/ path)
  • THEMES: Supports branding with use of logo, assets like fonts and stylesheets
• Speech Recognition using ASRaaS WebSockets API (web-gRPC not HTTP/2 gRPC)
• Natural Language Understanding using NLUaaS HTTP/1.1 API (not HTTP/2 gRPC)
• Text to Speech Synthesis using TTSaaS HTTP/1.1 API (not HTTP/2 gRPC)
• Event data using the Log Events HTTP/1.1 API

Intended Local Use

Use Data Access nodes with the client_fetch configuration and write the integration layer locally with the intention of separate hosted Functions (through external_fetch). These Functions would be referenced and configured within Mix.dialog and Mix.dashboard respectively.

Intended Hosted Use

The following illustrates a scenario where the client is deployed to Azure, and Data Access nodes have been configured to use external_fetch within Mix.

This simplifies the client handling, deferring to the Functions themselves, and offers lifecycle controls within Mix.

⚠️ There may be scenarios where client_fetch is appropriate; this has been set up such that dlgaas.js will invoke local ClientFetchHandlers.

Pre-Requisites

• Git
• Brew (MacOS/Linux) or Chocolatey (Windows)
• OpenSSL and Mkcert
• If using Docker..
  • Docker Engine 20.10.0+
• If using a Native Host..
  • Node.js 18+ (Client Toolchain)
  • Python 3.9+ (Functions)
  • Azure Functions Core Tools

Quick Start 🚀

git clone git@github.com:nuance-communications/mix-demo-client-azstaticwebapps.git

Pre-requisite for either Docker or Native Host based:

make certs-setup

Assumes running the application using Docker:

make launch

If using Native Host, run two processes:

make native-run-api-secure

make native-run-app-secure

Getting Started

Makefile

Various opertions are described in a Makefile.

make help

CERTS:  certs-(prep|setup)
*LAUNCH:  launch
DOCKER:  containers-(build|run|restart|stop|status|logs|clean)
NATIVE.BUILD:  native-build-(app|api)
NATIVE.RUN:  native-run-(app|api)-(secure|insecure)
NATIVE.CLEANUP:  native-clean
DATA.ACCESS:  new-data-access-endpoint

There are two main modes of operation intended: docker and native host. Launch defaults to docker.

Certificate Related

make certs-prep

Installs the pre-requisities for certificate creation, namely openssl and mkcert. A package manager is used here: brew or choco depending on your platform.

make certs-setup

Creates certificates and stores password as needed for local use.

Quick Start

make launch

Primary command, sets up certificates and runs with docker compose.

Docker Host

make containers-build

Builds the containers needed to run the app. Reflects the full static web app, including the APIs.

make containers-run

Runs the containers for the app. Reflects the full static web app, including the APIs.

make containers-restart

Restarts the containers of the app.

make containers-stop

Stops the containers running.

make containers-status

Provides status for the containers.

make containers-logs

Follow the logs for the running containers.

make containers-clean

Removes all the images for the app.

Native Host

make native-build-app

Builds the client application package and brings in associated dependencies.

make native-build-api

Builds the functions package and sets up a virtual environment.

make native-run-app-secure

Runs the native application securely (leveraging certs).

make native-run-api-secure

Runs the native functions securely (leveraging certs).

make native-run-app-insecure

Runs the native application insecurely. Must update local.settings.json accordingly. Not default mode.

make native-run-api-insecure

Runs the native functions insecurely. Must update local.settings.json accordingly. Not default mode.

make native-clean

Cleans all application and functions related resources.

Data Access Endpoint Related

make new-data-access-endpoint

Helper to bootstrap a data access endpoint, leveraging a template.

Setting Up Certificates

If you do not have mkcert or openssl installed:

make certs-prep

To set up the necessary certificates:

make certs-setup

Option 1: Run with Docker

make containers-run

Option 2: Run Natively

Run the two processes.

App:

make native-run-api-secure

API:

make native-run-api-secure

Create a New Data Access Endpoint

make new-data-access-endpoint

See the official Azure Functions Reference - Python for more information.

Configuration

If using Docker, update the values in .env.

If running Natively, update the following environment variables to override the default values:

export oauth_server_url="https://auth.crt.nuance.com/oauth2"
export base_url_dlgaas="https://dlg.api.nuance.com/dlg/v1"
export base_url_nluaas="https://nlu.api.nuance.com/nlu/v1"
export base_url_ttsaas="https://tts.api.nuance.com/api/v1"
export base_url_logapi="https://log.api.nuance.com"
export oauth_scope="dlg nlu tts log"

When deployed, these can be set per deployment environment in the Azure Portal for the Static Web App.

local.settings.json

See resources/local.settings.json for the base file.

The default setting enable CORS for the Functions, and sets a worker count as desired. Be mindful of the http vs https distinction. By default, the application is set up to use https.

Email capabilities (Optional)

This sample offers a stub integration with SendGrid for email capabilities.

This can be used if creating a Data Access node with the name Server_Send_Email, which in turn will call the email-api-send locally.

To configure this integration, provide the following environment variables:

export sendgrid_api_key="<REPLACE_ME>"
export sendgrid_from_email="<REPLACE_ME>"
export sendgrid_custom_token="<REPLACE_ME>"

Note: Override values as needed in .env when using docker-compose ⚠️ The custom token must be provided in the Mix project named SENDGRID_TOKEN. This is to thwart unintentional usage.

Conventions for Rich UI

This client employs certain conventions within Mix.dialog to offer special types of rendering in the presentation layer.

To take advantage of these, navigate to:

• QA node > Node Properties
  • User Input > Optional > Interactivity

Provide the following strings in the "type" field.

Special Input Types for QA Nodes:

• email
• phone
• currency
• date

To provide hints (aka placeholders), the sendData must include a variable with the type followed by 'Hint', ie. 'emailHint', 'phoneHint' and so on.

For Interactivity (aka selectables):

• carousel
• buttons
• colorpicker (description should contain the HEX value)

Rich Text Modality

Note: when using Rich Text markup in messages, if elements contains classes from Bootstrap they will be rendered accordingly in this client. See chat.js to see the allowed DOM elements and attributes.

Stub Images for Carousels with Selectables

Images can be rendered when using selectables and the carousel type. Edit STUB_SELECTABLE_IMAGES in shared.js to experiment with resources.

Conventions for Link Handling

This client employes certain conventions within Mix.dialog to offer special types of event handling in the presentation layer.

To take advantage of these, simply mark up your Rich Text messages appropriately with the data attributes, depending on the intended interaction.

Intent or Entity

To trigger an intent or entity selection, leverage the following attributes:

• data-mix-action="selectable"
• data-mix-selectable-id - can be INTENT or the entity name
• data-mix-selectable-value

Sample

<a href="#"
  data-mix-action="selectable"
  data-mix-selectable-id="INTENT"
  data-mix-selectable-value="iBuyPhone">
    Buy a Phone
</a>

<a href="#"
  data-mix-action="selectable"
  data-mix-selectable-id="ePhoneCapacity"
  data-mix-selectable-value="128GB">
    128GB
</a>

Simulated Inputs

To trigger a simulated user input, leverage the following attributes:

• data-mix-action="input"
• data-mix-input

Sample

<a href="#"
  data-mix-action="input"
  data-mix-input="$500">
    $500
</a>

Data Access Integrations

Using Client Fetch during development

This client offers developers the ability to use the client_fetch mode of the Data Accss node in Mix.dialog and integrate with an Azure Function.

This pattern would apply in a gateway-style integration, however the intention of this set up, is to eventually have the integrations use external_fetch pointing to deployed Functions.

How-To: Add Handlers for Data Access Nodes

Create a new Function specifying the endpoint URL and the data access node name to use if client-driven:

./scripts/create-da-handler.sh "weather-api-city-conditions" "Server_Weather_CityConditions"

Navigate to api/weather-api-city-conditions/__init__.py and start integrating.

By default, POST requests are expected with the body containing the sendData payload. Update api/weather-api-city-conditions/function.json if other methods are desired.

⚠️ Important: Once the function has been set up, add a handler to ExternalFetchHandlers in dlgaas.js. Create a function with the Data Access node's name, pointing to the newly defined endpoint.

Data Access Requests intended for the Client

There are cases where the use of Mix.dialog Data Access node's client fetch is intended for the end client (vs. the gateway pattern employed here).

How-To: Add Handlers for Client Actions

Add a handler to ClientFetchHandlers in dlgaas.js named with the Data Access node's name.

Client Location Capability

At this time, geolocation (lat/lng) is supported through the HTML5 API, when the client is running securely.

To request the client to provide it's location, set the Data Access node's ID to Client_Location_LatLng, and this will return a location object.

Sample Data Integrations

The data integration layer in this client is handled through Functions, and complimented by sample integrations in api/providers.py.

Each data access access is represented by it's own Function, in preparation for use when deployed and hosted.

This client illustrates integrations with: Yahoo Finance, AccuWeather, SendGrid and a Mock Phone Store.

Yahoo Finance

This API does not require authentication, in this case: the requests are made with no further checks.

• POST /api/finance-api-asset-price

AccuWeather

Third party service authentication is required and done through passing the token in the payload of the request. In this case: apiKeyAccuweather is used with the service and must be saved in the session (stored in a variable).

• POST /api/weather-api-city-search
• POST /api/weather-api-city-conditions

SendGrid

Third party service authentication is configured as part of the Function, but a SENDGRID_TOKEN is required to execute calls from known sources.

In this case: the token must been assigned in Mix at the PROJECT level.

• POST /api/email-api-send

Mock Phone Store

This integration illustrates the use of Dynamic Wordsets, consistently filtering options based on the user's selection, derived from static data.

• POST /api/store-api-filter-phone-wordset
• POST /api/store-api-request-purchase
• POST /api/store-api-purchase

Branding

The chat skin can be branded.

Resources

A typical brand definition includes:

• Logo (image)
• Fonts
• Stylesheet

Assets are located in /app/src/static/brands/

• Fonts can be stored here as well

Configuration

The query parameter brand can be used to preload this configuration. Assign the value to the name of the subfolder.

Here is a sample CSS, assuming that the font files are also located in the directory.

/* Brand Assets */
@font-face {
  font-family: 'Font-Bold';
  src: url('./Font-Bold.woff2') format('woff');
  font-weight: bold;
  font-style: normal;
}
@font-face {
  font-family: 'Font';
  src: url('./Font-Regular.woff2') format('woff');
  font-weight: normal;
  font-style: normal;
}

:root {
  /* MyBrand*/
  --skin-logo: url("/brands/mybrand/logo.svg");
  --skin-logo-header: url("/brands/mybrand/logo-header.svg");
  --skin-logo-header-width: 30px;
  --skin-logo-header-height: 30px;
  --skin-logo-main-width: 25%;
  --skin-logo-main-height: auto;
  --skin-color: #336699;
  --skin-font: 'Font', 'Helvetica';
  --skin-font-size: 14px;
  --skin-chat-bg-color: #fff;
  --skin-header-text-color: #fff;
  --skin-header-bg-color: #27251F;
  --skin-footer-bg-color: #fff;
  --skin-message-text-color: #27251F;
}

Deployment

Publishing to Azure

To deploy this client, follow the Azure StaticWebApps deployment guide to publish. Update app/static/staticwebapp.config.json as needed.

Essentially:

1. Create a GitHub Repo
2. Create an Azure StaticWebApp pointing to the GitHub Repo (use: gatsby, point to app/, and api/)
3. Configure accordingly in Azure
   • Create a Application Insights resource and link to StaticWebApp (APPINSIGHTS_INSTRUMENTATIONKEY)
   • Add environment variables (sengrid_api_key, sendgrid_from_email)

Considerations

1. update content-security-policy for client - staticwebapp.config.json
2. update client host (ASR_SERVICE_URL, DLG_SERVICE_URL) - shared.js
3. update environment variables for functions - .env

License

This source code is licensed under the Apache-2.0 license found in the LICENSE.md file in the root directory of this source tree.