- Maintainer: Chris Stone (chstone@microsoft.com)
- Project: Call-Center Automation (Solution How-to Guide)
- Use case: Provide an Interactive Voice Response (IVR) bot to process product orders for a fictitious company that sells bicycles and bicycle accessories.
- Architecture
- Build
- Manual Deployment
- Automated Deployment
- Usage
- Scaling
- Customization
- Copyright
- License
- Skype Client
User initiates call - Bot Connector + Skype Calling Channel
Routes calls from Skype to the bot - Azure App Services
Hosts the bot application, which manages logic and API calls - Cosmos DB
Stores bot state and event logs - Bing Speech Service
Processes speech-to-text - LUIS (Language Understanding Intelligent Service)
Extracts intent and entities from text - Azure Search
Indexes the product catalog for product-query matching - Azure SQL
Stores product and order data - Azure Storage
Stores bot audio data for debugging
This project is built using TypeScript. Your environment should have the "current" NodeJS runtime in order to build the project.
If you are only interested in automated deployment, you may skip this section.
After cloning this repo, run the following shell commands from the repo root:
npm installnpm run build- Copy
./package.json,./web.config,./src/bot-settings.json, and./datato./dist cd distnpm install --production
If you are only interested in automated deployment, you may skip this section.
Create the following resources using the Azure Portal, PowerShell, or Azure CLI.
Unless otherwise noted, use any configuration and scale parameters you like
Azure StorageCosmos DB(with SQL / DocumentDB API)Azure SQL(with AdventureWorksLT sample DB)Azure SearchAzure App ServiceCognitive Servicekeys:- Key for
Bing Speech API - Key for
Language Understanding Intelligent Service (LUIS)
- Key for
See guided screenshots for bot registration and enabling Skype Calling
- Register a new bot at the Bot Framework Portal
- Create a new Microsoft App, and make note of its ID and Secret
- Leave the messaging endpoint blank for now
- After your bot is registered, click through to its Skype Channel and ensure that Skype Calling is enabled. Your calling endpoint is
https://YOUR_WEB_APP.azurewebsites.net/api/calls
See guided screenshots for finding the LUIS programmatic key
- Log in to the LUIS Portal
- Navigate to the
My Keystab - Make a note of your
Programmatic API Key
Create the following application settings on your Web App:
Learn how to configure a site's App Settings using the Azure Portal
You can find all required resource keys and names (below, in bold) using the Azure Portal, with the exception of the LUIS Programmatic Key, which must be copied from the LUIS Portal.
| NAME | VALUE |
|---|---|
| WEBSITE_NODE_DEFAULT_VERSION | 7.7.4 |
| CALLBACK_URL | https://YOUR_WEB_APP.azurewebsites.net/api/calls |
| MICROSOFT_APP_ID | YOUR_APP_ID (GUID) |
| MICROSOFT_APP_PASSWORD | YOUR_APP_SECRET |
| LUIS_REGION | westus (or your region, if different) |
| LUIS_KEY | YOUR_LUIS_KEY |
| LUIS_MANAGER_KEY | YOUR_LUIS_PROGRAMATIC_KEY |
| LUIS_APP_ID | (empty) |
| SPEECH_KEY | YOUR_SPEECH_KEY |
| SPEECH_ENDPOINT | https://speech.platform.bing.com/recognize |
| SPEECH_REGION | (empty) |
| SEARCH_SERVICE | YOUR_SEARCH_ACCOUNT |
| SEARCH_KEY | YOUR_SEARCH_KEY |
| BLOB_ACCOUNT | YOUR_STORAGE_ACCOUNT |
| BLOB_KEY | YOUR_STORAGE_ACCOUNT_KEY |
| DDB_URL | https://YOUR_COSMOS_DB_ACCOUNT.documents.azure.com:443/ |
| DDB_KEY | YOUR_COSMOS_DB_KEY |
| SQL_HOST | YOUR_AZURE_SQL_HOST |
| SQL_USER | YOUR_AZURE_SQL_USER |
| SQL_PASSWORD | YOUR_AZURE_SQL_PASSWORD |
| SQL_DATABASE | YOUR_AZURE_SQL_DATABASE |
| LOG_BLOB_CONTAINER | bot-audio |
| LOG_DDB_DATABASE | bot-data |
| LOG_DDB_COLLECTION | bot-logs |
| STORE_DDB_DATABASE | bot-data |
| STORE_DDB_COLLECTION | bot-sessions |
After building the project (see build), upload the contents of dist to your App Service
Learn how to upload files to a web app using FTP and PowerShell
You can also deploy from the command line using WebDeploy
To automatically create, configure, and deploy all Azure resources at once, run the following commands in PowerShell (or use your favorite ARM deployment tool):
You will be prompted for three configuration parameters. See the Bot Registration Guide and the LUIS Programmatic-Key Guide if you need help finding these values.
$rg = "call-center"
$loc = "eastus"
New-AzureRmResourceGroup $rg $loc
New-AzureRmResourceGroupDeployment -Name CallCenterSolution -ResourceGroupName $rg -TemplateFile .\azuredeploy.jsonYou will use the Skype Client to initiate calls to your bot (Skype for Business is not current supported).
Windows Users may use the App Store Client
Before talking to your bot, you must add it to your Skype contacts list. You can find a link to add your bot to Skype on the Bot Portal under the channel listing.
Directly add the bot to your contacts: https://join.skype.com/bot/YOUR_APP_ID
Using your Skype client, initiate a call to your bot and follow the prompts. You can order any product in the standard adventure works category, such as a mountain bike, fenders, or bike wash. The bot will prompt you to disambiguate product names or to choose product attributes, if necessary.
- "mountain bike": a general product category
- "what is mountain 500?": get more information about a specific product
- "mountain 500": a specific product
- "mountain 500 in silver": a specific product with a specific color
- "bicycle for road use": a natural language query
- "extra large jersey": a general product category with specific size
A basic deployment will scale to around 10 concurrent requests per second. Each layer of the architecture supports a separate level of concurrency, but the entire solution is bound by the narrowest pipeline of all the services. Services may be scaled up to support higher throughput per resource, or scaled out to spread throughput across multiple resources.
| SERVICE | MAX RPS PER INSTANCE | SCALE UP | SCALE OUT |
|---|---|---|---|
| LUIS | 10 | N/A | Custom account partitioning |
| Bing Speech | 20 | N/A | Custom account partitioning |
| App Service | 100s | Add RAM/cores | Add instances |
| Search | ~60 | N/A | Add replicas |
| Cosmos DB (DocumentDB) | ~10K | N/A | Add partitioning |
| Blobs | ~20K | N/A | N/A |
CUSTOM ACCOUNT PARTITIONING: Scale-out of Bing Speech and LUIS is not currently available. If your scaling needs exceed 10 requests per second, you can implement a custom load balancing scheme across multiple service endpoints.
E.g. to double the capacity of LUIS, create a second LUIS application using the same JSON configuration as the first. Then modify your bot to perform round-robin (alternating) queries to each service.
This bot is tuned end-to-end to work specifically with the AdventureWorks sample product database. In order to transition to a custom data set, some consideration must be taken to account for the format and structure of your custom data and how to apply best practices for LUIS and Azure Search.
Intents drive the workflow of your bot. If a caller is prompted to speak a product query, but responds with a desire to get the shipping status on an existing order, the bot should seamlessly branch to a new dialog to fulfill the new request, and then return to the original prompt.
There are two primary challenges when working with open-ended, natural conversation dialogs:
- It is impractical to provide coverage for every conceivable intent. A bot is an intelligent conversationalist, not true Artificial Intelligence.
- User-discovery of the intent-recognition scenarios that the bot does support can be difficult to facilitate without rote narration of available options.
To overcome these challenges, it is often best to begin with a directed, default intent. Try to guide the conversation as much as possible with yes-or-no questions or short, multiple choice options.
However, the service should allow the caller to go off-script when the caller's spoken intent does not match the prompt. The Bot Framework manages a dialog stack for you, so your bot can easily fork to a new set of prompts and responses before returning to the original, directed, dialog. Your custom intents should be domain-specific, but still generic actions.
An intent describes the action and the domain of some user request. A domain represents a high level grouping of some logical section of your app (e.g. products, orders, calendar). The action is some verb that describes what to do against that domain (typically a variant of the typical database CRUD actions–Create, Read, Update, Delete). An intent returned from LUIS may include entities, which, in terms of natural language, can be thought of as the objects to a transitive verb. An entity is the "on what" or "against what" component of the intent's action.
Learn how to add entities to your LUIS app in the LUIS portal.
Learn how to automatically map your LUIS intents to Bot Dialogs in code.
Every domain has its own set of common entities. An entity represents a class of similar objects that are detected from raw text by LUIS. There are three main types of entities: prebuilt (cross-domain, provided by Bing), custom (learned from your labeled data), and closed-list (a static set of terms chosen by you). This app uses only closed-list entities across four classes: color, category, sex, and size.
Your goal when building and training custom entities should be to identify object classes that can be used by the search engine to boost results for the specified class.
There are two approaches to using entities with search: filtering and boosting. By applying a filter, you eliminate results that do not match the entity metadata. By applying a boost, you surface matching entities to the top of the result set, but you also return non-matches, albeit with a lower score.
Use a filter when the entity represents a broad or unambiguous category or if the utterance is comprised soley of entities. E.g.:
"bicycle" // "bicycle is a category entity
"clothing" // "clothing" is a category entity
"red bicycle" // "red" is a color entity; "bicycle" is a category entity
Apply a search filter to return only matches for red bicycle where the color field is red:
<url>?search=red bicycle&$filter=colors/any(x: x eq 'red')
Use a boost when the entity is included with other terms. E.g.:
"mountain bicycle" // a category->bicycle filter would be ok here
"bicycle rack" // but not here. "bicycle racks" are in the 'accessories' category; not the 'bicycles' category
Apply a search boost to raise the score for the same results from the filtered query (typically bringing matches to the top of the result set) while still including other colors as well (e.g. if red was not available):
<url>?red bicycle colors:red^2
Learn more about advanced query operators in Azure Search
Use custom analyzers in Azure Search to enable content matching against domain-specific synonyms, or to map between a product's written form and its spoken form. For instance, product sizes are represented in the database as S, M, L, and XL, however, when speaking, we refer to small, medium, large, and extra large. Use one or more #Microsoft.Azure.Search.SynonymTokenFilters to enable matching between these different forms.
Learn more about creating custom analyzers in Azure Search
This app uses three synonym groups to map both between language variants (hat/cap) and representational variants (S/small):
[
"S,small",
"M,medium",
"XL,extra large",
"L,large"
][
"bike=>bicycle",
"lady,girl=>woman",
"guy,boy=>man",
"clothes=>clothing",
"hat=>cap"
][
"lady,girl=>woman",
"guy,boy=>man"
]Azure Search now supports query-time synonym maps in public preview.
Source content (SQL tables, raw files, etc.) often is not in an ideal state for consumption by bots and search applications. This section describes common preprocessing transformations that can be applied to source content. In this case, the source content is an Azure SQL database containing a handful of tables describing a product catalog.
Azure Search offers a configurable Azure SQL indexer for no-code, automated ingest of your source content, given the name of a table or view in your database. For simple data sets, a table works well, but for most applications, you will want to connect to a custom view to account for SQL joins, predicates, and other custom result processing.
Learn more about connecting Azure Search and Azure SQL
Search documents, by nature, are denormalized (unjoined). Azure Search does not support joins, so all of the information describing a result must be attached to a single document.
To achieve a high level of performance and usability, it is critical to apply a proper denormalization strategy against your normalized (table-joined) data. A common mistake is to simply map each row of a single table to a corresponding search document. For data structures like a product catalog that are highly normalized, this can often lead to "noisy" data, or the reverse effect, information loss.
Consider the following two AdventureWorks tables. Both contain product names, but the latter has many repeated sections, varying only by a single attribute.
| Name |
|---|
| HL Road Frame |
| LL Road Frame |
| ML Road Frame |
| ML Road Frame-W |
| Name |
|---|
| HL Road Frame - Black, 44 |
| HL Road Frame - Black, 48 |
| HL Road Frame - Black, 52 |
| HL Road Frame - Black, 58 |
| HL Road Frame - Black, 62 |
| HL Road Frame - Red, 44 |
| HL Road Frame - Red, 48 |
| HL Road Frame - Red, 52 |
| HL Road Frame - Red, 56 |
| HL Road Frame - Red, 58 |
| HL Road Frame - Red, 62 |
| ... and so on, for LL, ML, and ML-W, etc. |
If we envision each row as a search result, the second table is quickly overwhelmed with near-duplicate results, leading to a poor user experience. However, the first table lacks valuable product information needed to identify a specific product SKU.
The solution is to collapse the information from the second table onto the first using a custom view and a handful of user-defined functions.
Other ETL techniques may be used to massage your content. This example uses functions.
See the custom view used by this solution.
Azure Search supports the Collection(Edm.String) document type for storing semi-complex, searchable metadata on a document. In this case, we will define two collections: one for color and one for size. Both of these product attributes should be searchable, but, because they are attached to a parent document, they will not return a new document for every possible combination. The ideal search document looks like:
{
"productModelId": "26",
"modifiedDate": "2006-06-01T00:00:00Z",
"name": "Road-250",
"category": ["bikes", "road bikes"],
"colors": ["black", "red"],
"sizes": ["44", "48", "52", "58"],
"sex": "Unisex",
"maxStandardCost": 1554.9479,
"minStandardCost": 1518.7864,
"maxListPrice": 2443.35,
"minListPrice": 2443.35,
"products": "<serialized-json-here>",
"description_HE": "<hebrew-text-here>",
"description_ZH_CHT": "<chinese-text-here>",
"description_EN": "<english-text-here>",
"description_AR": "<arabic-text-here>",
"description_TH": "<thai-text-here>",
"description_FR": "<french-text-here>"
}In order to properly prepare the values for Azure Search, we must coerce them into a JSON string. As of this writing, there is no built-in SQL functionality to achieve this, but we can apply the SQL coalesce operator inside a custom function to build the string:
CREATE FUNCTION ufnGetColorsJson(@productModelId int)
RETURNS nvarchar(max)
AS
BEGIN
DECLARE @vals AS nvarchar(max)
SELECT
@vals = coalesce(@vals + ',"', '"') + [t].[color] + '"'
FROM
(
SELECT
DISTINCT [color]
FROM
[SalesLt].[Product] [p]
WHERE
[p].[productModelId]=@productModelId
) [t]
RETURN lower('[' + @vals + ']')
ENDThen we create a new view to return the denormalized representation of our data:
CREATE VIEW [SalesLT].[vProductsForSearch]
AS
SELECT
[pm].[name],
[pm].[productModelId],
[pm].[modifiedDate],
(SELECT dbo.ufnGetColorsJson([productModelId])) [colors],
(SELECT dbo.ufnGetSizesJson([productModelId])) [sizes]
FROM
[SalesLt].[ProductModel] [pm]See the full view with more custom functions in this repo under
./data/sql
Azure Search executes this view when it indexes (and periodically re-indexes) the product database.
©2017 Microsoft Corporation. All rights reserved. This information is provided "as-is" and may change without notice. Microsoft makes no warranties, express or implied, with respect to the information provided here. Third party data was used to generate the solution. You are responsible for respecting the rights of others, including procuring and complying with relevant licenses in order to create similar datasets.
The MIT License (MIT) Copyright (c) 2017 Microsoft Corporation
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.