classic_dialog_skill_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dialog skill analysis for Classic Watson Assistant Experience\n",
    "\n",
    "## Introduction\n",
    "Dialog Skill Analysis for [Classic Watson Assistant (WA)](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-watson-assistant-faqs#faqs-new-experience) is intended for use by chatbot designers, developers and data scientists who would like to experiment with and improve their existing dialog skill design wih the classic experience.  \n",
    "\n",
    "This notebook assumes familiarity with the Watson Assistant product as well as concepts involved in dialog skill design such as intent, entities, and utterances.   \n",
    "\n",
    "### Environment\n",
    "- Python version 3.9 or above is required. \n",
    "- Install dependencies with `pip install -r requirements.txt` and refer to `requirements.txt`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install all the required packages and filter out any warnings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Markdown, display, HTML\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard python libraries\n",
    "import sys, os\n",
    "import json\n",
    "import importlib\n",
    "from collections import Counter\n",
    "\n",
    "# External python libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "nltk.download('punkt')\n",
    "import ibm_watson\n",
    "\n",
    "# Internal python libraries\n",
    "from assistant_skill_analysis.utils import skills_util, lang_utils\n",
    "from assistant_skill_analysis.highlighting import highlighter\n",
    "from assistant_skill_analysis.data_analysis import summary_generator\n",
    "from assistant_skill_analysis.data_analysis import divergence_analyzer\n",
    "from assistant_skill_analysis.data_analysis import similarity_analyzer\n",
    "from assistant_skill_analysis.term_analysis import chi2_analyzer\n",
    "from assistant_skill_analysis.term_analysis import keyword_analyzer\n",
    "from assistant_skill_analysis.term_analysis import entity_analyzer\n",
    "from assistant_skill_analysis.confidence_analysis import confidence_analyzer\n",
    "from assistant_skill_analysis.inferencing import inferencer\n",
    "from assistant_skill_analysis.experimentation import data_manipulator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pick the language code correspond to your workspace data:   \n",
    "*Supported Language codes:* **en, fr, de, es, cs, it, pt, nl, zh-cn, zh-tw**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(lang_utils)\n",
    "LANGUAGE_CODE=\"en\" # change the language code to work with other languages\n",
    "lang_util = lang_utils.LanguageUtility(LANGUAGE_CODE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of contents\n",
    "\n",
    "1. [Part 1: Prepare the training data](#part1)<br>\n",
    "2. [Part 2: Prepare the test data](#part2)<br>\n",
    "3. [Part 3: Perform advanced analysis](#part3)<br>\n",
    "4. [Part 4: Summary](#part4)<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='part1'></a>\n",
    "# Part 1: Prepare the training data\n",
    "1.1 [Set up access to the training data](#part1)<br>\n",
    "1.2 [Process Dialog Skill Training Data](#part1.2)<br>\n",
    "1.3 [Analyze data distribution](#part1.3)<br>\n",
    "1.4 [Perform a correlation analysis](#part1.4)<br>\n",
    "1.5 [Visualize terms using a heat map](#part1.5)<br>\n",
    "1.6 [Ambiguity in the training data](#part1.6)<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='setup'></a>\n",
    "## 1.1 Set up access to the training data\n",
    "\n",
    "Provide access credentials for an existing dialog skill that you would like to analyze. For this you need your API Key and Workspace ID values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(skills_util)\n",
    "\n",
    "# Change Assistant API version if needed\n",
    "# Find Latest --> https://cloud.ibm.com/docs/services/assistant?topic=assistant-release-notes\n",
    "API_VERSION = '2020-04-01'\n",
    "\n",
    "# choose a datacenter to use\n",
    "datacenters = {\n",
    "    'dallas': ('https://api.us-south.assistant.watson.cloud.ibm.com', 'https://iam.cloud.ibm.com/identity/token'),\n",
    "    'washington': ('https://api.us-east.assistant.watson.cloud.ibm.com',     'https://iam.cloud.ibm.com/identity/token'),\n",
    "    'frankfurt' : ('https://api.eu-de.assistant.watson.cloud.ibm.com',     'https://iam.cloud.ibm.com/identity/token'),\n",
    "    'sydney'    : ('https://api.au-syd.assistant.watson.cloud.ibm.com',     'https://iam.cloud.ibm.com/identity/token'),\n",
    "    'tokyo'     : ('https://api.jp-tok.assistant.watson.cloud.ibm.com',     'https://iam.cloud.ibm.com/identity/token'),\n",
    "    'london'    : ('https://api.eu-gb.assistant.watson.cloud.ibm.com',     'https://iam.cloud.ibm.com/identity/token'),\n",
    "}\n",
    "URL, authenticator_url = datacenters['dallas']\n",
    "\n",
    "# For ICP(IBM Cloud Private), you can disable SSL verification by changing this to True\n",
    "DISABLE_SSL_VERTIFICATION = False \n",
    "\n",
    "# By default we only need the IAM API Key & the Workspace ID\n",
    "\n",
    "# If you run the notebook regularly you can uncomment the two lines below\n",
    "# & comment out the line after it\n",
    "#iam_apikey = '###'\n",
    "#skill_id = '###'\n",
    "iam_apikey, skill_id, _ = skills_util.input_credentials()\n",
    "conversation = skills_util.retrieve_conversation(iam_apikey=iam_apikey,\n",
    "                                                 url=URL,\n",
    "                                                 api_version=API_VERSION,\n",
    "                                                 authenticator_url=authenticator_url)\n",
    "\n",
    "#If you do not have IAM based API Keys\n",
    "#but have access to a Username, Password & Workspace ID\n",
    "#You can comment out the two lines above & uncomment the lines below to authenticate\n",
    "# username = 'apikey'\n",
    "# password = '###'\n",
    "# skill_id = '###'\n",
    "# conversation = skills_util.retrieve_conversation(username=username,\n",
    "#                                              password=password,                                                       \n",
    "#                                              url=URL,\n",
    "#                                              api_version=API_VERSION)\n",
    "\n",
    "\n",
    "conversation.set_disable_ssl_verification(DISABLE_SSL_VERTIFICATION)\n",
    "\n",
    "workspace = skills_util.retrieve_workspace(skill_id=skill_id,\n",
    "                                           conversation=conversation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract user workspace\n",
    "workspace_pd, workspace_vocabulary, entities, _ = skills_util.extract_workspace_data(workspace, language_util=lang_util)\n",
    "entities_list = [item['entity'] for item in entities]\n",
    "\n",
    "\n",
    "display(Markdown(\"### Sample of Utterances & Intents\"))\n",
    "display(HTML(workspace_pd.sample(n = len(workspace_pd) if len(workspace_pd)<10 else 10)\n",
    "             .to_html(index=False)))\n",
    "if entities_list:\n",
    "    display(Markdown(\"### Sample of Entities\"))\n",
    "    display(HTML(pd.DataFrame({\"Entity\":entities_list})\n",
    "                 .sample(n = len(entities_list) if len(entities_list)<10 else 10)\n",
    "                 .to_html(index=False)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='part1.2'></a>\n",
    "## 1.2 Process the dialog skill training data\n",
    "\n",
    "Generate summary statistics related to the given skill and workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(summary_generator)\n",
    "summary_generator.generate_summary_statistics(workspace_pd, entities_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='part1.3'></a>\n",
    "## 1.3 Analyze the data distribution\n",
    "\n",
    "- [Analyze class imbalance](#imbalance)\n",
    "- [List the distribution of user examples by intent](#distribution)\n",
    "- [Actions for class imbalance](#actionimbalance)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyze class imbalance<a id='imbalance'></a>\n",
    "\n",
    "Analyze whether the data set contains class imbalance by checking whether the largest intent contains less than double the number of user examples contained in the smallest intent. If there is an imbalance it does not necessarily indicate an issue; but you should review the [actions](#actionimbalance) section below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(summary_generator)\n",
    "class_imb_flag = summary_generator.class_imbalance_analysis(workspace_pd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List the distribution of user examples by intent<a id='distribution'></a>\n",
    "Display the distribution of intents versus the number of examples per intent (sorted by the number of examples per intent) below. Ideally you should not have large variations in terms of number of user examples for various intents. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(summary_generator)\n",
    "summary_generator.scatter_plot_intent_dist(workspace_pd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(summary_generator)\n",
    "summary_generator.show_user_examples_per_intent(workspace_pd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Actions for class imbalance<a id='actionimbalance'></a>\n",
    "\n",
    "Class imbalance will not always lead to lower accuracy, which means that all intents (classes) do not need to have the same number of examples.\n",
    "\n",
    "Given a hypothetical chatbot related to banking:<br>\n",
    "\n",
    "- For intents like `updateBankAccount` and `addNewAccountHolder` where the semantics difference between them is subtler, the number of examples per intent needs to be somewhat balanced otherwise the classifier might favor the intent with the higher number of examples.\n",
    "- For intents like `greetings` that are semantically distinct from other intents like `updateBankAccount`, it may be acceptable for it to have fewer examples per intent and still be easy for the intent detector to classify.\n",
    "\n",
    "\n",
    "\n",
    "If the intent classification accuracy is lower than expected during testing, you should re-examine the distribution analysis.  \n",
    "\n",
    "With regard to sorted distribution of examples per intent, if the sorted number of user examples varies a lot across different intents, it can be a potential source of bias for intent detection. Large imbalances in general should be avoided. This can potentially lead to lower accuracy. If your graph displays this characteristic, this could be a source of error.\n",
    "\n",
    "For further guidance on adding more examples to help balance out your distribution, refer to \n",
    "<a href=\"https://cloud.ibm.com/docs/services/assistant?topic=assistant-intent-recommendations#intent-recommendations-get-example-recommendations\" target=\"_blank\" rel=\"noopener no referrer\">Intent Example Recommendation</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='part1.4'></a>\n",
    "## 1.4  Perform correlation analysis\n",
    "\n",
    "- [Retrieve the most correlated unigrams and bigrams for each intent](#retrieve)\n",
    "- [Actions for anomalous correlations](#anomalous)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve the most correlated unigrams and bigrams for each intent<a id='retrieve'></a>\n",
    "\n",
    "Perform a chi square significance test using count features to determine the terms that are most correlated with each intent in the data set. \n",
    "\n",
    "A `unigram` is a single word, while a `bigram` is two consecutive words from within the training data. For example, if you have a sentence like `Thank you for your service`, each of the words in the sentence are considered unigrams while terms like `Thank you`, `your service` are considered bigrams.\n",
    "\n",
    "Terms such as `hi`, `hello` correlated with a `greeting` intent are reasonable. But terms such as `table`, `chair` correlated with the `greeting` intent are anomalous. A scan of the most correlated unigrams & bigrams for each intent can help you spot potential anomalies within your training data.\n",
    "\n",
    "**Note**: We ignore the following common words (\\\"stop words\\\") from consideration `an, a, in, on, be, or, of, a, and, can, is, to, the, i`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "importlib.reload(chi2_analyzer)\n",
    "unigram_intent_dict, bigram_intent_dict = chi2_analyzer.get_chi2_analysis(workspace_pd, lang_util=lang_util)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Actions for anomalous correlations<a id='anomalous'></a>\n",
    "\n",
    "If you identify unusual or anomalous correlated terms such as: numbers, names and so on, which should not be correlated with an intent, consider the following:\n",
    "  \n",
    "- **Case 1** : If you see names appearing amongst correlated unigrams or bigrams, add more variation of names so no specific names will be correlated  \n",
    "- **Case 2** : If you see specific numbers like 1234 amongst correlated unigrams or bigrams and these are not helpful to the use case, remove or mask these numbers from the examples\n",
    "- **Case 3** : If you see terms which should never be correlated to that specific intent, consider adding or removing terms/examples so that domain specific terms are correlated with the correct intent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='part1.5'></a>\n",
    "## 1.5 Visualize terms using a heat map\n",
    "\n",
    "- [Display term analysis for a custom intent list](#customintent)\n",
    "- [Actions for anomalous terms in the heat map](#heatmap)\n",
    "\n",
    "A heat map of terms is a method to visualize terms or words that frequently occur within each intent. Rows are the terms, and columns are the intents. \n",
    "\n",
    "The code below displays the top 30 intents with the highest number of user examples in the analysis. This number can be changed if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(keyword_analyzer)\n",
    "\n",
    "INTENTS_TO_DISPLAY = 30  # Total number of intents for display\n",
    "MAX_TERMS_DISPLAY = 30  # Total number of terms to display\n",
    "\n",
    "intent_list = []\n",
    "keyword_analyzer.seaborn_heatmap(workspace_pd, lang_util, INTENTS_TO_DISPLAY, MAX_TERMS_DISPLAY, intent_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display term analysis for a custom intent list<a id='customintent'></a>\n",
    "\n",
    "If you wish to see term analysis for specific intents, feel free to add those intents to the intent list. This generates a custom term heatmap. The code below displays the top 20 terms, but this can be changed if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(keyword_analyzer)\n",
    "# intent_list = ['intent1','intent2','intent3'] \n",
    "intent_list = [] \n",
    "\n",
    "\n",
    "MAX_TERMS_DISPLAY = 20  # Total number of terms to display\n",
    "\n",
    "if intent_list: \n",
    "    keyword_analyzer.seaborn_heatmap(workspace_pd, lang_util, INTENTS_TO_DISPLAY, MAX_TERMS_DISPLAY, intent_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Actions for anomalous terms in the heat map<a id='heatmap'></a>\n",
    "\n",
    "If you notice any terms or words which should not be frequently present within an intent, consider modifying examples in that intent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='part1.6'></a>\n",
    "## 1.6 Ambiguity in the training data\n",
    "\n",
    "- [Uncover ambiguous utterances across intents](#uncover)\n",
    "- [Actions for ambiguity in the training data](#ambiguityaction)\n",
    "\n",
    "Run the code blocks below to uncover possibly ambiguous terms based on feature correlation.\n",
    "\n",
    "Based on the chi-square analysis above, generate intent pairs which have overlapping correlated unigrams and bigrams.\n",
    "This allows you to get a glimpse of which unigrams or bigrams might cause potential confusion with intent detection:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A. Top intent pairs with overlapping correlated unigrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(chi2_analyzer)\n",
    "ambiguous_unigram_df = chi2_analyzer.get_confusing_key_terms(unigram_intent_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### B. Top intent pairs with overlapping correlated bigrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(chi2_analyzer)\n",
    "ambiguous_bigram_df = chi2_analyzer.get_confusing_key_terms(bigram_intent_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### C. Overlap checker for specific intents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add specific intent or intent pairs for which you would like to see overlap\n",
    "importlib.reload(chi2_analyzer)\n",
    "intent1 = 'Goodbye'\n",
    "intent2 = ''\n",
    "chi2_analyzer.chi2_overlap_check(ambiguous_unigram_df,ambiguous_bigram_df,intent1,intent2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Uncover ambiguous utterances across intents<a id='uncover'></a>\n",
    "The following analysis shows user examples that are similar but fall under different intents.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(similarity_analyzer)\n",
    "similar_utterance_diff_intent_pd = similarity_analyzer.ambiguous_examples_analysis(workspace_pd, lang_util)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Actions for ambiguity in the training data<a id='ambiguityaction'></a>\n",
    "\n",
    "**Ambiguous intent pairs**  \n",
    "If you see terms which are correlated with more than 1 intent, review if this seems anomalous based on the use case for that intent. If it seems reasonable, it is probably not an issue.  \n",
    "\n",
    "**Ambiguous utterances across intents** \n",
    "- **Duplicate utterances**: For duplicate or almost identical utterances, remove those that seem unnecessary.\n",
    "- **Similar utterances**: For similar utterances, review the use case for those intents and make sure that they are not accidental additions caused by human error when the training data was created.  \n",
    "\n",
    "For more information about entity, refer to the <a href=\"https://cloud.ibm.com/docs/services/assistant/services/assistant?topic=assistant-entities\" target=\"_blank\" rel=\"noopener no referrer\">Entity Documentation</a>.\n",
    "\n",
    "For more in-depth analysis related to possible conflicts in your training data across intents, try the conflict detection feature in Watson Assistant. Refer to <br> <a href=\"https://cloud.ibm.com/docs/services/assistant?topic=assistant-intents#intents-resolve-conflicts\" target=\"_blank\" rel=\"noopener no referrer\">Conflict Resolution Documentation</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='part2'></a>\n",
    "# Part 2: Prepare the test data\n",
    "\n",
    "Analyze your existing Watson Assistant Dialog Skill with the help of a test set.\n",
    "\n",
    "2.1. [Obtain test data from Cloud Object Storage](#cos)<br>\n",
    "2.2. [Evaluate the test data](#evaluate) <br>\n",
    "2.3. [Analyze the test data](#testanalysis) <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Obtain test data from Cloud Object Storage<a id='cos'></a>\n",
    "\n",
    "Upload a test set in tsv format. Each line in the file should have only `User_Input<tab>Intent`  \n",
    "\n",
    "For example:\n",
    "```\n",
    "hello how are you<tab>Greeting  \n",
    "I would like to talk to a human<tab>AgentHandoff  \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "importlib.reload(skills_util)\n",
    "\n",
    "#Separator: Use '\\t' for tab separated data, ',' for comma separated data\n",
    "separator = \"\\t\"\n",
    "\n",
    "test_set_path = \"./test.tsv\"\n",
    "test_df = skills_util.process_test_set(test_set_path, lang_util,separator)\n",
    "\n",
    "display(Markdown(\"### Random Test Sample\"))\n",
    "display(HTML(test_df.sample(n=10).to_html(index=False)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Evaluate the test data<a id='evaluate'></a>\n",
    "These steps can take time if you have a large test set.  \n",
    "\n",
    "**<font color=red>Note</font>**: You will be charged for calls made from this notebook based on your Watson Assistant plan. The user_id will be the same for all message calls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "THREAD_NUM = min(4, os.cpu_count() if os.cpu_count() else 1)\n",
    "# increase timeout if you experience `TimeoutError`. \n",
    "# Increasing the `TIMEOUT` allows the process more breathing room to compete\n",
    "TIMEOUT = 1 # `TIMEOUT` is set to 1 second\n",
    "full_results = inferencer.inference(conversation=conversation,\n",
    "                                    test_data=test_df,\n",
    "                                    max_thread=THREAD_NUM, \n",
    "                                    skill_id=skill_id,\n",
    "                                    timeout=TIMEOUT\n",
    "                                   )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='part2.1'></a>\n",
    "## 2.3 Analyze the test data<a id='testanalysis'></a>\n",
    "\n",
    "- [Display an overview of the test data](#overview)\n",
    "- [Compare the test data and the training data](#compare)\n",
    "- [Determine the overall accuracy on the test set](#accuracy)\n",
    "- [Analyze the errors](#errors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display an overview of the test data<a id='overview'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(summary_generator)\n",
    "summary_generator.generate_summary_statistics(test_df)\n",
    "summary_generator.show_user_examples_per_intent(test_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare the test data and the training data<a id='compare'></a>\n",
    "\n",
    "Ideally the test and training data distributions should be similar. The following metrics can help identify gaps between the test set and the training set:\n",
    "\n",
    "**1.**  The distribution of user examples per intent for the test data should be comparable to the training data   \n",
    "**2.**  The average length of user examples for test and training data should be comparable to the training data <br>\n",
    "**3.**  The vocabulary and phrasing of utterances in the test data should be comparable to the training data\n",
    "\n",
    "If your test data comprises of examples labelled from your logs, and the training data comprises of examples created by human subject matter experts, there may be discrepancies between what the virtual assistant designers thought the end users would type and the way they actually type in production. Thus, if you find discrepancies in this section, consider changing your design to resemble the way in which end users use your system more closely.\n",
    "\n",
    "**<font color=red>Note</font>**: You will be charged for calls made from this notebook  based on your WA plan. The user_id will be the same for all message calls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(divergence_analyzer)\n",
    "divergence_analyzer.analyze_train_test_diff(workspace_pd, test_df, full_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Determine the overall accuracy on the test set<a id='accuracy'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(inferencer)\n",
    "results = full_results[['correct_intent', 'top_confidence','top_intent','utterance']]\n",
    "accuracy = inferencer.calculate_accuracy(results)\n",
    "display(Markdown(\"### Accuracy on Test Data: {} %\".format(accuracy)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyze the errors<a id='errors'></a>\n",
    "\n",
    "This section gives you an overview of the errors made by the intent classifier on the test set.  \n",
    "\n",
    "**Note**: `System Out of Domain` labels are assigned to user examples which get classified with confidence scores less than 0.2 as Watson Assistant considers them to be irrelevant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(inferencer)\n",
    "wrongs_df = inferencer.calculate_mistakes(results)\n",
    "display(Markdown(\"### Intent Detection Mistakes\"))\n",
    "display(Markdown(\"Number of Test Errors: {}\".format(len(wrongs_df))))\n",
    "\n",
    "with pd.option_context('max_colwidth', 250):\n",
    "    if not wrongs_df.empty:\n",
    "        display(wrongs_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='part3'></a>\n",
    "# Part 3: Perform advanced analysis\n",
    "\n",
    "3.1 [Perform analysis using confidence thresholds](#part3.1)<br>\n",
    "3.2 [Analysis interpretation @ confidence level T](#levelT)<br>\n",
    "3.3 [Highlighting term importance](#part3.2)<br>\n",
    "3.4 [Analyzing abnormal confidence levels](#part3.3)<br>\n",
    "3.5 [Perform an analysis using correlated entities per intent](#part3.4)<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='part3.1'></a>\n",
    "## 3.1 Perform analysis using  confidence thresholds\n",
    "\n",
    "This analysis illustrates how a confidence threshold is used to determine which data considered irrelevant or out of domain can be used for analysis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(confidence_analyzer)\n",
    "analysis_df= confidence_analyzer.analysis(results,None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Analysis interpretation @ confidence level T <a id='levelT'></a>\n",
    "\n",
    "If a certain confidence threshold T is selected, then: \n",
    "- The on-topic accuracy for test examples which cross the threshold is ***TOA***\n",
    "- The percentage of total test examples which returns confidences higher than the threshold is measured as ***Bot Coverage %***\n",
    "- If out of domain examples exist, falsely accept out of domain examples as on topic examples at a rate measured by ***FAR*** (False Acceptance Rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analysis_df.index = np.arange(1, len(analysis_df)+1)\n",
    "display(analysis_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select the threshold value\n",
    "\n",
    "By selecting a higher threshold, you can potentially bias your systems so that they are more accurate in terms of determining whether an utterance is on topic or out of domain. The default confidence threshold for Watson Assistance is 0.2.  \n",
    "\n",
    "**Effect on accuracy**: When you select a higher threshold T, this can result in higher accuracy (TOA) because only examples with confidences greater than the threshold T are included.\n",
    "\n",
    "**Effect on bot coverage %**: However, when you select a higher threshold T, this can also result in the virtual assistant responding to less examples.\n",
    "\n",
    "**Deflection to human agent**: In the scenarios where the virtual assistant is setup to hand off to a human agent when it is less confident, having a higher threshold T can:  \n",
    "\n",
    "- Improve end user experience when interacting with a virtual assistant, as it continues interaction only when its highly confident\n",
    "- Result in higher costs to the customer as this can result in more deflections to the human agents \n",
    "\n",
    "Thus, there is a trade-off and you need to decide on a threshold value on a per customer basis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examine the threshold selection on individual intents\n",
    "This section allows the examination of thresholds on specific intents.\n",
    "\n",
    "- Use `INTENT_LIST = []` to get analysis which averages across all intents\n",
    "- Use `INTENT_LIST = ['intent1', 'intent2']` to examine specific intents and threshold analysis on these intents\n",
    "- Use `INTENT_LIST = ['ALL_INTENTS']` to examine all intents and threshold analysis for each\n",
    "- Use `INTENT_LIST = [MOST_FREQUENT_INTENT]` to get analysis on the intent with the most test examples (DEFAULT)\n",
    "\n",
    "**False Acceptance Rate (FAR) for specific intents**  \n",
    "When we calculate FAR across all intents (as in previous section) we calculate fraction of out of domain examples falsely considered on topic. When we calculate FAR for specific intents, we calculate the fraction of examples which were falsely predicted to be that specific intent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(confidence_analyzer)\n",
    "\n",
    "# Calculate intent with most test examples\n",
    "for label in list(test_df['intent'].value_counts().index):\n",
    "    if label != skills_util.OFFTOPIC_LABEL:\n",
    "        MOST_FREQUENT_INTENT = label \n",
    "        break\n",
    "        \n",
    "# Specify intents of interest for analysis      \n",
    "INTENT_LIST = [MOST_FREQUENT_INTENT]  \n",
    "\n",
    "analysis_df_list = confidence_analyzer.analysis(results, INTENT_LIST)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='part3.2'></a>\n",
    "## 3.3 Highlight term importance\n",
    "\n",
    "This intent can be ground-truth or an incorrectly predicted intent. It provides term level insights about which terms the classifier thought were important in relation to that specific intent.\n",
    "\n",
    "Even if the system predicts an intent correctly, the terms which the intent classifier thought were important may not be as expected by human insight. Human insight might suggest that the intent classifier is focusing on the wrong terms.  \n",
    "\n",
    "The score of each term in the following highlighted images can be viewed as importance factor of that term for that specific intent. The larger the score, the more important the term."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get the highlighted images for either wrongly-predicted utterances or utterances where the classifier returned a low confidence.   \n",
    "\n",
    "**<font color=red>Note</font>**: You will be charged for calls made from this notebook  based on your WA plan. The user_id will be the same for all message calls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(highlighter)\n",
    "\n",
    "# Pick an example from section 1 which was misclassified\n",
    "# Add the example and correct intent for the example\n",
    "utterance = \"where is the closest agent\"  # input example\n",
    "intent = \"General_Connect_to_Agent\"  # input an intent in your workspace which you are interested in.\n",
    "\n",
    "# increase timeout if you experience `TimeoutError`. \n",
    "# Increasing the `TIMEOUT` allows the process more breathing room to compete\n",
    "TIMEOUT = 1 # `TIMEOUT` is set to 1 second\n",
    "inference_results = inferencer.inference(conversation=conversation, \n",
    "                                    skill_id=skill_id, \n",
    "                                    test_data=pd.DataFrame({'utterance':[utterance], \n",
    "                                                            'intent':[intent]}), \n",
    "                                    max_thread = 1,\n",
    "                                    timeout=TIMEOUT\n",
    "                                    )\n",
    "\n",
    "highlighter.get_highlights_in_batch_multi_thread(conversation=conversation, \n",
    "                                                 full_results=inference_results, \n",
    "                                                 output_folder=None,\n",
    "                                                 confidence_threshold=1,\n",
    "                                                 show_worst_k=1,\n",
    "                                                 lang_util=lang_util,\n",
    "                                                 skill_id=skill_id,\n",
    "                                                )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the section below you analyze your test results and produce highlighting for the top 25 problematic utterances which were either mistakes or had confidences below the threshold that was set.    \n",
    "\n",
    "**<font color=red>Note</font>**: You will be charged for calls made from this notebook  based on your WA plan. The user_id will be the same for all message calls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(highlighter)\n",
    "\n",
    "# The output folder for generated images\n",
    "# Note modify this if you want the generated images to be stored in a different directory\n",
    "\n",
    "highlighting_output_folder = './highlighting_images/'\n",
    "if not os.path.exists(highlighting_output_folder):\n",
    "    os.mkdir(highlighting_output_folder)\n",
    "\n",
    "# The threshold the prediction needs to achieve below which  \n",
    "# it will be considered as `out of domain` or `offtopic` utterances. \n",
    "threshold = 0.2\n",
    "\n",
    "# Maximum number of test set examples whose highlighting analysis will be conducted\n",
    "K=25\n",
    "highlighter.get_highlights_in_batch_multi_thread(conversation=conversation, \n",
    "                                                 full_results=full_results, \n",
    "                                                 output_folder=highlighting_output_folder,\n",
    "                                                 confidence_threshold=threshold,\n",
    "                                                 show_worst_k=K,\n",
    "                                                 lang_util=lang_util,\n",
    "                                                 skill_id=skill_id,\n",
    "                                                )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='part3.3'></a>\n",
    "## 3.4 Analyze abnormal confidence levels\n",
    "Every test utterance is classified as a specific intent with a specific confidence by the Watson Assistant intent classifier. It is expected that model would be confident when it correctly predicts examples and not highly confident when it incorrectly predicts examples. \n",
    "\n",
    "But this is not always true. This can be because there are anomalies in the design. Examples that are predicted correctly with low confidence and the examples that are predicted incorrectly with high confidence are cases which need to be reviewed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(confidence_analyzer)\n",
    "correct_thresh, wrong_thresh = 0.3, 0.7\n",
    "correct_with_low_conf_list, incorrect_with_high_conf_list = confidence_analyzer.abnormal_conf(\n",
    "    full_results, correct_thresh, wrong_thresh)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if len(correct_with_low_conf_list) > 0:\n",
    "    display(Markdown(\"#### Examples correctedly predicted with low confidence\"))\n",
    "    with pd.option_context('max_colwidth', 250):\n",
    "        display(HTML(correct_with_low_conf_list.to_html(index=False)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if len(incorrect_with_high_conf_list) > 0:\n",
    "    display(Markdown(\"#### Examples incorrectedly predicted with high confidence\"))\n",
    "    with pd.option_context('max_colwidth', 250):\n",
    "        display(HTML(incorrect_with_high_conf_list.to_html(index=False)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Actions to take when you have examples of abnormal confidence\n",
    "\n",
    "If there are examples which are incorrectly classified with high confidence for specific intents, it may indicate an issue in the design of those specific intents because the user examples provided for that intent may be overlapping with the design of other intents.\n",
    "\n",
    "If intent A seems to always get misclassified as intent B with high confidence or gets correctly predicted with low confidence, consider using intent conflict detection. For more information, refer to the <a href=\"https://cloud.ibm.com/docs/services/assistant?topic=assistant-intents#intents-resolve-conflicts\" target=\"_blank\" rel=\"noopener no referrer\">Conflict Resolution Documentation</a>.\n",
    "\n",
    "Also consider whether those two intents need to be two separate intents or whether they need to be merged. If they can't be merged, then consider adding more user examples which distinguish intent A specifically from intent B."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='part3.4'></a>\n",
    "## 3.5 Perform an analysis using correlated entities per intent\n",
    "\n",
    "Perform a chi square significance test for entities such as we or you for unigrams and bigrams in the previous section. For each utterance in the training data, this analysis will call the message API for entity detection on each utterance and find the most correlated entities for each intent.\n",
    "\n",
    "**<font color=red>Note</font>**: You will be charged for calls made from this notebook  based on your Watson Assistant plan. The user_id will be the same for all message calls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(entity_analyzer)\n",
    "importlib.reload(inferencer)\n",
    "if entities_list:\n",
    "    THREAD_NUM = min(4, os.cpu_count() if os.cpu_count() else 1)\n",
    "    # increase timeout if you experience `TimeoutError`. \n",
    "    # Increasing the `TIMEOUT` allows the process more breathing room to compete\n",
    "    TIMEOUT = 1 # `TIMEOUT` is set to 1 second\n",
    "    train_full_results = inferencer.inference(conversation=conversation,\n",
    "                                              test_data=workspace_pd, \n",
    "                                              max_thread=THREAD_NUM,\n",
    "                                              skill_id=skill_id,\n",
    "                                              timeout=TIMEOUT\n",
    "                                             )\n",
    "    entity_label_correlation_df = entity_analyzer.entity_label_correlation_analysis(\n",
    "        train_full_results, entities_list)\n",
    "    with pd.option_context('display.max_colwidth', 200):\n",
    "        entity_label_correlation_df.index = np.arange(1, len(entity_label_correlation_df) + 1)\n",
    "        display(entity_label_correlation_df)\n",
    "else:\n",
    "    display(Markdown(\"### Target workspace has no entities.\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='part4'></a>\n",
    "## Part 4: Summary\n",
    "Congratulations! You have successfully completed the dialog skill analysis training. <br>\n",
    "This notebook is designed to improve our dialog skill analysis in an iterative fashion. Use it to tackle one aspect of your dialog skill at a time and start over for another aspect later for continuous improvement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Glossary\n",
    "\n",
    "**True Positives (TP):** True Positive measures the number of correctly predicted positive values meaning that predicted class is the same as the actual class which is the target intent.\n",
    "\n",
    "**True Negatives (TN):** True Negative measures the number of correctly predicted negative values meaning that the predicted class is the same as the actual class which is not the target intent.\n",
    "\n",
    "**False Positives (FP):** False Positive measures the number of incorrectly predicted positive values meaning that the predicted class is the target intent but the actual class is not the target intent.  \n",
    "\n",
    "**False Negatives (FN):** False Negatives measures the number of incorrectly predicted negative values meaning that the predicted class is not the target intent but the actual class is the target intent. \n",
    "\n",
    "**Accuracy:** Accuracy measures the ratio of corrected predicted user examples out of all user examples.   \n",
    "Accuracy = (TP + TN) / (TP + TN + FP + FN)  \n",
    "\n",
    "**Precision:** Precision measures the ratio of correctly predicted positive observations out of total predicted positive observations.   \n",
    "Precision = TP / (TP + FP)  \n",
    "\n",
    "**Recall:** Recall measures the ratio of correctly predicted positive observations out of all observations of the target intent.  \n",
    "Recall = TP / (TP + FN)\n",
    "\n",
    "**F1 Score:** F1 Score is the harmonic average of Precision and Recall.  \n",
    "F1 = 2 \\* (Precision \\* Recall)/ (Precision + Recall)\n",
    "\n",
    "For more information related to Watson Assistant, refer to the <a href=\"https://cloud.ibm.com/docs/services/assistant\" target=\"_blank\" rel=\"noopener no referrer\">Watson Assistant Documentation</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Authors\n",
    "\n",
    "**Haode Qi** is a data scientist at IBM Watson who delivers new machine learning algorithms into IBM Watson's market leading conversational AI service. He works with clients to help improve their conversational AI agents and helps them tackle complex challenges at scale with tools like Dialog Skill Analysis. His work primarily focuses on natural language technology with focuses on intent recognition and multingual technology.\n",
    "\n",
    "**Navneet Rao** is an engineering lead at IBM Watson who believes in building unique AI-powered experiences which augment human capabilities. He currently works on AI innovation & research for IBM's award-winning conversational computing platform, the IBM Watson Assistant. His primary areas of interest include machine learning problems related to conversational AI, natural language understanding, semantic search & transfer learning.\n",
    "\n",
    "**Ming Tan**, PhD, is a research scientist at IBM Watson who works on prototyping and productizing various algorithmic features for the IBM Watson Assistant. His research interests include a broad spectrum of problems related to conversational AI such as low-resource intent classification, out-of-domain detection, multi-user chat channels, passage-level semantic matching and entity detection. His work has been published at various top tier NLP conferences.\n",
    "\n",
    "**Yang Yu**, PhD, is a research scientist at IBM Watson focusing on problems related to language understanding, question answering, deep learning and representation learning for various NLP tasks. He has been awarded by IBM for his contributions to several internal machine learning competitions which have included researchers from across the globe. Novel machine learning solutions designed by him have helped solve critical question answering and human-computer dialog problems for various IBM Watson products."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "Copyright &copy; IBM Corp. 2019. This notebook and its source code are released under the terms of the Apache License, Version 2.0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background:#F5F7FA; height:110px; padding: 2em; font-size:14px;\">\n",
    "<span style=\"font-size:18px;color:#152935;\">Love this notebook? </span>\n",
    "<span style=\"font-size:15px;color:#152935;float:right;margin-right:40px;\">Don't have an account yet?</span><br>\n",
    "<span style=\"color:#5A6872;\">Share it with your colleagues and help them discover the power of Watson Studio!</span>\n",
    "<span style=\"border: 1px solid #3d70b2;padding:8px;float:right;margin-right:40px; color:#3d70b2;\"><a href=\"https://ibm.co/wsnotebooks\" target=\"_blank\" style=\"color: #3d70b2;text-decoration: none;\">Sign Up</a></span><br>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}