notebooks/Dialog Flow Analysis Notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Watson Assistant Dialog Flow Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/watson-developer-cloud/assistant-dialog-flow-analysis/master/notebooks/images/flow-vis.png\" width=\"70%\"> \n",
    "\n",
    "## Introduction\n",
    "\n",
    "This notebook demonstrates the use of visual analytics tools that help you measure and understand user journeys within the dialog logic of your Watson Assistant system, discover where user abandonments take place, and reason about possible sources of issues. The visual interface can also help to better understand how users interact with different elements of the dialog flow, and gain more confidence in making changes to it. The source of information is your Watson Assistant skill definitions and conversation logs.\n",
    "\n",
    "As described in <a href=\"https://github.com/watson-developer-cloud/assistant-improve-recommendations-notebook/raw/master/notebook/IBM%20Watson%20Assistant%20Continuous%20Improvement%20Best%20Practices.pdf\" target=\"_blank\" rel=\"noopener noreferrer\">Watson Assistant Continuous Improvement Best Practices</a>, you can use this notebook to measure and understand in detail the behavior of users in areas that are not performing well, e.g. having low **Task Completion Rates**.\n",
    "\n",
    " \n",
    ">**Task Completion Rate** - is the percentage of user journeys within key tasks/flows of your virtual assistant that reach a successful resolution. This metric is one of the metrics you can use to measure the **Effectiveness** of your assistant.\n",
    "\n",
    "  \n",
    "### Prerequisites \n",
    "This notebook assumes some familiarity with the Watson Assistant dialog programming model, such as skills (formerly workspaces), and dialog nodes. Some familiarity with Python is recommended. This notebook runs on Python 3.5.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. [Installation and setup](#setup)<br>\n",
    "\n",
    "2. [Load Assistant Skills and Logs](#load)<br>\n",
    "    2.1 [Load option one: from a Watson Assistant instance](#load_api)<br>\n",
    "    2.2 [Load option two: from JSON files](#load_file)<br>\n",
    "    2.3 [Load option three: from IBM Cloud Object Storage (using Watson Studio)](#load_cos_studio)<br>\n",
    "    2.4 [Load option four: from custom location](#load_custom)<br>\n",
    "\n",
    "3. [Extract and transform](#extract_transform)<br>\n",
    "\n",
    "4. [Visualizing user journeys and abandonments](#dialog_flow)<br>\n",
    "    4.1 [Visualize dialog flow (turn-based)](#flow_turn_based)<br> \n",
    "    4.1.1. [Visualize flows in all conversations](#flows_all_conversations)<br>\n",
    "    4.1.2. [Visualize flows in subset of conversations](#flow_subset_conversations)<br>\n",
    "    4.1.3. [Analyzing reverse flows](#flow_reverse)<br>\n",
    "    4.1.4. [Analyzing trends in flows](#flow_trend)<br>\n",
    "    4.2 [Visualize dialog flow (milestone-based)](#flow_milestone_based)<br> \n",
    "    4.3 [Select conversations at point of abandonment](#flow_selection)<br>\n",
    "\n",
    "5. [Analyzing abandoned conversations](#analyze_abandonment)<br>\n",
    "    5.1 [Explore conversation transcripts for qualitative analysis](#transcript_vis)<br>\n",
    "    5.2 [Identify key words and phrases at point of abandonment](#keywords_analysis)<br>\n",
    "    5.2.1 [Summarize frequent keywords and phrases](#keywords_analysis_summarize)<br>\n",
    "\n",
    "6. [Measuring high level tasks of the Assistant](#tasks)<br>\n",
    "\n",
    "7. [Advanced Topics](#advanced_topics)<br>\n",
    "    7.1 [Locating important dialog nodes in your assistant](#search)<br>\n",
    "    7.1.1 [Searching programatically](#search_programatically)<br>\n",
    "    7.1.2 [Interactive Search and Exploration](#search_visually)<br>\n",
    "    7.2 [Filtering](#filtering)<br>\n",
    "    7.3 [Advanced keyword analysis: Comparing abandoned vs. Successful conversations](#keywords_analysis_compare)<br>\n",
    "\n",
    "8. [Summary and next steps](#summary)<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"setup\"></a>\n",
    "## 1. Configuration and Setup\n",
    "\n",
    "In this section, we install and import relevant python modules. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"install\"></a>\n",
    "#### Install required Python libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note, on Watson Studio the pip magic command `%pip` is not supported from within the notebook.  Use !pip instead. \n",
    "#!pip install --user conversation_analytics_toolkit\n",
    "%pip install --user conversation_analytics_toolkit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download('words')\n",
    "nltk.download('punkt')\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"import\"></a>\n",
    "#### 1. Import required modules "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import conversation_analytics_toolkit\n",
    "from conversation_analytics_toolkit import wa_assistant_skills\n",
    "from conversation_analytics_toolkit import transformation\n",
    "from conversation_analytics_toolkit import filtering2 as filtering\n",
    "from conversation_analytics_toolkit import analysis \n",
    "from conversation_analytics_toolkit import visualization \n",
    "from conversation_analytics_toolkit import selection as vis_selection\n",
    "from conversation_analytics_toolkit import wa_adaptor \n",
    "from conversation_analytics_toolkit import transcript \n",
    "from conversation_analytics_toolkit import flows \n",
    "from conversation_analytics_toolkit import keyword_analysis \n",
    "from conversation_analytics_toolkit import sentiment_analysis \n",
    "\n",
    "import json\n",
    "import pandas as pd\n",
    "from pandas.io.json import json_normalize\n",
    "from IPython.core.display import display, HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"configure\"></a>\n",
    "#### 2. Configure the notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set pandas to show more rows and columns\n",
    "import pandas as pd\n",
    "#pd.set_option('display.max_rows', 200)\n",
    "pd.set_option('display.max_colwidth', -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"load\"></a>\n",
    "## 2. Load Assistant Skills and Logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img style=\"float:right;margin-left: 10px\" src=\"https://raw.githubusercontent.com/watson-developer-cloud/assistant-dialog-flow-analysis/master/notebooks/images/load.png\" width=\"40%\">\n",
    "The analytics in this notebook are based on two main artifacts: \n",
    "\n",
    "1. **Assistant logs** - logs generated from the communication with Watson Assistant via the /message API;\n",
    "2. **Assistant skill(s)** - the skills that comprise the virtual assistant (aka [\"workspace\"](https://cloud.ibm.com/apidocs/assistant#get-information-about-a-workspace)).\n",
    "<br><br>\n",
    "\n",
    "These artifacts can be loaded from multiple sources, such as directly from Watson Assistant [log](https://cloud.ibm.com/apidocs/assistant#list-log-events-in-a-workspace) or [message](https://cloud.ibm.com/apidocs/assistant#get-response-to-user-input) APIs, or from other locations such as local/remote file system, Cloud Object Storage, or a Database.  \n",
    "\n",
    "**Note**: below are a set of options to load workspace(s) and log data from different sources into the notebook. Use only **one** of these methods, and then skip to [section 2.2](#organize_skills) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"load_api\"></a>\n",
    "## 2.1 Load option one: from a Watson Assistant instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.1 Add Watson Assistant configuration\n",
    "\n",
    "This notebook uses the Watson Assistant v1 API to access your skill definition and your logs. Provide your Watson Assistant credentials and the workspace id that you want to fetch data from.\n",
    "\n",
    "You can access the values you need for this configuration from the Watson Assistant user interface. Go to the Skills page and select View API Details from the menu of a skill tile.\n",
    "\n",
    "- The string to set in the call to `IAMAuthenticator` is your API Key under Service Credentials\n",
    "- The string to set for version is a date in the format version=YYYY-MM-DD. The version date string determines which version of the Watson Assistant V1 API will be called. For more information about version, see [Versioning](https://cloud.ibm.com/apidocs/assistant/assistant-v1#versioning).\n",
    "- The string to pass into `service.set_service_url` is the portion of the Legacy v1 Workspace URL that ends with `/api`. For example, `https://gateway.watsonplatform.net/assistant/api`. This value will be different depending on the location of your service instance. Do not pass in the entire Workspace URL.\n",
    "\n",
    "For Section 2.1.2, the value of `workspace_id` can be found on the same View API Details page. The value of the Skill ID can be used for the workspace_id variable. If you are using versioning in Watson Assistant, this ID represents the Development version of your skill definition.\n",
    "\n",
    "For more information about authentication and finding credentials in the Watson Assistant UI, please see [Watson Assistant v1 API](https://cloud.ibm.com/apidocs/assistant/assistant-v1) in the offering documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ibm_watson\n",
    "from ibm_watson import AssistantV1\n",
    "from ibm_cloud_sdk_core.authenticators import IAMAuthenticator\n",
    "\n",
    "authenticator = IAMAuthenticator(\"<YOUR APIKEY>\")\n",
    "service = AssistantV1(version='2019-02-28',authenticator = authenticator)\n",
    "service.set_service_url(\"<YOUR SERVICE URL>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.2 Fetch and load a workspace\n",
    "Fetch the workspace for the workspace id given in `workspace_id` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#select a workspace by specific id\n",
    "workspace_id = ''\n",
    "# or fetch one via the APIs\n",
    "# workspaces=service.list_workspaces().get_result()\n",
    "# workspace_id = service['workspaces'][0]['workspace_id']\n",
    "\n",
    "#fetch the workspace\n",
    "workspace=service.get_workspace(\n",
    "    workspace_id=workspace_id,\n",
    "    export=True\n",
    ").get_result()\n",
    "\n",
    "# set query parameters\n",
    "limit_number_of_records=5000\n",
    "# example of time range query\n",
    "query_filter = \"response_timestamp>=2019-10-30,response_timestamp<2019-10-31\"\n",
    "#query_filter = None\n",
    "# Fetch the logs for the workspace\n",
    "df_logs = wa_adaptor.read_logs(service, workspace_id, limit_number_of_records, query_filter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"load_file\"></a>\n",
    "## 2.2 Load option two: from JSON files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "# this example uses Watson Assistant data sample on github\n",
    "\n",
    "# pull sample workspace from watson developer cloud\n",
    "response = requests.get(\"https://raw.githubusercontent.com/watson-developer-cloud/assistant-dialog-flow-analysis/master/data/banking-sample/wa-workspace.json\").text    \n",
    "workspace = json.loads(response)\n",
    "\n",
    "# NOTE: the workspace_id is typically available inside the workspace object.  \n",
    "# If you've used the `export skill` feature in Watson Assistant UI, you can find the skill id \n",
    "# by clicking the `skill`-->`View API details` and copying the value of skill_id  \n",
    "workspace_id = workspace[\"workspace_id\"] \n",
    "#workpace_id = ''\n",
    "\n",
    "# pull logs sample from watson develop cloud\n",
    "response = requests.get(\"https://raw.githubusercontent.com/watson-developer-cloud/assistant-dialog-flow-analysis/master/data/banking-sample//wa-logs.json\").text\n",
    "df_logs = pd.DataFrame.from_records(json.loads(response))\n",
    "print(\"loaded {} log records\".format(str(len(df_logs))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"load_cos_studio\"></a>\n",
    "## 2.3 Load option three: from IBM Cloud Object Storage (using Watson Studio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# @hidden_cell\n",
    "# The project token is an authorization token that is used by Watson Studio to access project resources. \n",
    "# For more details on project tokens, refer to https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/token.html\n",
    "from project_lib import Project\n",
    "project = Project(project_id='<replace by Watson Studio project id>', project_access_token='<replace by Watson Studio access token')\n",
    "\n",
    "workspace_file = \"wa-workspace.json\"\n",
    "log_files = \"wa-logs.json\"\n",
    "\n",
    "workspace = json.loads(project.get_file(workspace_file))\n",
    "df_logs = pd.DataFrame.from_records(json.loads(project.get_file(log_files)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"load_custom\"></a>\n",
    "## 2.4 Load option four: from custom location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Depending on your production environment, your logs and workspace files might be stored in different locations.\n",
    "# such as NoSQL databases, Cloud Object Storage files, etc.\n",
    "\n",
    "# use custom code here, and make sure you load the workspace as a python dictionary, and the df_logs as a pandas DataFrame.\n",
    "#workspace = \n",
    "#df_logs = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"extract_transform\"></a>\n",
    "## 3 Extract and Transform  \n",
    "<img src=\"https://raw.githubusercontent.com/watson-developer-cloud/assistant-dialog-flow-analysis/master/notebooks/images/extract-transform.png\" width=\"50%\"> \n",
    "During this step, relevant information is extracted from the assistant logs, and combined with information from the assistant skills to create single (\"canonical\") dataframe for the analyses in this notebook.  \n",
    "\n",
    "**Note**: If your logs are in a custom format or you wish to extract additional fields, you may need to customize the extraction process.  You can learn more about this topic <a href=\"https://github.com/watson-developer-cloud/assistant-dialog-flow-analysis/blob/master/FAQ.md#what-is-the-format-of-the-canonical-data-model\" target=\"_blank\">here</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"organize_skills\"></a>\n",
    "#### Step 1: Prepare skills\n",
    "\n",
    "Create the **WA_Assistant_Skills** class, to organize relevant workspace objects for the analysis.  The **Extract and Transform** phase will use this class to match the workspace_id column in your logs, and fetch additional relevant information from the skill.  \n",
    "\n",
    "Call the `add_skill()` function to add relevant workspaces for the analysis.  You can add multiple workspaces that correspond to different versions of a skill, or multiple skills of a **multi-skill** assistant (e.g. if you have your own custom code that routes messages to different dialog skills and want to analyze the collection of skills together)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#if you have more than one skill, you can add multiple skill definitions\n",
    "skill_id = workspace_id\n",
    "assistant_skills = wa_assistant_skills.WA_Assistant_Skills()\n",
    "assistant_skills.add_skill(skill_id, workspace)\n",
    "#validate the number of workspace_ids\n",
    "print(\"workspace_ids in skills: \" + pd.DataFrame(assistant_skills.list_skills())[\"skill_id\"].unique())\n",
    "print(\"workspace_ids in logs: \"+ df_logs.workspace_id.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 2: Extract and Transform  \n",
    "Call the `to_canonical_WA_v2()` function to perform the extract and transform steps\n",
    "\n",
    "If your assistant is **multi-skill**, set `skill_id_field=\"workspace_id\"` to link information in the logs, with the corresponding workspace object based on the value of the `workspace_id` attribute in the logs.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_logs_canonical = transformation.to_canonical_WA_v2(df_logs, assistant_skills, skill_id_field=None, include_nodes_visited_str_types=True, include_context=True)\n",
    "#df_logs_canonical = transformation.to_canonical_WA_v2(df_logs, assistant_skills, skill_id_field=\"workspace_id\", include_nodes_visited_str_types=True, include_context=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set this to True if you need to fill response_text with node options when response_type == \"options\"\n",
    "collect_options_response = False\n",
    "\n",
    "if collect_options_response:\n",
    "    df_logs_canonical[\"response_text\"] = df_logs_canonical.apply(\n",
    "    (\n",
    "        lambda row: transformation.collect_option_response_text(workspace[\"dialog_nodes\"], row[\"nodes_visited\"]) \n",
    "        if len(row[\"response_text\"]) == 0 else row[\"response_text\"]\n",
    "    ), \n",
    "    axis=1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the rest of the notebook runs on the df_logs_to_analyze object.  \n",
    "df_logs_to_analyze = df_logs_canonical.copy(deep=False)\n",
    "df_logs_to_analyze.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"dialog_flow\"></a>\n",
    "## 4. Visualizing user journeys and abandonments\n",
    "\n",
    "The dialog flow visualization is an interactive tool for investigating user journeys, visits and abandonments within the steps of the dialog system. \n",
    "\n",
    "The visualization aggregates the temporal sequences of steps from Watson Assistant logs.  The interaction allows to explore the distribution of visits across dialog steps and where abandonment takes place, and select the conversations that visit a certain step for further exploration and analysis.  \n",
    "\n",
    "You can use the visualization to:\n",
    "* **understand end-to-end journeys** with the complete log dataset\n",
    "* **understand sub-journeys of interest** by using filters\n",
    "* **understand which flows end at a certain point** by using reverse flow aggregation\n",
    "* **undersatnd how flows are trending** by using merging the flows of two consecutive time periods\n",
    "\n",
    "This notebook demonstrates how to construct the visualization at two levels of abstraction\n",
    "* **turn-based** - for measuring and discovering how conversations progress across every turn in the conversation.\n",
    "* **milestone-based** - for measuring how conversations progress between key points of interest (\"milestones\") within the conversation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img style=\"float: right;margin: 13px;\" src=\"https://raw.githubusercontent.com/watson-developer-cloud/assistant-dialog-flow-analysis/master/notebooks/images/turn-flow.png\" width=\"40%\"> \n",
    "<a id=\"flow_turn_based\"></a>\n",
    "## 4.1 Visualize dialog flow (turn-based)\n",
    "Use the `draw_flowchart` function to create a flow-based visualization on the canonical dataset.  By default, the visualization will present the progression of conversations on a turn-by-turn basis, aggregated and sorted by visit sequence frequencies.  You can control many of the default, by modifying the `config` object.  \n",
    "\n",
    "The following mouse interactions are supported:\n",
    "* **click** on a node to expand/collapse its children\n",
    "* **drag** to pan\n",
    "* **scroll** to zoom in/out\n",
    "\n",
    "When you hover or click on a node the following information is displayed: \n",
    "* **visits:** the number of conversations that progressed along a specific path of conversation steps.\n",
    "* **drop offs:** the number of conversations that ended at this conversation step \n",
    "\n",
    "The examples below demonstrate how to create a turn-based flow analysis: (1) for all conversations in the dataset; and (2) for a subset of conversations that were escalated (denoted by visiting the node **Transfer to Live Agent**)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"flows_all_conversations\"></a>\n",
    "### 4.1.1. Visualize flows in all conversations   \n",
    "Visualizing all conversations on a turn-by-turn basis can help you to discover all existing conversation flows in your assistant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "title = \"All Conversations\"\n",
    "turn_based_path_flows = analysis.aggregate_flows(df_logs_to_analyze, mode=\"turn-based\", on_column=\"turn_label\", max_depth=400, trim_reroutes=False)\n",
    "# increase the width of the Jupyter output cell   \n",
    "display(HTML(\"<style>.container { width:95% !important; }</style>\"))\n",
    "config = {\n",
    "    'commonRootPathName': title, # label for the first root node \n",
    "    'height': 800, # control the visualization height.  Default 600\n",
    "    'nodeWidth': 250, \n",
    "    'maxChildrenInNode': 6, # control the number of immediate children to show (and collapse rest into *others* node).  Default 5\n",
    "    'linkWidth' : 400,  # control the width between pathflow layers.  Default 360     'sortByAttribute': 'flowRatio'  # control the sorting of the chart. (Options: flowRatio, dropped_offRatio, flows, dropped_off, rerouted)\n",
    "    'sortByAttribute': 'flowRatio',\n",
    "    'title': title,\n",
    "    'mode': \"turn-based\"\n",
    "}\n",
    "jsondata = json.loads(turn_based_path_flows.to_json(orient='records'))\n",
    "visualization.draw_flowchart(config, jsondata, python_selection_var=\"selection\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"flow_subset_conversations\"></a>\n",
    "### 4.1.2. Visualize flows in subset of conversations\n",
    "Sometime you might want to explore a subset of conversations that meet a certain criteria, or look at the conversations only from a specific point onwards.  The example below shows how to filter conversations that pass through two specific dialog nodes.  \n",
    "\n",
    "For more details about using filters, please refer to [section 7.2](#filtering)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# filter the conversations that include escalation\n",
    "title2=\"Banking Card Escalated\"\n",
    "filters = filtering.ChainFilter(df_logs_to_analyze).setDescription(title2) \n",
    "# node with condition on the #Banking-Card_Selection (node_1_1510880732839) and visit the node \"Transfer To Live Agent\" (node_25_1516679473977)\n",
    "filters.by_dialog_node_id('node_1_1510880732839')\\\n",
    "       .by_dialog_node_id('node_25_1516679473977')\n",
    "filters.printConversationFilters() \n",
    "# get a reference to the dataframe.  Note: you can get access to intermediate dataframes by calling getDataFrame(index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_df = filters.getDataFrame()\n",
    "turn_based_path_flows = analysis.aggregate_flows(filtered_df, mode=\"turn-based\", on_column=\"turn_label\", max_depth=400, trim_reroutes=False)  \n",
    "config = {\n",
    "    'commonRootPathName': title2,  'title': title2,\n",
    "    'height': 800,  'nodeWidth': 250, 'maxChildrenInNode': 6, 'linkWidth' : 400, 'sortByAttribute': 'flowRatio',\n",
    "    'mode': \"turn-based\"\n",
    "}\n",
    "jsondata = json.loads(turn_based_path_flows.to_json(orient='records'))\n",
    "visualization.draw_flowchart(config, jsondata, python_selection_var=\"selection\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"flow_reverse\"></a>\n",
    "### 4.1.3. Analyzing reverse flows\n",
    "Sometime you might want to understand which flows lead to a certain outcome which is represented by some node of interest, e.g. escalation to a live human agent.  This can be achieved by passing `reverse=True` to the `aggregate_flow` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# filter only conversations that include Confirming Live Agent Transfer.  In addition, remove true nodes, which simplifies the journey\n",
    "#removeing the true node, helps to make the visualization simpler\n",
    "cf = filtering.ChainFilter(df_logs_canonical).setDescription(\"Escalated conversations\")\\\n",
    "    .by_turn_label(\"Confirming Live Agent Transfer\")\\\n",
    "    .remove_turn_by_label(\"true\")\\\n",
    "    .printConversationFilters()\n",
    "\n",
    "escalated_df = cf.getDataFrame()\n",
    "#addiginal remove duplicates consecutive states will also make the visualization simpler\n",
    "escalated_simplified_df = analysis.simplify_consecutive_duplicates(escalated_df, on_column=\"turn_label\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#note reverse=True in the aggregation step\n",
    "escalated_similified_reversed_df = analysis.aggregate_flows(escalated_simplified_df, mode=\"turn-based\",\\\n",
    "                                on_column=\"turn_label\", max_depth=30, trim_reroutes=False, reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "title3 =  'Escalation Flows (reversed)'\n",
    "config = {\n",
    "    'commonRootPathName': title3,  'title': title3,\n",
    "    'height': 800,  'nodeWidth': 250, 'maxChildrenInNode': 6, 'linkWidth' : 400, 'sortByAttribute': 'flowRatio',\n",
    "    'mode': \"turn-based\"\n",
    "}\n",
    "jsondata = json.loads(escalated_similified_reversed_df.to_json(orient='records'))\n",
    "visualization.draw_flowchart(config, jsondata, python_selection_var=\"selection\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"flow_trend\"></a>\n",
    "### 4.1.4. Analyzing trends in flows\n",
    "Occasionally you may want to observe how the flows change across time periods, for example, after you have improved the training of an intent, or the dialog steps, and want to observe the improvement.\n",
    "\n",
    "To achieve this, you should aggregate each time period separately and use the `merge_compare_flows` function.  This will create a union of the unique flows and the visualization will show you how the flow changes between the two periods.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we will simiulate this on the previous chart (reverse escalated flows) by comparing the first and second 1000 logs\n",
    "prev_period_df = escalated_simplified_df[0:1000]\n",
    "curr_period_df = escalated_simplified_df[1001:2000]\n",
    "\n",
    "prev_flows = analysis.aggregate_flows(prev_period_df, mode=\"turn-based\", on_column=\"turn_label\", max_depth=30, trim_reroutes=False, reverse=True)\n",
    "curr_flows = analysis.aggregate_flows(curr_period_df, mode=\"turn-based\", on_column=\"turn_label\", max_depth=30, trim_reroutes=False, reverse=True)\n",
    "\n",
    "df_logs_compare_periods = analysis.merge_compare_flows(curr_flows, prev_flows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "title3 =  'Trending flows (curr vs. prev)'\n",
    "config = {\n",
    "    'commonRootPathName': title3,  'title': title3,\n",
    "    'height': 800,  'nodeWidth': 250, 'maxChildrenInNode': 6, 'linkWidth' : 400, 'sortByAttribute': 'flowRatio',\n",
    "    'mode': \"turn-based\"\n",
    "}\n",
    "jsondata = json.loads(df_logs_compare_periods.to_json(orient='records'))\n",
    "visualization.draw_flowchart(config, jsondata, python_selection_var=\"selection\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"flow_milestone_based\"></a>\n",
    "## 4.2 Visualize dialog flow (milestone-based)\n",
    "Use the milestone-based dialog visualization to measure the flow of visits and abandonment between key points of interest (aka \"milestones\") in your assistant. \n",
    "\n",
    "The milestone-based visualization requires two extra steps: \n",
    "1. definition of the milestones names, and mapping of which dialog nodes map to each milestone. \n",
    "2. processing of the log data to find and filter log rows that are part of a milestone.  \n",
    "\n",
    "Use the `mode=\"milestone-based\"` to configure the flow aggregation and visualization steps.  The visualization uses a special **Other** node to model conversations that are flowing to other parts of the dialog which were not defined to be of interest in the milestone definitions.  \n",
    "\n",
    "In this notebook we demonstrate how to produce a milestone dialog flow key points of interest that are part of the **Schedule Appointment** task of the assistant.\n",
    " \n",
    "<img src=\"https://raw.githubusercontent.com/watson-developer-cloud/assistant-dialog-flow-analysis/master/notebooks/images/milestone-flow.png\" width=\"80%\"> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Define milestones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#define the milestones and corresponding node ids for the `Schedule Appointment` task\n",
    "milestone_analysis = analysis.MilestoneFlowGraph(assistant_skills.get_skill_by_id(skill_id))\n",
    "\n",
    "milestone_analysis.add_milestones([\"Appointment scheduling start\",  \"Schedule time\", \"Enter zip code\", \"Branch selection\", \n",
    "                    \"Enter purpose of appointment\", \"Scheduling completion\"])\n",
    "\n",
    "milestone_analysis.add_node_to_milestone(\"node_21_1513047983871\", \"Appointment scheduling start\")   \n",
    "milestone_analysis.add_node_to_milestone(\"handler_28_1513048122602\", \"Schedule time\")\n",
    "milestone_analysis.add_node_to_milestone(\"handler_31_1513048234102\", \"Enter zip code\") \n",
    "milestone_analysis.add_node_to_milestone(\"node_3_1517200453933\", \"Branch selection\") \n",
    " \n",
    "milestone_analysis.add_node_to_milestone(\"node_41_1513049128006\", \"Enter purpose of appointment\")\n",
    "milestone_analysis.add_node_to_milestone(\"node_43_1513049260736\", \"Scheduling completion\")  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Derive a new dataset, using enrichment & filtering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#enrich with milestone information - will add a column called 'milestone'\n",
    "milestone_analysis.enrich_milestones(df_logs_to_analyze)\n",
    "#remove all log records without a milestone\n",
    "df_milestones = df_logs_to_analyze[pd.isna(df_logs_to_analyze[\"milestone\"]) == False]\n",
    "#optionally, remove consecutive milestones for a more simplified flow visualization representation\n",
    "df_milestones = analysis.simplify_flow_consecutive_milestones(df_milestones)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Aggregate and visualize "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute the aggregate flows of milestones \n",
    "computed_flows= analysis.aggregate_flows(df_milestones, mode=\"milestone-based\", on_column=\"milestone\", max_depth=30, trim_reroutes=False)\n",
    "config = {\n",
    "    'commonRootPathName': 'All Conversations', # label for the first root node \n",
    "    'height': 800, # control the visualization height.  Default 600\n",
    "    'maxChildrenInNode': 6, # control the number of immediate children to show (and collapse the rest into *other* node).  Default 5\n",
    "#     'linkWidth' : 400,  # control the width between pathflow layers.  Default 360     '\n",
    "    'sortByAttribute': 'flowRatio', # control the sorting of the chart. (Options: flowRatio, dropped_offRatio, flows, dropped_off, rerouted)\n",
    "    'title': \"Abandoned Conversations in Appointment Schedule Flow\",\n",
    "    'showVisitRatio' : 'fromTotal', # default: 'fromTotal'.  'fromPrevious' will compute percentages from previous step,\n",
    "    'mode': 'milestone-based'\n",
    "}\n",
    "jsondata = json.loads(computed_flows.to_json(orient='records'))\n",
    "visualization.draw_flowchart(config, jsondata, python_selection_var=\"milestone_selection\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: The rest of this notebook will demonstrate selection and analysis on selections made in the milestone-based dialog flow (designated by setting the python_selection_var variable to `milestone_selection`. To select and analyze conversations from the turn-based dialog flow, set the variable to `selection` instead)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"flow_selection\"></a>\n",
    "## 4.3 Select conversations at the point of abandonment\n",
    "<img style=\"float: right;margin:15px;\" src=\"https://raw.githubusercontent.com/watson-developer-cloud/assistant-dialog-flow-analysis/master/notebooks/images/abandonment.png\" width=\"40%\"> \n",
    "Selecting nodes in the visualization enables you to explore and analyze abandoned conversations with a common conversation path. You can use the visualization to identify large volumes of conversations that users abandon at unexpected locations during the conversation.\n",
    "\n",
    "\n",
    "**Note:**\n",
    "> Selecting a node in the visualization will also copy the selection from the visualization into the variable designated by `python_selection_var`, thus making the selection available to other cells of this notebook.   \n",
    "\n",
    "Before you run the next cell, you will interact with the milestone dialog visualization above to select a portion of the dialog to analyze. First, interact with the milestone dialog visualization to observe visit frequencies and abandonments within the milestones of `Schedule Appointment`. Click on nodes to drill down and expand the next step in sequence. Navigate along this path:  `Appointment scheduling start`-->`Schedule time`-->`Enter zip code`-->`Branch selection` to observe a relative high proportion of abandonments that occur in the middle of the flow. Select the `Branch selection` node. Note the large volume and ratio of abandoned conversations. Now run the following cell to process conversations that were abandoned at your point of selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#the selection variable contains details about the selected node, and conversations that were abandoned at that point\n",
    "print(\"Selected Path: \",milestone_selection[\"path\"])\n",
    "#fetch the dropped off conversations from the selection  \n",
    "dropped_off_conversations = vis_selection.to_dataframe(milestone_selection)[\"dropped_off\"]\n",
    "print(\"The selection contains {} records, with a reference back to the converstion logs\".format(str(len(dropped_off_conversations))))\n",
    "dropped_off_conversations.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"analyze_abandonment\"></a>\n",
    "## 5. Analyzing abandoned conversations\n",
    "After selecting a large group of abandoned conversations and their corresponding log records, you can apply additional analyses to better understand why these conversations were lost.  \n",
    "\n",
    "Some possible reasons could be:\n",
    "* The assistant didn't anticipate or understand the user utterance. This means the dialog skill is not trained to “cover” the user utterance. Coverage is an important measure of how well your assistant is performing. See the [Watson Assistant Continuous Improvement Best Practices](https://github.com/watson-developer-cloud/assistant-improve-recommendations-notebook/raw/master/notebook/IBM%20Watson%20Assistant%20Continuous%20Improvement%20Best%20Practices.pdf) for details on how to measure and improve coverage \n",
    "* The options presented to the user were not relevant\n",
    "* The conversation didn't progress in the right direction\n",
    "* The assistant’s response wasn't accurate or relevant\n",
    "\n",
    "This section of the notebook demonstrates two visual components that can help you investigate the user utterances in the abandoned conversations:\n",
    "* The transcript visualization can show you every user utterance and assistant response until the point of abandonment.\n",
    "* The key words and phrases analysis can help find frequent terms in abandoned conversations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"transcript_vis\"></a>\n",
    "## 5.1 Explore conversation transcripts for qualitative analysis\n",
    "<img style=\"float: right;margin:13px;\" src=\"https://raw.githubusercontent.com/watson-developer-cloud/assistant-dialog-flow-analysis/master/notebooks/images/transcript-vis.png\" width=\"50%\"> \n",
    "The transcript visualization shows the full sequence of conversation steps in the selected group of conversations. You can view the user utterances as well as observe the assistant response and the dialog nodes that were triggered to produce those responses.\n",
    "\n",
    "Try to navigate to the 3rd conversation (conversation_id == `0Aw68rNq6kSGxaDurGG1NSf3c9LtLK3kurWm`) using the toggle buttons, and scroll down to view the user's last utterance before abandoning the conversation (where user utterance is `wrong map`). In this conversation, the assistant response in the previous step wasn't satisfactory and when the user communicated that to the assistant, the assistant didn't understand his utterance. This may indicate that some modification to the dialog logic might be needed to better respond in this situation, as well as the service itself might need to be fixed.\n",
    "\n",
    "You might want to check if this situation occurs in other conversations too. A complementary approach is to try to find frequent terms in user utterances and see how prevalent this is across all abandoned conversations (see [next section](#keywords_analysis) for details)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optionally enrich with sentiment information\n",
    "Adding sentiment will allow you to observe negative utterances more quickly in the transcripts. You can generate other type of analysis insights, by enriching the `insights_tag` column  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_logs_to_analyze = sentiment_analysis.add_sentiment_columns(df_logs_to_analyze) \n",
    "#create insights, and highlights annotation for the transcript visualization\n",
    "NEGATIVE_SENTIMENT_THRESHOLD=-0.15 \n",
    "df_logs_to_analyze[\"insights_tags\"] = df_logs_to_analyze.apply(lambda x: [\"Negative Sentiment\"] if x.sentiment < NEGATIVE_SENTIMENT_THRESHOLD else [], axis=1)\n",
    "df_logs_to_analyze[\"highlight\"] = df_logs_to_analyze.apply(lambda x: True if x.sentiment < NEGATIVE_SENTIMENT_THRESHOLD else False, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fetch the conversation records\n",
    "dropped_off_conversations = vis_selection.fetch_logs_by_selection(df_logs_to_analyze, dropped_off_conversations)\n",
    "# visualize using the transcript visualization \n",
    "dfc = transcript.to_transcript(dropped_off_conversations)\n",
    "config = {'debugger': True} \n",
    "visualization.draw_transcript(config, dfc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"keywords_analysis\"></a>\n",
    "## 5.2 Identify key words and phrases at point of abandonment\n",
    "\n",
    "<img style=\"float: right;\" src=\"https://raw.githubusercontent.com/watson-developer-cloud/assistant-dialog-flow-analysis/master/notebooks/images/keywords.png\" width=\"30%\"> \n",
    "The keywords and phrases analysis allows you to check if the phrase `wrong map` is prevalent in many of the abandoned conversations, or what are the most common words or phrases overall.\n",
    "\n",
    "The analysis performs some basic linguistic processing from a group of utterances, such as removal of stop words, or extraction of the base form of words, and then computes their frequencies. Frequencies for words that appear together in sequence (bi-grams, tri-grams) are also computed.\n",
    "\n",
    "\n",
    "Finally, the visualization displays the most frequent words and phrases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Gather utterances from abandoned conversations "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# gather user utterances from the dropped off conversations - last utterances and all utterances\n",
    "last_utterances_abandoned=vis_selection.get_last_utterances_from_selection(milestone_selection, df_logs_to_analyze)\n",
    "all_utterances_abandoned=vis_selection.get_all_utterances_from_selection(milestone_selection, df_logs_to_analyze)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"keywords_analysis_summarize\"></a>\n",
    "## 5.2.1 Summarize frequent keywords and phrases  \n",
    "Analyze the last user utterances prior to abandonment to potentially identify common issues at that point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# analyze the last user input before abandonment\n",
    "num_unigrams=10\n",
    "num_bigrams=15\n",
    "custom_stop_words=[\"would\",\"pm\",\"ok\",\"yes\",\"no\",\"thank\",\"thanks\",\"hi\",\"i\",\"you\"]\n",
    "data = keyword_analysis.get_frequent_words_bigrams(last_utterances_abandoned, num_unigrams,num_bigrams,custom_stop_words)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = {'flattened': True, 'width' : 800, 'height' : 500}\n",
    "visualization.draw_wordpackchart(config, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: in the visual above, the term `wrong map` appears quite often.  Other relevant keywords and phrases such as `error`, `map error`, `wrong location`, `wrong branches` are also observed.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"tasks\"></a>\n",
    "## 6. Measuring high level tasks of the Assistant\n",
    "<img style=\"float:right;margin-left: 10px\" src=\"https://raw.githubusercontent.com/watson-developer-cloud/assistant-dialog-flow-analysis/master/notebooks/images/flow-def.png\" width=\"60%\"> \n",
    "In some scenarios you might want to first measure the effectiveness of your assistant at key areas within the skill, before drilling into specific flows.  Measuring the volume and effectiveness of specific key areas, can help you detect trends around your release dates, as well as prioritize your improvement efforts.\n",
    "\n",
    "A conversation can be view and measured as being composed of one or more logical tasks (aka \"high level flows\"). This section of the notebook demonstrates measuring **transactional tasks** (aka **\"flows\"**) by defining their corresponding starting (parent) and successful ending dialog nodes.\n",
    "\n",
    "You can define and measure a task by providing a mapping to dialog nodes that correspond to the start and successful end of a task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Define tasks\n",
    "<a id=\"define_tasks\"></a>\n",
    "\n",
    "You can use the programmatic and interactive search options as showed in [section 7.1](#search) to locate and copy corresponding dialog node ids, and use them in the flows definition as shown below.\n",
    "\n",
    "\n",
    "The example below shows how to define the **Credit card payment** and **Schedule appointments** tasks.  In this example, the starting point was mapped to the node that has a condition on the corresponding intent, and the completion nodes to the nodes that generate the confirmation response.\n",
    "\n",
    "**Note:**\n",
    ">Parent and completion nodes can be defined using one or more nodes. Defining multiple parent nodes is useful if a single logical flow is implemented across different branches of the dialog tree. Defining multiple completion nodes can be relevant if you have more than one location in the dialog that can determine successful ending of the flow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a flow is defined by a name, one or more \"starting/parent_nodes\" and one or more \"success/completion nodes\".   \n",
    "# All the nodes which are descendants to the parent nodes are considered to be part of the flow\n",
    "# A flow is considered successful if reaches the completion node\n",
    "flow_defs_initial = {\n",
    "  'flows': [{\n",
    "              'name': 'Credit card payment',\n",
    "              'parent_nodes': ['node_3_1511326054650'], #condition on #Banking-Billing_Payment_Enquiry || #Banking-Billing_Making_Payments\n",
    "              'completion_nodes': ['node_8_1512531332315'] # Display of confirmation \"Thank you for your payment...\" \n",
    "            },\n",
    "            {\n",
    "              'name': 'Schedule appointment',\n",
    "              'parent_nodes': ['node_21_1513047983871'], #condition on '#Business_Information-Make_Appointment'\n",
    "              'completion_nodes': ['node_43_1513049260736'] #Display Appointment Confirmation\n",
    "            }]\n",
    "}\n",
    "#create a list of all the nodes that map to a flow including descendant nodes\n",
    "flow_defs = flows.enrich_flows_by_workspace(flow_defs_initial, workspace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.  Measure task volume and completion rates\n",
    "<a id=\"measure_tasks\"></a>\n",
    "Using the task's flow definition and enrichment of the logs, we can now measure the visits in each flow, and the completion percentages.\n",
    "\n",
    "The **Abandoned** state refers to conversation that terminated in the middle of the flow.  **Rerouted** refers to conversation that left the scope of the flow and didn't return.  **Completed** refers to conversations that successfully reached the completion point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#enrich the logs dataframe with additional columns [\"flow\", \"flow_state\"] that represent the state of the flow\n",
    "df_logs_to_analyze = flows.enrich_canonical_by_flows(df_logs_to_analyze, flow_defs)\n",
    "flow_outcome_summary = flows.count_flows(df_logs_to_analyze, flow_defs)\n",
    "print(flow_outcome_summary) \n",
    "flows.plot_flow_outcomes(flow_outcome_summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: as shown in above figure, the **Schedule Appointment** task has a relatively large volume of conversations with poor **effectiveness** (as 65% of conversations are abandoned). As a next step you might want to drill down to understand in more detail where exactly in the dialog logic the conversations were abandoned and why.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"advanced_topics\"></a>\n",
    "## 7. Advanced Topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"search\"></a>\n",
    "## 7.1 Locating important dialog nodes in your assistant\n",
    "\n",
    "In order to measure the performance of specific tasks, or understand user journeys between specific points of the dialog, you will need to be able to reference dialog nodes by their unique `node_id`.  This section demonstrates how to find the `node_id` of nodes of interest in your dialog using two complementary techniques: a programmatic API, and an interactive visual component.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"search_programatically\"></a>\n",
    "## 7.1.2 Searching programmatically\n",
    "\n",
    "The `WA_Assistant_Skills` class provides utility functions for searching dialog nodes in your assistant or in a specific skill.\n",
    "\n",
    "The `re_search_in_dialog_nodes()` supports a case-insensitive, regular expression-based search.  You can search for strings that appears in the node's title, condition, or id. \n",
    "\n",
    "Sample usage of the API:\n",
    "* `re_search_in_dialog_nodes(search_term)` - search in all fields, in all skills\n",
    "* `re_search_in_dialog_nodes(search_term, keys=['condition'], in_skill=skill_id)` - search only in condition fields of nodes in specific skill \n",
    "\n",
    "Examples of search terms:\n",
    "* `\"card\"` - search for a word \n",
    "* `\"@CC_Types\"` - search for an entity\n",
    "* `\"#General_Conversation-VA_State\"` - search for an intent\n",
    "* `'#.*banking.*card'` - search for intent that includes banking and card"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example of searching for all occurences of the word 'Card'\n",
    "search_term='Card'\n",
    "results = assistant_skills.re_search_in_dialog_nodes(search_term)\n",
    "results.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img style=\"float: right;margin: 35px;\" src=\"https://raw.githubusercontent.com/watson-developer-cloud/assistant-dialog-flow-analysis/master/notebooks/images/search.png\" width=\"40%\">\n",
    "\n",
    "<a id=\"search_visually\"></a>\n",
    "## 7.1.3 Interactive search and exploration \n",
    " \n",
    "You can use the `draw_wa_dialog_chart()` to visualize the dialog nodes of a specific skill in the same tree layout as in Watson Assistant Dialog Editor.  You can interact with the visualization to navigate to, or search for, a particular node, from which you can copy its `node_id`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workspace = assistant_skills.get_skill_by_id(skill_id)\n",
    "data = {\n",
    "    'workspace': workspace\n",
    "}\n",
    "config = {}\n",
    "visualization.draw_wa_dialog_chart(config, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"filtering\"></a>\n",
    "## 7.2 Filtering\n",
    "You can use a built-in filter to narrow down your log records and focus on specific conversations or journey steps.  There are two types of filters\n",
    "1. **selection** - filters that select a subset of conversations based on some criteria.  Example of selection filters: `by_dialog_node_id`, `by_dialog_not_node_id`, `by_turn_label`, `by_date_range`, `by_dialog_node_str`, `by_column_values`\n",
    "2. **selection and trimming** - filters that select a subset of conversations and in addition remove some of the conversation steps before a specific point of interest. This allows you to focus on how conversation journeys progress from that point on.  Example of these filters: `trim_from_node_id`, `trim_from_turn_label`\n",
    "3. **removing turns** - filters that remove some of the conversation steps based on some criteria.  This allows you to simplify journeys, by removing steps that don't add value to your exploration, e.g. true nodes.  Example of selection filters: `remove_turn_by_label`.\n",
    "4. **simplification of journeys** - remove duplicate consecutive turns.  This allows you to see a more simplified verstion of the journeys when the same turn is repeated several times in the journey.   Note, this filter is not part of the ChainFilter module and can be invoked by calling the `analysis.simplify_consecutive_duplicates` function.\n",
    "\n",
    "You can create a chain of filters to work in sequence to narrow down the log records for specific exploration activities.  \n",
    "\n",
    "Below is an example of a chained filter that finds conversations that pass through the 'Collect Appointment Data' node during Jan 2020"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "import pytz\n",
    "\n",
    "filters = filtering.ChainFilter(df_logs_to_analyze).setDescription(\"Filter: collect Appointment Data during Jan 2020\") \n",
    "filters.by_dialog_node_id('node_22_1513048049461') #  corresponding to 'Collect Appointment Data' node. \n",
    " \n",
    "# You can use the search utilities described earlier in the notebook to find this node\n",
    "# You can also use cf.by_turn_label('Collect Appointment Data') to filter on information in the turn label\n",
    "\n",
    "start_date = datetime.datetime(2020, 1, 1, 0, 0, 0, 0, pytz.UTC)\n",
    "end_date = datetime.datetime(2020, 1, 31, 0, 0, 0, 0, pytz.UTC)\n",
    "filters.by_date_range(start_date,end_date)\n",
    "filters.printConversationFilters() \n",
    "\n",
    "# get a reference to the dataframe.  Note: you can get access to itermediate dataframes by calling getDataFrame(index)\n",
    "filtered_df = filters.getDataFrame()\n",
    "print(\"number of unique conversations in filtered dataframe: {}\".format(len(filtered_df[\"conversation_id\"].unique())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"keywords_analysis_compare\"></a>\n",
    "## 7.3 Advanced keyword analysis: Comparing abandoned vs. successful conversations\n",
    "Sometimes looking at the last utterances of the abandoned conversations is not enough to find the **root cause** of a problem.  A more advanced approach is to look also at the conversations that successfully completed the flow, and compare which keywords and phrases, are statistically associated more with the abandoned group, not only for the last utterance before the drop off point, but in general at all the utterances of the conversation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Gather utterances from all conversations that completed the journey on the same flow   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get the logs of conversations that continue to successful completion\n",
    "scheduling_completed_filter = filtering.ChainFilter(df_logs_to_analyze).setDescription(\"Appointement Scheduling flow - Completed\") \n",
    "scheduling_completed_filter.by_dialog_node_id('node_21_1513047983871')   # started the Appointment Scheduling flow \n",
    "scheduling_completed_filter.by_dialog_node_id('node_3_1517200453933')    # passed through the \"Branch selection\" node\n",
    "scheduling_completed_filter.by_dialog_node_id('node_43_1513049260736')  # reached the completion node of Scheduling Appointment flow\n",
    "scheduling_completed_filter.printConversationFilters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get the user utterances \n",
    "scheduling_completed_df = scheduling_completed_filter.getDataFrame()\n",
    "all_utterances_completed=scheduling_completed_df[scheduling_completed_df.request_text!=\"\"].request_text.tolist()\n",
    "print(\"Gathered {} utterances from {} successful journeys\".format(str(len(all_utterances_completed)), \n",
    "                                                                  str(len(scheduling_completed_df[\"conversation_id\"].unique()))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Outcome analysis: all utterances prior to abandonment vs completed\n",
    "Which keywords/phrases are statistically more associated with the **all utterances** in abandoned conversations than with completed ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_keywords=25\n",
    "custom_stop_words=[\"would\",\"pm\",\"ok\",\"yes\",\"no\",\"thank\",\"thanks\",\"hi\",\"i\",\"you\"]\n",
    "data = keyword_analysis.get_data_for_comparison_visual(all_utterances_abandoned, all_utterances_completed, num_keywords,custom_stop_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = {'debugger': True,  'flattened': True, 'width' : 800, 'height' : 600}\n",
    "visualization.draw_wordpackchart(config, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: as shown above when doing an outcome-driven analysis **only** terms that are statistically associated with the dropped off conversations are highlighted, for example `next`, `day`, and `day tomorrow`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"summary\"></a>\n",
    "## 8. Summary and next steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The analysis presented in this notebook can help you measure the effectiveness of specific tasks/flows within the dialog flows of your assistant skills. The visual components can be used to find large groups of conversations with potentially common issues to improve. The flow analysis can help you discover existing journeys, and focus on specific journey points where many conversations are lost. The transcript and visual keywords/phrases analysis helps you explore those conversations to a greater depth and detect potential issues. \n",
    "\n",
    "\n",
    "We suggest the following possible next steps:\n",
    "- **Identify candidate areas to focus your improvement efforts**\n",
    "    - Use the filtering and the flow visualization capabilities to focus on low performing flows, and identify high volume of problematic areas (e.g. abandonments)\n",
    "- **Perform initial assessment of potential issues**\n",
    "    - Use the transcript visualization and text analysis components to identify a list of potential issues to further narrow down groups of common conversations that require a common solution\n",
    "        - missing intent or entities\n",
    "        - existing intent, candidate for more training\n",
    "        - out of scope, candidate for more training\n",
    "        - dialog logic (sequence of steps, context, changes to condition, list and order of options,. etc)\n",
    "        - bot response was not relevant\n",
    "        - backend/webhook related \n",
    "- **Export candidate utterances for expanding the coverage of your intents**\n",
    "    - Export candidate utterances from selected conversations\n",
    "    - Upload the utterances as a CSV to Watson Assistant using the [Intent Recommendations](https://cloud.ibm.com/docs/services/assistant?topic=assistant-intent-recommendations#intent-recommendations-log-files-add) feature \n",
    "- **Perform detailed analysis to identify the changes needed to the dialog logic**\n",
    "    - Export a selected group of conversations that require detailed analysis\n",
    "    - Perform an assessment of the conversations using the provided spreadsheet template\n",
    "    - Import the assessment spreadsheet to the [Effectiveness Notebook](https://github.com/watson-developer-cloud/assistant-improve-recommendations-notebook/blob/master/notebook/Effectiveness%20Notebook.ipynb) to identify the most commonly occurring problems and prioritize a list of improvement actions\n",
    "- **Measure change in user journeys after deploying to production**\n",
    "    - Gain more confidence in making changes, by tracking changes to the flow metrics and user journeys in the dialog flow before and after deploying to production\n",
    "    - Load data of similar time range from before and after deployment of a new release. Create a separate flow analysis for each time period, and observe expected changes to user journeys that were updated from the last release, and that no negative side-effects impact otherwise stable flows.\n",
    "\n",
    "For more information, please check <a href=\"https://github.com/watson-developer-cloud/assistant-improve-recommendations-notebook/raw/master/notebook/IBM%20Watson%20Assistant%20Continuous%20Improvement%20Best%20Practices.pdf\" target=\"_blank\" rel=\"noopener noreferrer\">Watson Assistant Continuous Improvement Best Practices</a>.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Here are a few useful exports you can use to support above activities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#to export all user utterances in the dropoff point of flow visualization selection  \n",
    "dropped_off_conversations[dropped_off_conversations[\"request_text\"] != \"\"].to_csv('abandoned-user-utterances.csv', columns=[\"request_text\"], index=False, header=False)\n",
    "\n",
    "#to export all user utterances in the dropoff point of flow visualization selection\n",
    "dropped_off_conversations.to_csv('abandoned-conversation-ids.csv', columns=[\"conversation_id\"], index=False,header=False) \n",
    "\n",
    "#to export all columns of the canonical model for abandoned conversations  \n",
    "dropped_off_conversations.to_csv('abandoned-logs.csv', index=False,)\n",
    "\n",
    "#to export specific conversation, e.g. 00KjvlWcGozRTcSYTrlGqj4JYtYH5gjbvw3j\n",
    "conversation_id_to_export = '00KjvlWcGozRTcSYTrlGqj4JYtYH5gjbvw3j'\n",
    "df_logs_to_analyze[df_logs_to_analyze[\"conversation_id\"] == conversation_id_to_export].to_csv(conversation_id_to_export + \".csv\", index=False)\n",
    "\n",
    "#to export user utterances for intent training with Watson Recommends\n",
    "from conversation_analytics_toolkit import export\n",
    "sentences = dropped_off_conversations[dropped_off_conversations[\"request_text\"] != \"\"].reset_index()\n",
    "sentences = sentences[[\"request_text\"]]\n",
    "sentences.columns = ['example']\n",
    "filtered_sentences = export.filter_sentences(sentences, min_complexity = 3, max_complexity = 20)\n",
    "df_sentences = pd.DataFrame(data={\"training_examples\": filtered_sentences})\n",
    "df_sentences.to_csv(\"./utterances-for-Watson-Intent-Recommendations.csv\", sep=',',index=False, header=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id=\"authors\"></a>Authors\n",
    "\n",
    "**Avi Yaeli** is a Research Staff Member at IBM Research AI organization, where he develops algorithms and tools for discovering insights in complex data. His research interests include data mining, information visualization, natural language processing, and cloud computing. Avi has published more than 30 papers and patents. \n",
    "\n",
    "**Sergey Zeltyn**, Ph.D. in Statistics, is a Data Scientist in IBM Research - Haifa. His research agenda includes text analytics, machine learning, forecasting and operations research. Sergey has broad experience in data analysis, model development and implementation. His research has been published at top operations research and statistical journals."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id=\"acknowledgement\"></a> Acknowledgement\n",
    "\n",
    "The authors would like to thank the following members of IBM Watson, Research and Services for their contributions and feedback of the underlying technology and notebook: Eric Wayne, Zhe Zhang, Kyle Croutwater, David Boaz, Kalyan Dutia, Erika Agostinelli, and Tony Hickman. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2020 IBM. This notebook and its source code are released under the terms of the MIT License."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}