notebook/usage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9a1ac07e-1d5e-4a64-b235-3d57c1adb274",
   "metadata": {},
   "source": [
    "# PyConverse Basic usage:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5647d1ed-1675-497e-b793-344e1b7d54d3",
   "metadata": {},
   "source": [
    "Import necessary functions and class from pyconverse\n",
    "\n",
    "_**Note: first time you install pyconverse and make these imports it downloads few transformers models, sentence-transformers in the backend hence the first import might take few minutes.**_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a65a0438-3d5a-4b2d-b7f6-55a1ce418533",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rita/opt/anaconda3/lib/python3.8/site-packages/spacy/util.py:275: UserWarning: [W031] Model 'en_core_web_sm' (2.2.0) requires spaCy v2.2 and is incompatible with the current spaCy version (2.3.5). This may lead to unexpected results or runtime errors. To resolve this, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate\n",
      "  warnings.warn(warn_msg)\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from pprint import pprint\n",
    "\n",
    "from pyconverse import Callyzer, SpeakerStats\n",
    "from pyconverse import SemanticTextSegmentation, ZeroShotTopicFinder, TranscriptSummarization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df8fd363-3e3c-44d0-985d-9d0343ac7a7f",
   "metadata": {},
   "source": [
    "## Load sample dataset\n",
    "\n",
    "_**Note: to load your own transcript dataset, let's say from aws-transcribe, google-cloud, azure etc or any other services, you would need to convert your transcripts into a pandas dataframe and while you initialize the `Callyzer` class, you need to point towards the speaker, utterance, start-time & end-time for each utterance.**_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e2c183ff-229e-44b1-b145-57463c516fa1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>speaker</th>\n",
       "      <th>utterance</th>\n",
       "      <th>start_time</th>\n",
       "      <th>end_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CHI</td>\n",
       "      <td>he wan an explorer</td>\n",
       "      <td>0.000</td>\n",
       "      <td>1.033</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CHI</td>\n",
       "      <td>and he discovered America</td>\n",
       "      <td>1.033</td>\n",
       "      <td>3.400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CHI</td>\n",
       "      <td>and he went for the king and queen of Spain.</td>\n",
       "      <td>3.400</td>\n",
       "      <td>7.983</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>INV</td>\n",
       "      <td>tell me what you know about the Pilgrims.</td>\n",
       "      <td>16.161</td>\n",
       "      <td>20.777</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>CHI</td>\n",
       "      <td>well they moved from England and came to Ameri...</td>\n",
       "      <td>20.777</td>\n",
       "      <td>28.711</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  speaker                                          utterance  start_time  \\\n",
       "0     CHI                                 he wan an explorer       0.000   \n",
       "1     CHI                          and he discovered America       1.033   \n",
       "2     CHI       and he went for the king and queen of Spain.       3.400   \n",
       "3     INV          tell me what you know about the Pilgrims.      16.161   \n",
       "4     CHI  well they moved from England and came to Ameri...      20.777   \n",
       "\n",
       "   end_time  \n",
       "0     1.033  \n",
       "1     3.400  \n",
       "2     7.983  \n",
       "3    20.777  \n",
       "4    28.711  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transcript_df = pd.read_csv(\"sample_transcript_data.csv\"); transcript_df.head() #read sample data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "977c8bdc-16e2-4ad5-a9d7-6445b5462fb0",
   "metadata": {},
   "source": [
    "## Analyse the Call transcript"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "096d1d03-79ff-4ed3-b826-e79a32f86eda",
   "metadata": {},
   "source": [
    "Initialise the core call analysis class `Callyzer` with your dataset represented as a pandas dataframe and point towards utterance, speaker, start-time & end-time columns in it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "376ce707-8d81-4f21-b85b-e81ffb2ec3ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "transcript_analysis = Callyzer(data=transcript_df, utterance=\"utterance\", speaker=\"speaker\", starttime=\"start_time\", endtime=\"end_time\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91bb77ce-70d2-4cb5-879c-0ad953facf3e",
   "metadata": {},
   "source": [
    "compute and access various attributes of the call as follows: \n",
    "\n",
    "## Find Interruptions and periods of silence in a call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dbcca4d5-9bf6-4a24-b220-27acface05f2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. INTERRUPTIONS:\n",
      "\n",
      "{'CHI': {'count': 1,\n",
      "         'metadata': [{'end_time': 1223.032,\n",
      "                       'index': 200,\n",
      "                       'start_time': 1222.365}]},\n",
      " 'INV': {'count': 1,\n",
      "         'metadata': [{'end_time': 574.931,\n",
      "                       'index': 102,\n",
      "                       'start_time': 574.198}]},\n",
      " 'total_interruption': 2}\n",
      "\n",
      "2. PERIODS OF SILENCE:\n",
      "\n",
      "{'CHI': {'count': 3,\n",
      "         'metadata': [{'end_time': 189.865, 'index': 34, 'start_time': 188.765},\n",
      "                      {'end_time': 279.215, 'index': 48, 'start_time': 275.082},\n",
      "                      {'end_time': 3190.575,\n",
      "                       'index': 617,\n",
      "                       'start_time': 3182.483}]},\n",
      " 'INV': {'count': 6,\n",
      "         'metadata': [{'end_time': 20.777, 'index': 3, 'start_time': 16.161},\n",
      "                      {'end_time': 964.2, 'index': 166, 'start_time': 963.0},\n",
      "                      {'end_time': 1512.706,\n",
      "                       'index': 253,\n",
      "                       'start_time': 1509.049},\n",
      "                      {'end_time': 1822.656,\n",
      "                       'index': 315,\n",
      "                       'start_time': 1820.52},\n",
      "                      {'end_time': 3486.594,\n",
      "                       'index': 685,\n",
      "                       'start_time': 3485.7},\n",
      "                      {'end_time': 3540.255,\n",
      "                       'index': 700,\n",
      "                       'start_time': 3539.524}]},\n",
      " 'total_interruption': 9}\n"
     ]
    }
   ],
   "source": [
    "interruptions = transcript_analysis.get_interruption() #interruption periods in a call\n",
    "silence = transcript_analysis.get_silence() #periods of silence in a call\n",
    "\n",
    "print(\"1. INTERRUPTIONS:\\n\")\n",
    "pprint(interruptions)\n",
    "\n",
    "print(\"\\n2. PERIODS OF SILENCE:\\n\")\n",
    "pprint(silence)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b12d924f-f9b3-4cba-b300-46a7c4fa945f",
   "metadata": {},
   "source": [
    "## Find the Backchannel utterances in a call transcripts.\n",
    "\n",
    "\n",
    "Backchannels can be verbal, non-verbal (visual) or both. Vocalisations like 'hmm' or 'uh-huh', gestures such as head nods or head shakes, and a combination of verbal and non-verbal responses are common examples of backchannels. `pyconverse` identifies verbal backchannels using two different methods: \n",
    "\n",
    "1. default : via a set of commonly used backchannel keywords dictionary - fast, slightly low accuracy.\n",
    "2. nlp: via sentence similarity with sentence-transformers - slow, high accuracy. \n",
    "\n",
    "_**Note: the backchannel identification with sentence similarity implementation is  highly inspired by facebook's [Unsupervised Topic Segmentation of Meetings with BERT Embeddings](https://arxiv.org/abs/2106.12978) paper.**_\n",
    "\n",
    "The way this works is by taking common backchannel words like \"okay\", \"thats it\", \"ummhhh\" etc as backchannel samples and then do maxpool and  we apply sentence similarity with all utterances in the transcript."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "dc396b98-916f-4a77-b4de-3add2d0a4390",
   "metadata": {},
   "outputs": [],
   "source": [
    "backchannels_via_keywords = transcript_analysis.tag_backchannel().query(\"is_backchannel == True\") #identify backchannel utterances via keywords\n",
    "backchannels_via_transformers = transcript_analysis.tag_backchannel(type='nlp').query(\"is_backchannel == True\") #identify backchannel utterances with sentence-transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7cc5c81e-5075-40bf-a98a-12544020b7a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>speaker</th>\n",
       "      <th>utterance</th>\n",
       "      <th>start_time</th>\n",
       "      <th>end_time</th>\n",
       "      <th>is_backchannel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>156.133</td>\n",
       "      <td>156.866</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>284.348</td>\n",
       "      <td>284.914</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>342.564</td>\n",
       "      <td>343.030</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>450.400</td>\n",
       "      <td>451.066</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>585.980</td>\n",
       "      <td>586.630</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>701.130</td>\n",
       "      <td>701.530</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>741.783</td>\n",
       "      <td>742.149</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>135</th>\n",
       "      <td>CHI</td>\n",
       "      <td>no</td>\n",
       "      <td>770.545</td>\n",
       "      <td>771.411</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>136</th>\n",
       "      <td>CHI</td>\n",
       "      <td>yeah</td>\n",
       "      <td>771.411</td>\n",
       "      <td>771.895</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>158</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>916.974</td>\n",
       "      <td>917.574</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>170</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>973.350</td>\n",
       "      <td>974.100</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>193</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>1175.099</td>\n",
       "      <td>1175.432</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>201</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>1223.631</td>\n",
       "      <td>1224.547</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>231</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>1379.564</td>\n",
       "      <td>1380.214</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>241</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>1452.549</td>\n",
       "      <td>1453.399</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>252</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>1502.216</td>\n",
       "      <td>1502.899</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>1576.526</td>\n",
       "      <td>1577.664</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>286</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>1669.388</td>\n",
       "      <td>1672.105</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>297</th>\n",
       "      <td>CHI</td>\n",
       "      <td>yeah</td>\n",
       "      <td>1719.619</td>\n",
       "      <td>1719.991</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>300</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>1723.521</td>\n",
       "      <td>1724.206</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>2153.041</td>\n",
       "      <td>2153.459</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>402</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>2198.258</td>\n",
       "      <td>2198.723</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>419</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>2284.706</td>\n",
       "      <td>2285.159</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>445</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>2382.087</td>\n",
       "      <td>2382.482</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>452</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>2420.042</td>\n",
       "      <td>2420.947</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>459</th>\n",
       "      <td>CHI</td>\n",
       "      <td>yep</td>\n",
       "      <td>2474.888</td>\n",
       "      <td>2475.956</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>462</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>2488.773</td>\n",
       "      <td>2489.389</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>473</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>2539.035</td>\n",
       "      <td>2539.778</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>483</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>2593.276</td>\n",
       "      <td>2593.973</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>497</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>2665.049</td>\n",
       "      <td>2665.989</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>532</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>2804.691</td>\n",
       "      <td>2805.190</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>542</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>2855.566</td>\n",
       "      <td>2856.901</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>571</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>2989.355</td>\n",
       "      <td>2990.690</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>584</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>3038.250</td>\n",
       "      <td>3038.958</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>626</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>3236.295</td>\n",
       "      <td>3237.572</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>642</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>3307.859</td>\n",
       "      <td>3308.370</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>652</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>3361.161</td>\n",
       "      <td>3361.590</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>685</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>3485.700</td>\n",
       "      <td>3486.594</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>696</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>3532.419</td>\n",
       "      <td>3533.789</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    speaker utterance  start_time  end_time  is_backchannel\n",
       "28      INV      okay     156.133   156.866            True\n",
       "52      INV      okay     284.348   284.914            True\n",
       "65      INV      okay     342.564   343.030            True\n",
       "83      INV      okay     450.400   451.066            True\n",
       "105     INV      okay     585.980   586.630            True\n",
       "120     INV      okay     701.130   701.530            True\n",
       "128     INV      okay     741.783   742.149            True\n",
       "135     CHI        no     770.545   771.411            True\n",
       "136     CHI      yeah     771.411   771.895            True\n",
       "158     INV      okay     916.974   917.574            True\n",
       "170     INV      okay     973.350   974.100            True\n",
       "193     INV      okay    1175.099  1175.432            True\n",
       "201     INV      okay    1223.631  1224.547            True\n",
       "231     INV      okay    1379.564  1380.214            True\n",
       "241     INV      okay    1452.549  1453.399            True\n",
       "252     INV      okay    1502.216  1502.899            True\n",
       "264     INV      okay    1576.526  1577.664            True\n",
       "286     INV      okay    1669.388  1672.105            True\n",
       "297     CHI      yeah    1719.619  1719.991            True\n",
       "300     INV      okay    1723.521  1724.206            True\n",
       "389     INV      okay    2153.041  2153.459            True\n",
       "402     INV      okay    2198.258  2198.723            True\n",
       "419     INV      okay    2284.706  2285.159            True\n",
       "445     INV      okay    2382.087  2382.482            True\n",
       "452     INV      okay    2420.042  2420.947            True\n",
       "459     CHI       yep    2474.888  2475.956            True\n",
       "462     INV      okay    2488.773  2489.389            True\n",
       "473     INV      okay    2539.035  2539.778            True\n",
       "483     INV      okay    2593.276  2593.973            True\n",
       "497     INV      okay    2665.049  2665.989            True\n",
       "532     INV      okay    2804.691  2805.190            True\n",
       "542     INV      okay    2855.566  2856.901            True\n",
       "571     INV      okay    2989.355  2990.690            True\n",
       "584     INV      okay    3038.250  3038.958            True\n",
       "626     INV      okay    3236.295  3237.572            True\n",
       "642     INV      okay    3307.859  3308.370            True\n",
       "652     INV      okay    3361.161  3361.590            True\n",
       "685     INV      okay    3485.700  3486.594            True\n",
       "696     INV      okay    3532.419  3533.789            True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "backchannels_via_keywords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0662b3b9-eeb4-4a82-b4c0-6c0c246e6798",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>speaker</th>\n",
       "      <th>utterance</th>\n",
       "      <th>start_time</th>\n",
       "      <th>end_time</th>\n",
       "      <th>is_backchannel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay great</td>\n",
       "      <td>79.315</td>\n",
       "      <td>81.382</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>CHI</td>\n",
       "      <td>well he</td>\n",
       "      <td>103.298</td>\n",
       "      <td>114.700</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>156.133</td>\n",
       "      <td>156.866</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay good</td>\n",
       "      <td>196.855</td>\n",
       "      <td>198.232</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>284.348</td>\n",
       "      <td>284.914</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>667</th>\n",
       "      <td>INV</td>\n",
       "      <td>that's it</td>\n",
       "      <td>3423.566</td>\n",
       "      <td>3424.170</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>681</th>\n",
       "      <td>INV</td>\n",
       "      <td>mhm.</td>\n",
       "      <td>3460.900</td>\n",
       "      <td>3461.353</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>685</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>3485.700</td>\n",
       "      <td>3486.594</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>696</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>3532.419</td>\n",
       "      <td>3533.789</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>700</th>\n",
       "      <td>INV</td>\n",
       "      <td>yeah.</td>\n",
       "      <td>3539.524</td>\n",
       "      <td>3540.255</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    speaker   utterance  start_time  end_time  is_backchannel\n",
       "17      INV  okay great      79.315    81.382            True\n",
       "21      CHI     well he     103.298   114.700            True\n",
       "28      INV        okay     156.133   156.866            True\n",
       "37      INV   okay good     196.855   198.232            True\n",
       "52      INV        okay     284.348   284.914            True\n",
       "..      ...         ...         ...       ...             ...\n",
       "667     INV   that's it    3423.566  3424.170            True\n",
       "681     INV        mhm.    3460.900  3461.353            True\n",
       "685     INV        okay    3485.700  3486.594            True\n",
       "696     INV        okay    3532.419  3533.789            True\n",
       "700     INV       yeah.    3539.524  3540.255            True\n",
       "\n",
       "[68 rows x 5 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "backchannels_via_transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c03daf33-92dc-4654-8f3b-0175e587510f",
   "metadata": {},
   "source": [
    "backchannel detection with keywords returned with **39 utterances** vs backchannel detection with sentence-transformers returned with **68 utterances**! "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b045ff1-a678-43af-b246-82ddc57dcb4e",
   "metadata": {},
   "source": [
    "## Find the utterances which are interrogative questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "82beca7a-7ad1-4038-8d55-3b81954ea9d2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>speaker</th>\n",
       "      <th>utterance</th>\n",
       "      <th>start_time</th>\n",
       "      <th>end_time</th>\n",
       "      <th>is_backchannel</th>\n",
       "      <th>is_question</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>INV</td>\n",
       "      <td>what else.</td>\n",
       "      <td>451.066</td>\n",
       "      <td>451.916</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>167</th>\n",
       "      <td>INV</td>\n",
       "      <td>what else can you tell me.</td>\n",
       "      <td>964.200</td>\n",
       "      <td>966.666</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>446</th>\n",
       "      <td>INV</td>\n",
       "      <td>what do you know about ballet.</td>\n",
       "      <td>2382.482</td>\n",
       "      <td>2384.188</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>474</th>\n",
       "      <td>INV</td>\n",
       "      <td>what do you know about the Blues.</td>\n",
       "      <td>2539.778</td>\n",
       "      <td>2541.891</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>572</th>\n",
       "      <td>INV</td>\n",
       "      <td>what do you know about the Columbine shooting.</td>\n",
       "      <td>2990.690</td>\n",
       "      <td>2993.140</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>578</th>\n",
       "      <td>INV</td>\n",
       "      <td>what do you know about the Oklahoma_City bombing.</td>\n",
       "      <td>3018.327</td>\n",
       "      <td>3021.090</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>585</th>\n",
       "      <td>INV</td>\n",
       "      <td>what do you know about Princess_Diana.</td>\n",
       "      <td>3038.958</td>\n",
       "      <td>3040.908</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>616</th>\n",
       "      <td>INV</td>\n",
       "      <td>what do you know about John_F_Kennedy_Junior.</td>\n",
       "      <td>3172.962</td>\n",
       "      <td>3175.517</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>663</th>\n",
       "      <td>INV</td>\n",
       "      <td>what do you know about the Million_Man_March.</td>\n",
       "      <td>3393.043</td>\n",
       "      <td>3395.748</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>668</th>\n",
       "      <td>INV</td>\n",
       "      <td>what do you know about Moesha.</td>\n",
       "      <td>3424.170</td>\n",
       "      <td>3425.586</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>679</th>\n",
       "      <td>INV</td>\n",
       "      <td>what do you know about Spike_Lee.</td>\n",
       "      <td>3458.463</td>\n",
       "      <td>3459.473</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>687</th>\n",
       "      <td>INV</td>\n",
       "      <td>what do you know about Ebonics.</td>\n",
       "      <td>3488.777</td>\n",
       "      <td>3489.926</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    speaker                                          utterance  start_time  \\\n",
       "84      INV                                         what else.     451.066   \n",
       "167     INV                         what else can you tell me.     964.200   \n",
       "446     INV                     what do you know about ballet.    2382.482   \n",
       "474     INV                  what do you know about the Blues.    2539.778   \n",
       "572     INV     what do you know about the Columbine shooting.    2990.690   \n",
       "578     INV  what do you know about the Oklahoma_City bombing.    3018.327   \n",
       "585     INV             what do you know about Princess_Diana.    3038.958   \n",
       "616     INV      what do you know about John_F_Kennedy_Junior.    3172.962   \n",
       "663     INV      what do you know about the Million_Man_March.    3393.043   \n",
       "668     INV                     what do you know about Moesha.    3424.170   \n",
       "679     INV                  what do you know about Spike_Lee.    3458.463   \n",
       "687     INV                    what do you know about Ebonics.    3488.777   \n",
       "\n",
       "     end_time  is_backchannel  is_question  \n",
       "84    451.916           False         True  \n",
       "167   966.666           False         True  \n",
       "446  2384.188           False         True  \n",
       "474  2541.891           False         True  \n",
       "572  2993.140           False         True  \n",
       "578  3021.090           False         True  \n",
       "585  3040.908           False         True  \n",
       "616  3175.517           False         True  \n",
       "663  3395.748           False         True  \n",
       "668  3425.586           False         True  \n",
       "679  3459.473           False         True  \n",
       "687  3489.926           False         True  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "questions = transcript_analysis.tag_questions().query(\"is_question == True\") #identiy utterances which are questions\n",
    "questions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bdd4cc4-0fab-426f-921a-13d3737dcaac",
   "metadata": {},
   "source": [
    "## Identify the emotions of the utterances\n",
    "\n",
    "note: this might take some time as it uses miniLM language model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "958f1d63-94ca-4407-8b33-0f63f9a182b6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rita/opt/anaconda3/lib/python3.8/site-packages/pyconverse-0.2.0-py3.8.egg/pyconverse/insights.py:303: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self.data['emotion'] = classes\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>speaker</th>\n",
       "      <th>utterance</th>\n",
       "      <th>emotion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>696</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>not found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>697</th>\n",
       "      <td>INV</td>\n",
       "      <td>that's everything</td>\n",
       "      <td>Impressed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>698</th>\n",
       "      <td>INV</td>\n",
       "      <td>took a long time huh</td>\n",
       "      <td>not found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>699</th>\n",
       "      <td>INV</td>\n",
       "      <td>did you get tired</td>\n",
       "      <td>not found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>700</th>\n",
       "      <td>INV</td>\n",
       "      <td>yeah.</td>\n",
       "      <td>not found</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    speaker             utterance    emotion\n",
       "696     INV                  okay  not found\n",
       "697     INV     that's everything  Impressed\n",
       "698     INV  took a long time huh  not found\n",
       "699     INV     did you get tired  not found\n",
       "700     INV                 yeah.  not found"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transcript_analysis_ = Callyzer(transcript_df.tail(), utterance=\"utterance\", speaker=\"speaker\", starttime=\"start_time\", endtime=\"end_time\")\n",
    "\n",
    "emotions = transcript_analysis_.tag_emotion(); emotions[[\"speaker\", \"utterance\", \"emotion\"]]\n",
    "#if no emotionis identified, it returns 'not found'."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60ec8ed2-408b-40c2-8726-cecdc20156e4",
   "metadata": {},
   "source": [
    "## Identify if a given utterance is empathetic or not"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e8f6df43-11c2-49d3-8728-f58aca2ffd66",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rita/opt/anaconda3/lib/python3.8/site-packages/pyconverse-0.2.0-py3.8.egg/pyconverse/insights.py:316: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self.data['is_empathy'] = classes\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>speaker</th>\n",
       "      <th>utterance</th>\n",
       "      <th>is_empathy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>696</th>\n",
       "      <td>INV</td>\n",
       "      <td>okay</td>\n",
       "      <td>Neutral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>697</th>\n",
       "      <td>INV</td>\n",
       "      <td>that's everything</td>\n",
       "      <td>not found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>698</th>\n",
       "      <td>INV</td>\n",
       "      <td>took a long time huh</td>\n",
       "      <td>not found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>699</th>\n",
       "      <td>INV</td>\n",
       "      <td>did you get tired</td>\n",
       "      <td>not found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>700</th>\n",
       "      <td>INV</td>\n",
       "      <td>yeah.</td>\n",
       "      <td>not found</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    speaker             utterance is_empathy\n",
       "696     INV                  okay    Neutral\n",
       "697     INV     that's everything  not found\n",
       "698     INV  took a long time huh  not found\n",
       "699     INV     did you get tired  not found\n",
       "700     INV                 yeah.  not found"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "empathy = transcript_analysis_.tag_empathy(); empathy[[\"speaker\", \"utterance\", \"is_empathy\"]]\n",
    "#if no empathy is identified, it classifies the sentence as 'non_empathetic', if identified it returns 'empathetic'."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42b082c3-42a1-4569-8128-5720c5eaa111",
   "metadata": {},
   "source": [
    "## Collapse utterances into Turn level text chunks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2e308127-14f7-44be-b22c-53eaa85726b1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Original Utterance count: 701\n",
      "2. After collapsing the utterance to turn level: 194\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rita/opt/anaconda3/lib/python3.8/site-packages/pandas/core/series.py:1056: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  cacher_needs_updating = self._check_is_chained_assignment_possible()\n"
     ]
    }
   ],
   "source": [
    "# convert the data at speaker level to turn level\n",
    "df = transcript_analysis.convert_at_turn()\n",
    "\n",
    "print(f\"1. Original Utterance count: {transcript_df.shape[0]}\\n2. After collapsing the utterance to turn level: {df.shape[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c863fa99-f01d-4c37-9790-17f59221e73b",
   "metadata": {},
   "source": [
    "## Identiy the overall Psycological correlatedness nature of the speakers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f319f7f8-8728-4062-8082-06f0a9efabb0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'CHI': ['Focused on the past', 'Verbal fluency, cognitive complexity'],\n",
      " 'INV': ['Use of concrete nouns, interest in objects/things',\n",
      "         'Education, concern with precision']}\n"
     ]
    }
   ],
   "source": [
    "ss = SpeakerStats(df, speaker='speaker')\n",
    "pprint(ss.get_stats())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89641437-bfc7-4e62-8b42-c9cfe86b10c5",
   "metadata": {},
   "source": [
    "## Call segmentation\n",
    "\n",
    "lets segment our calls into bigger chunks of texts via semantic sentence similairty & text tilling algorithms. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "43b4e3f6-63ea-40f5-8956-e613b3cee155",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('he wan an explorer and he discovered America and he went for the king and '\n",
      " 'queen of Spain. tell me what you know about the Pilgrims. well they moved '\n",
      " 'from England and came to America to Plymouth_Rock and it was hard for them '\n",
      " 'at first when they came and a_lot_of them died and stuff and the Indians '\n",
      " 'they met the Indians and they came along and helped them and that was how '\n",
      " 'the first Thanksgiving started. can you tell me anything else about the '\n",
      " \"Pilgrims. they moved from England because of the king he wasn't treating \"\n",
      " \"them right and they couldn't have what they they had to worship what he \"\n",
      " \"wanted to worship and do what he wanted to do and they didn't like that so \"\n",
      " 'they moved.')\n",
      "--------------------------------------------------\n",
      "(' can you tell me anything else okay great tell me everything you know about '\n",
      " \"George_Washington. he was the first president he was well he I'm trying to \"\n",
      " 'well he fought in the Civil_War he was a general in the Civil_War and '\n",
      " \"chopped down his father's cherry tree when he was a little boy he that's it.\")\n",
      "--------------------------------------------------\n",
      "(\" okay now you can tell me everything even if you're not sure tell me anyway \"\n",
      " \"okay can you tell me anything else about George_Washington even if you're \"\n",
      " \"not sure it's okay. unhunh unhunh. can't think of anything okay good now \"\n",
      " 'tell me everything you know about the Declaration_of_Independence.')\n",
      "--------------------------------------------------\n",
      "(' the Declaration it was written in I think seventeen seventy six it was '\n",
      " 'written by Thomas_Jefferson they wrote it because they needed a set of rules '\n",
      " 'when they broke free from Britain and it oh lots of delegates from the '\n",
      " 'thirteen colonies came and they held a meeting that lasted for a long time '\n",
      " 'to talk about it.')\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "sts = SemanticTextSegmentation(df)\n",
    "segments = sts.get_segments()\n",
    "\n",
    "for segment in segments[0:4]:\n",
    "    pprint(segment)\n",
    "    print(\"-\"*50)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4947df56-4480-4402-8d4b-6eb35f771b48",
   "metadata": {},
   "source": [
    "## ZeroShot topic identification\n",
    "\n",
    "Identify topics being discussed in a call via zero shot topic infernce at utterance/segment level (works best on segments)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "07c52d1e-0a19-4a6e-8c9f-85c0a9d3df71",
   "metadata": {},
   "outputs": [],
   "source": [
    "zst = ZeroShotTopicFinder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "98644e5f-8da0-4c8c-bcb5-e26e4d9547e6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text: he wan an explorer and he discovered America and he went for the king and queen of Spain. tell me what you know about the Pilgrims. well they moved from England and came to America to Plymouth_Rock and it was hard for them at first when they came and a_lot_of them died and stuff and the Indians they met the Indians and they came along and helped them and that was how the first Thanksgiving started. can you tell me anything else about the Pilgrims. they moved from England because of the king he wasn't treating them right and they couldn't have what they they had to worship what he wanted to worship and do what he wanted to do and they didn't like that so they moved.\n",
      "\n",
      "Topics: ['Settler', 'Wayfarer']\n",
      "\n",
      "--------------------------------------------------\n",
      "Text:  can you tell me anything else okay great tell me everything you know about George_Washington. he was the first president he was well he I'm trying to well he fought in the Civil_War he was a general in the Civil_War and chopped down his father's cherry tree when he was a little boy he that's it.\n",
      "\n",
      "Topics: ['War', 'Head Of State']\n",
      "\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "for text in segments[0:2]:\n",
    "    print(f\"Text: {text}\\n\")\n",
    "    print(f\"Topics: {zst.find_topic(text)}\\n\")\n",
    "    print(\"-\"*50)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d5da0e0",
   "metadata": {},
   "source": [
    "### Transcript Summarization\n",
    "\n",
    "Summarize the whole transcript"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "139d6110",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rita/opt/anaconda3/lib/python3.8/site-packages/pyconverse-0.2.0-py3.8.egg/pyconverse/summarization.py:37: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n"
     ]
    }
   ],
   "source": [
    "sample = transcript_df.iloc[:30]\n",
    "ts = TranscriptSummarization(sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "df4eba18",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rita/opt/anaconda3/lib/python3.8/site-packages/pyconverse-0.2.0-py3.8.egg/pyconverse/summarization.py:52: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "100%|██████████| 2/2 [00:15<00:00,  7.85s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CHI tells INV about the Pilgrims who came to America from England and settled at Plymouth Rock. They met the Indians who helped them. George Washington discovered America and went for the king and queen of Spain.He was the first president. He fought in the Civil War as a general. He chopped down his father's cherry tree as a boy.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "print(ts.get_summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2769991",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}