notebooks/Step1-speech-to-text.ipynb

{
    "nbformat_minor": 1, 
    "cells": [
        {
            "source": "# Call Center Instrumentation & Analytics (CCIA)\n\n## Watson-Call-Center-Think18 Lab March 2015\n\nThis document provides guidance and background for a hands-on Python + IBM Watson lab being presented at an IBM Think2018 conference in March 2018, and for an IPython / Jupyter notebook and python code available for open source after the event.\n\nThe focus is Call Center Instrumentation and Analytics (CCIA) pattern.   The Notebook and information here seek to help organizations beginning to explore how to better understand the unstructured \"dark data\" that arises from phone calls to call centers. \n\n### Why is this useful?\nEnterprises spend more than $1 trillion on 250 billion customer service calls each year.  By using multiple IBM Watson \"signal services\" to extract signal from raw audio data; perform data analytics, clustering, unsupervised and machine learning, and visualizations, technical teams can use data understand patterns in call centers. KPI and ROI positive.\n\n### What is the process? And what Watson services are used?\n- Step 1 - Speech to Text (STT) \u2013 Converts Raw Audio to Transcripts; \n- Step 2 - Natural Language Understanding (NLU) - extracts features concepts, entities, keywords, categories/topics, sentiment and emotion; \n- Step 3 - Natural Language Classifier (NLC) - is a user trained classification service, with user defined \u201cground truth\u201d that classifies text chunks; \n- Step 4 - Tone Analyzer (Tone) \u2013 uses linguistic analysis to detect emotional and language tones in written text; \n- Step 5 - Call Center Analytics \u2013 analyzes and visualizes the data signal to allow for interpretation of data and in cases, actionable insights; \n\n\n### Beginner Audience & Focus on Basics\n- This is a beginner lab intended to educate on the fundamentals of getting from data to insights with IBM Watson and open source tools \n- Audience may include IT and operations teams curious about enriching unstructured data \u2013 the lab is NOT intended for sophisticated call center technologists \n- Lab/code does NOT purport to compete with expensive and sophisticated solutions already in market \n- The lab and code cover the basics \u2013 to educate on the fundamental plumbing and steps, to provide base for instrumentation \n\n### Success Metrics\nIf successful \u2013 the lab participants or notebook users will\n1.\tGain experience in using an IPython / Jupyter notebook\nhttps://ipython.org/notebook.html\n2.\tConnect to four Watson Developer Cloud \u2018signal service\u2019 APIs \nhttps://www.ibm.com/watson/developer/ \n3.\tConnect to IBM Cloud Object storage for data read and write \nhttps://www.ibm.com/cloud/object-storage \n4.\tUnderstand whether/how the tools and methods might benefit org\nhttps://github.com/mamoonraja/call-center-think18/tree/master/notebooks\n\n", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## Notebook 1 \u2013 Speech to Text (STT) & First Contact\n## Install Python Dependencies\n\nPython\u2019s standard library is very extensive, offering a wide range of facilities.  It contains built-in modules like JSON a lightweight data interchange format.  https://docs.python.org/2/library/index.html and https://docs.python.org/2/library/json.html\n\nIBM Watson Developer Cloud has a Python client library to quickly get started with the various Watson APIs services.\nhttps://pypi.python.org/pypi/watson-developer-cloud\n\nUsing Python with IBM COS: Python support is provided through the Boto 3 library. The boto3 library provides complete access and can source credentials. The IBM COS endpoint must be specified when creating a service resource or low-level client as shown in documentation\nhttps://ibm-public-cos.github.io/crs-docs/python\n", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "#imports.... Run this each time after restarting the Kernel\n!pip install watson_developer_cloud\nimport watson_developer_cloud as watson\nimport json\nfrom botocore.client import Config\nimport ibm_boto3\nimport requests\nfrom urllib.request import urlopen \n"
        }, 
        {
            "source": "## Set up Cloud Object Storage\nIBM Cloud Object Storage is a highly scalable cloud storage service, designed for high durability, resiliency and security. Store, manage and access your data via our self-service portal and RESTful APIs. Connect applications directly to Cloud Object Storage use other IBM Cloud Services with your data.  If creating separate of lab - go here: https://console.bluemix.net/catalog/services/cloud-object-storage to create one\n\nOnce Cloud object storage instance is created (pre-condition for this project), go to Cloud Object Storage dashboard: Login at http://ibm.com/cloud/ aka https://console.ng.bluemix.net/  and from Dashboard select your Cloud object storage instance, which will take you to service dashboard page.\n\n### Credentials\nCredentials are also created for you when you create project. From service dashboard page select `Service Credentials` from left navigation menu item, and copy/paste the credentials below:\n\n### Bucket name\nBuckets are created for you when you create project. From service dashboard page select `Buckets` from left navigation menu item, and get your bucket name and copy/paste bucket name below:\n", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 303, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# For Cloud Object Storage - populate your own information here from \"SERVICES\" on this page, or Console Dashboard on ibm.com/cloud\n\n# From service dashboard page select Service Credentials from left navigation menu item\ncredentials_os = {\n  \"apikey\": \"\",\n  \"cos_hmac_keys\": {\n    \"access_key_id\": \"\",\n    \"secret_access_key\": \"\"\n  },\n  \"endpoints\": \"https://cos-service.bluemix.net/endpoints\",\n  \"iam_apikey_description\": \"Auto generated apikey during resource-key operation for Instance\",\n  \"iam_apikey_name\": \"\",\n  \"iam_role_crn\": \"\",\n  \"iam_serviceid_crn\": \"\",\n  \"resource_instance_id\": \"\"\n}\n\n# Buckets are created for you when you create project. From service dashboard page select Buckets from left navigation menu item, \ncredentials_os['BUCKET'] = '<bucket_name>' # copy bucket name from COS"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# The code was removed by DSX for sharing."
        }, 
        {
            "source": "### How to get audio tester files for this lab?\n\nYou can follow following sources to get audio files:\n    - https://github.com/mamoonraja/call-center-think18/tree/master/resources/audio_samples\n    - https://github.com/rustyoldrake/call_center_instrumentation_analytics/blob/master/CCIA_lab_test_audio.zip\n\nReady for you!  Audio files are alson uploaded to a cloud object storage bucket for lab purposes, and you can get audio files by using read-only credentials provided by us.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 305, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "credentials_audio_samples = {\n    \"apikey\": \"RRKwTeEGGDJtPDii9SPRxOHooiXZnUdpCRcjnzdPo3uh\",\n    \"endpoints\": \"https://cos-service.bluemix.net/endpoints\",\n    \"iam_apikey_description\": \"Auto generated apikey during resource-key operation for Instance - crn:v1:bluemix:public:cloud-object-storage:global:a/8739a0c318b37263a932b45c1947965d:7ce353a1-fa6f-4e25-a311-e29b0b2a8ad8::\",\n    \"iam_apikey_name\": \"auto-generated-apikey-3e922f8a-7453-4038-8410-5399f9306530\",\n    \"iam_role_crn\": \"crn:v1:bluemix:public:iam::::serviceRole:Reader\",\n    \"iam_serviceid_crn\": \"crn:v1:bluemix:public:iam-identity::a/8739a0c318b37263a932b45c1947965d::serviceid:ServiceId-3039af12-8feb-4ef5-8306-d6acd6a0ca31\",\n    \"resource_instance_id\": \"crn:v1:bluemix:public:cloud-object-storage:global:a/8739a0c318b37263a932b45c1947965d:7ce353a1-fa6f-4e25-a311-e29b0b2a8ad8::\",\n    \"BUCKET\": 'audio-samples',\n}\n\ncredentials_audio_samples['BUCKET'] = 'audio-samples'"
        }, 
        {
            "source": "## Create Speech to Text (STT) service and get STT Credentials \nImporting Credentials \u2013 Each Watson signal service (STT, NLC, NLU and Tone) will require credentials - a username and password\nIf you already have an IBM Cloud / Bluemix account login here https://console.bluemix.net/ but if you have not yet registered for IBM Cloud - you will need to Register for a Free account here https://www.ibm.com/watson/developer/ registration takes less than 4 minutes and is free. More information here https://www.ibm.com/watson/developer-resources/ \n\nOnce logged in - go to https://console.bluemix.net/developer/watson/dashboard - browse services for SPEECH TO TEXT, and select Details, Create service from here https://console.bluemix.net/catalog/services/speech-to-text  - for free you can select LITE Plan \nLITE plan for STT \u201cgets you started with 100 minutes per month at no cost\u201d\n\nThe Username and Password (and URL) is found by clicking on service credentials and then \u201cview credential\u201d, and copy/paste credentials below: ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 306, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "credentials_stt = {\n  \"url\": \"\",\n  \"username\": \"\",\n  \"password\": \"\"\n}"
        }, 
        {
            "execution_count": 318, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# The code was removed by DSX for sharing."
        }, 
        {
            "source": "### Set up Object Storage Client", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 320, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "def set_up_object_storage(credentials_object_storage):\n    endpoints = requests.get(credentials_object_storage['endpoints']).json()\n\n    iam_host = (endpoints['identity-endpoints']['iam-token'])\n    cos_host = (endpoints['service-endpoints']['cross-region']['us']['public']['us-geo'])\n\n    auth_endpoint = \"https://\" + iam_host + \"/oidc/token\"\n    service_endpoint = \"https://\" + cos_host\n\n\n    client = ibm_boto3.client(\n        's3',\n        ibm_api_key_id = credentials_object_storage['apikey'],\n        ibm_service_instance_id = credentials_object_storage['resource_instance_id'],\n        ibm_auth_endpoint = auth_endpoint,\n        config = Config(signature_version='oauth'),\n        endpoint_url = service_endpoint\n       )\n    return client\n\nclient = set_up_object_storage(credentials_os)\nclient_global = set_up_object_storage(credentials_audio_samples)\n\n"
        }, 
        {
            "source": "## Speect to Text \n\nFollowing cell has two methods:\n - `get_transcript()` calls speech to text enpoint and generates a text transcript for you for a sample audio file.\n - `analyze_sample()` gets the sample object from cloud storage, calls get_transcript to fetch the tranccript, and saves your transcript in cloud storage as `<file_name>_text.json`.\n \nOGG, WAV FLAC, L16, MP3, MPEG formats are options for the IBM Watson STT service.  For the lab we use OGG samples. ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 321, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "#STT\n\nimport json\nimport io\nfrom os.path import join, dirname\nfrom watson_developer_cloud import SpeechToTextV1\n\nspeech_to_text = SpeechToTextV1(\n    username = credentials_stt['username'],\n    password = credentials_stt['password'],\n    url = 'https://stream.watsonplatform.net/speech-to-text/api',\n)\n\n\n# OGG, WAV FLAC, L16, MP3, MPEG formats are options for the STT service \n# with Narrowband (generaly telco) and Broadband (e.g. higher quality USB mic) audio.  \n# For the LAB \u2013 OGG format was used for sample files in lab. Of other audio formats e.g. WAV - remember to change 'OGG' content_type='audio/ogg' in code below if you do.\n\n#get transcript Very basic one\ndef get_transcript(audio):\n    transcript = json.dumps(speech_to_text.recognize(audio=audio, content_type='audio/ogg', timestamps=True,\n        word_confidence=True), indent=2)\n    return transcript\n\ndef download_file(path, filename):\n    url = path + filename\n    print(url)\n    r = requests.get(url, stream=True)\n    return r.content\n\ndef analyze_sample(sample):\n    streaming_body = client_global.get_object(Bucket = credentials_audio_samples['BUCKET'], Key=sample)['Body'] #http\n    audio = streaming_body.read()\n    text = get_transcript(audio)\n    client.put_object(Bucket = credentials_os['BUCKET'], Key = sample.split('.')[0] + '_text.json', Body = text)\n    return text\n\ndef visualize(transcript):\n    for result in json.loads(transcript)['results']:\n        print(result['alternatives'][0]['transcript'], result['alternatives'][0]['confidence'])    \n"
        }, 
        {
            "source": "## More about audio files\n\n`file_list` provides list of audio file in an array, each OGG file produces its own transcript.  \n\n    - Samples 1,2,3,4,5 are short - about 2 minutes \n    - Samples 6 and 7 are about 7 minutes (the STT SERVICE WILL NEED TIME TO PROCESS)\n\nFor longer files and transcription at scale: https://www.ibm.com/watson/developercloud/speech-to-text/api/v1/\n\n### WebSockets\nWebSockets includes a single method that establishes a persistent connection with the service over the WebSocket protocol.\n\n### Sessionless\nSessionless includes a method that provides a simple means of transcribing audio without the overhead of establishing and maintaining a session. Sessions provides methods that allow a client to maintain a long, multi-turn exchange, or session, with the service or to establish multiple parallel conversations with a particular instance of the service.\n\n### Asynchronous\nAsynchronous provides a non-blocking interface for transcribing audio. You can register a callback URL to be notified of job status and, optionally, results, or you can poll the service to learn job status and retrieve results manually. Longer (e.g. 1 hour) audio files may justify using asynchronous method, and a real time a sessions method (both defined below)\n\n", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 322, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# List of Audio files that are available in the read only Cloud Object Storage for lab re: credentials_audio_samples\n\nfile_list = ['sample1-addresschange-positive.ogg',\n             'sample2-address-negative.ogg',\n             'sample3-shirt-return-weather-chitchat.ogg',\n             'sample4-angryblender-sportschitchat-recovery.ogg',\n             'sample5-calibration-toneandcontext.ogg',\n             'jfk_1961_0525_speech_to_put_man_on_moon.ogg',\n             'May 1 1969 Fred Rogers testifies before the Senate Subcommittee on Communications.ogg']"
        }, 
        {
            "source": "## Transcription Test - Point to first file in list (position 0) and analyze ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# TRANSCRIBE \u2013 this is where STT receives the OGG files provided and returns text to TRANSCRIPT\n# this is a test of ONE transcription in the list - place '0' - may take a minute\ntranscript = analyze_sample(file_list[0])\nvisualize(transcript)"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "for filename in file_list:\n    print('Processing file: ', filename)\n    transcript = analyze_sample(filename)\n    visualize(transcript)"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": ""
        }
    ], 
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.5 with Spark 2.1", 
            "name": "python3-spark21", 
            "language": "python"
        }, 
        "language_info": {
            "mimetype": "text/x-python", 
            "nbconvert_exporter": "python", 
            "version": "3.5.4", 
            "name": "python", 
            "file_extension": ".py", 
            "pygments_lexer": "ipython3", 
            "codemirror_mode": {
                "version": 3, 
                "name": "ipython"
            }
        }
    }, 
    "nbformat": 4
}