RelevancyTrainingBeta.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Relevancy Training Beta in Watson Discovery Service\n",
    "This notebook is meant to provide a basic example of how to use the Relevancy Training beta capabilities in Watson Discovery \n",
    "service. See the note on Beta capabilities in the release notes here https://www.ibm.com/watson/developercloud/doc/discovery/release-notes.html\n",
    "\n",
    "Relevancy Training allows developers to train Watson Discovery to find signals in the language of questions and documents to help surface the most relevant documents to the top of the results. Documentation for the capability is here https://www.ibm.com/watson/developercloud/doc/discovery/train.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Collect Representative Queries\n",
    "In order to perform Relevancy Training, you first need a set of representative queries that reflect what real users will ask of the Discovery service when integrated in your app. In general Relevancy Training is best suited to deal with queries expressed in natural language or phrases where there are multiple important terms. \n",
    "There are two common ways to collect these questions. One is to work with Subject Matter Experts (SMEs) to create questions. The other is to deploy a simple application to a set of pilot or alpha users and log/track usage. For this example we will assume questions have already been collected. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Collect Training Examples\n",
    "After collecting questions, you need to provide Relevancy training with examples of good and bad answer documents for those questions. To do this, we will prepare a file with the training queries and results from the untrained Discovery service. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import watson_developer_cloud\n",
    "import json\n",
    "import csv\n",
    "import requests\n",
    "\n",
    "#create a new Discovery object using the python SDK and credentials from bluemix. \n",
    "username=\"INSERT CREDENTIALS HERE\"\n",
    "password=\"INSERT CREDENTIALS HERE\"\n",
    "\n",
    "discovery = watson_developer_cloud.DiscoveryV1(\n",
    "    '2016-11-07',\n",
    "    username=username,\n",
    "    password=password)\n",
    "\n",
    "#specify the environment and collection where the content lives. These ids can be collected from \n",
    "#the discovery web tooling collection details page.\n",
    "environment = \"7455992d-3f0c-4936-b4d3-1dc8d0277e50\"\n",
    "collection = \"800ef70c-7ac9-4a77-9311-6ab96adc7751\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this sample, we will work with a predifined set of questions, when using this example, you will need to fill in the path to a txt file containing a single training question per line. \n",
    "\n",
    "This step may take a few minutes to run through all the queries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with open (\"/bww_data/questions_train.txt\") as questions:\n",
    "    #open an output file to place the responses \n",
    "    filestr = \"/bww_data/training_file.tsv\"\n",
    "    of = open(filestr, \"w\")\n",
    "    writer = csv.writer(of, delimiter=\"\\t\")\n",
    "    \n",
    "    #go through each question in file and prepare Discovery query paramaters \n",
    "    for line in questions:\n",
    "        question = line.replace(\"\\n\", \"\")\n",
    "        params = {}\n",
    "        params[\"query\"] = \"%s\" % (question)\n",
    "        params[\"return\"] = \"_id,body,title\" #these fields may need to be updated depending on the content being used \n",
    "        params[\"count\"] = 4 \n",
    "        \n",
    "        #run Discovery query to get results from untrained service \n",
    "        result = discovery.query(environment_id=environment, collection_id=collection, query_options=params)\n",
    "        \n",
    "        #create a row for each query and results \n",
    "        result_list = [question.encode(\"utf8\")]\n",
    "        for resultDoc in result[\"results\"]:\n",
    "            id = resultDoc[\"id\"]\n",
    "            body = resultDoc[\"body\"].encode(\"utf8\")\n",
    "            title = resultDoc[\"title\"].encode(\"utf8\")\n",
    "            result_list.extend([id,title,body,' ']) #leave a space to enter a relevance label for each doc \n",
    "        \n",
    "        #write the row to the file \n",
    "        writer.writerow(result_list)\n",
    "    \n",
    "    of.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting file contains the question and potential answers from an untrained Discovery instance. This file can be shared with SMEs to help rate each of the answers. These ratings will be used as relevance labels for the training data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Upload training data\n",
    "The next step is to take the queries, documents, and relevance labels, and create training data objects to send to the Discovery service. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#function for posting to training data endpoint \n",
    "def training_post(discovery_path, training_obj):\n",
    "    training_json = json.dumps(training_obj)\n",
    "    headers = {\n",
    "        'content-type': \"application/json\"\n",
    "        }\n",
    "    auth = (username, password)\n",
    "    r = requests.request(method=\"POST\",url=discovery_path,data=training_json,headers=headers,auth=auth)\n",
    " \n",
    "#open the training file and create new training data objects\n",
    "with open(filestr,'r') as training_doc:\n",
    "    training_csv = csv.reader(training_doc, delimiter='\\t')    \n",
    "    training_obj = {}\n",
    "    training_obj[\"examples\"] = []\n",
    "    \n",
    "    discovery_path = \"https://gateway.watsonplatform.net/discovery/api/v1/environments/\" + environment + \"/collections/\" + collection \n",
    "    discovery_training_path = discovery_path + \"/training_data?version=2016-11-07\"\n",
    "    \n",
    "    #create a new object for each example \n",
    "    for row in training_csv:\n",
    "        training_obj[\"natural_language_query\"] = row[0]\n",
    "        i = 1 \n",
    "        for j in range(1,3):\n",
    "            example_obj = {}\n",
    "            example_obj[\"relevance\"] = row[i+3]\n",
    "            example_obj[\"document_id\"] = row[i]\n",
    "            training_obj[\"examples\"].append(example_obj)\n",
    "            i = i + 4 \n",
    "\n",
    "        #send the training data to the discovery service \n",
    "        training_post(discovery_training_path, training_obj)\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Check training status \n",
    "After uploading data, you can check the status of the training data to determine if all criteria have been met. The training data requirements are listed in the documentation here https://www.ibm.com/watson/developercloud/doc/discovery/train.html#training-data-requirements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "status = discovery.get_collection(environment,collection)\n",
    "print(json.dumps(status))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Run natural_language_query \n",
    "Once training is ready, you can start to query the service using the training and see the results. To do this you can put aside a set of collected questions to use as a test set. Training is utilized with the natural_langauge_query parameter in the Discovery query language. The function below will write out the json results from this query. These results could be incorporated into your application "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def relevance_query(path, query):\n",
    "    headers = {\n",
    "        'content-type': \"application/json\"\n",
    "        }\n",
    "    params = {}\n",
    "    params[\"natural_language_query\"] = query\n",
    "    params[\"version\"] = \"2016-11-07\"\n",
    "    params[\"return\"] = \"_id,body,title\"\n",
    "    params[\"count\"] = 3\n",
    "    auth = (username, password)\n",
    "    r = requests.request(method=\"GET\",url=path,params=params,headers=headers,auth=auth)\n",
    "    #print(r.text) \n",
    "\n",
    "#replace with path to your questions \n",
    "test_questions_path = \"c:/users/IBM_ADMIN/documents/data/wimbledon/wimbledon/questions/questions_test.txt\" \n",
    "\n",
    "discovery_query_path = discovery_path + \"/query\"\n",
    "\n",
    "#perform a natural_language_query \n",
    "with open(test_questions_path, 'r') as test_questions:\n",
    "    for question in test_questions:\n",
    "        print(discovery_query_path)\n",
    "        relevance_query(discovery_query_path, question)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Measure Results\n",
    "In order to judge the effectivness of the training, you can compute standard information retrieval metrics. There are number of options to use for this. One common metric is NDCG (Normalized Discounted cumulative gain) \n",
    "https://en.wikipedia.org/wiki/Discounted_cumulative_gain\n",
    "These metrics should be computed on a held out test set. This set should not be sumbitted to the service, but should still be rated for relevance in order to judge the results. It should be randomly selected from the pool of representative questions and exmples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes\n",
    "* This code is not officially maintained or validated, and is used only for demonstration purposes. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}