example_notebooks/toree/train_bm25.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a parameter\n",
    "\n",
    "This notebook demonstrates the use of `ArbitraryParameterTrainingEstimator` to train the $b$ parameter of BM25.\n",
    "\n",
    "This notebook requires [Apache-Toree](https://toree.apache.org/) to be installed and working. You will also need your Terrier index, and the various topics and qrels to be set appropriately. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, lets import Terrier and Terrier-Spark using the `%AddDeps` magic command. Toree can use Maven Central for this, as well as your local Maven repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Marking org.terrier:terrier-core:5.0 for download\n",
      "Preparing to fetch from:\n",
      "-> file:/tmp/toree-tmp-dir6296464374568874300/toree_add_deps/\n",
      "-> file:/home/user/.m2/repository\n",
      "-> https://repo1.maven.org/maven2\n",
      "-> New file at /home/user/.m2/repository/org/tukaani/xz/1.0/xz-1.0.jar\n"
     ]
    }
   ],
   "source": [
    "%AddDeps org.terrier terrier-core 5.0 --transitive --exclude org.slf4j:slf4j-log4j12  \n",
    "%AddDeps org.terrier terrier-spark 0.0.1-SNAPSHOT --repository file:/home/user/.m2/repository --transitive\n",
    "\n",
    "//if you are using a local Maven repository, you can use \n",
    "//%AddDeps org.terrier terrier-spark 0.0.1-SNAPSHOT --repository file:/home/user/.m2/repository --transitive\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets check that we have Terrier correctly imported. Do you get the correct version number?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.0"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "org.terrier.Version.VERSION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets setup the location appropriate for our local setting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "terrierHome = /home/user/projects/tr5/Terrier\n",
       "wt2gIndex = /home/user/indices/index_wt2g/data.properties\n",
       "topicsFile = file:/home/user/TopicsQrels/TREC/WT2G/small_web/topics.401-450\n",
       "qrelsFile = file:/home/user/TopicsQrels/TREC/WT2G/small_web/qrels.trec8\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "file:/home/user/TopicsQrels/TREC/WT2G/small_web/qrels.trec8"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val terrierHome = \"/home/user/projects/tr5/Terrier\"\n",
    "val wt2gIndex = \"/home/user/indices/index_wt2g/data.properties\"\n",
    "val topicsFile = \"file:/home/user/TopicsQrels/TREC/WT2G/small_web/topics.401-450\"\n",
    "val qrelsFile = \"file:/home/user/TopicsQrels/TREC/WT2G/small_web/qrels.trec8\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now use `ArbitraryParameterTrainingEstimator` to identify a value for property `c`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TERRIER_HOME=/home/user/projects/tr5/Terrier\n",
      "terrier.etc=null\n",
      "TERRIER_HOME=/home/user/projects/tr5/Terrier\n",
      "TERRIER_ETC=/home/user/projects/tr5/Terrier/etc\n",
      "Evaluating List(c)=Vector(0.25)\n",
      "[Stage 0:=============================>                             (1 + 1) / 2]Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "[Stage 2:=============================>                             (1 + 1) / 2]Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.25) => -0.7065811663845499\n",
      "Evaluating List(c)=Vector(0.0125)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.0125) => -0.6810591735163936\n",
      "Evaluating List(c)=Vector(0.4875)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.4875) => -0.6872177217334728\n",
      "Evaluating List(c)=Vector(0.012500000000000122)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.012500000000000122) => -0.6810591735163936\n",
      "Evaluating List(c)=Vector(0.36875)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.36875) => -0.6991013431255564\n",
      "Evaluating List(c)=Vector(0.1312500000000002)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.1312500000000002) => -0.7026126462280805\n",
      "Evaluating List(c)=Vector(0.19062500000000016)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.19062500000000016) => -0.7065940942123677\n",
      "Evaluating List(c)=Vector(0.13125000000000053)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.13125000000000053) => -0.7026126462280805\n",
      "Evaluating List(c)=Vector(0.22031250000000013)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.22031250000000013) => -0.7078793150887679\n",
      "Evaluating List(c)=Vector(0.25000000000000033)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.25000000000000033) => -0.7065811663845499\n",
      "Evaluating List(c)=Vector(0.2054687500000002)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.2054687500000002) => -0.7057828771988326\n",
      "Evaluating List(c)=Vector(0.20546875000000014)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.20546875000000014) => -0.7057828771988326\n",
      "Evaluating List(c)=Vector(0.23515625000000012)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.23515625000000012) => -0.7071877027550083\n",
      "Evaluating List(c)=Vector(0.22773437500000013)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.22773437500000013) => -0.7069473757711673\n",
      "Evaluating List(c)=Vector(0.2128906250000002)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.2128906250000002) => -0.7070854890884989\n",
      "Evaluating List(c)=Vector(0.21660156250000018)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.21660156250000018) => -0.7085222490893869\n",
      "Evaluating List(c)=Vector(0.21289062500000022)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.21289062500000022) => -0.7070854890884989\n",
      "Evaluating List(c)=Vector(0.21845703125000016)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.21845703125000016) => -0.7072659451120847\n",
      "Evaluating List(c)=Vector(0.2147460937500002)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.2147460937500002) => -0.7084238116386832\n",
      "Evaluating List(c)=Vector(0.2156738281250002)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.2156738281250002) => -0.7088774988007488\n",
      "Evaluating List(c)=Vector(0.21613769531250018)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.21613769531250018) => -0.7086509631711906\n",
      "Evaluating List(c)=Vector(0.21520996093750025)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.21520996093750025) => -0.7094020242217248\n",
      "Evaluating List(c)=Vector(0.21474609375000028)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.21474609375000028) => -0.7084238116386832\n",
      "Evaluating List(c)=Vector(0.21474609375000037)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.21474609375000037) => -0.7084238116386832\n",
      "Evaluating List(c)=Vector(0.21544189453125023)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.21544189453125023) => -0.708661262231125\n",
      "Evaluating List(c)=Vector(0.21497802734375032)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.21497802734375032) => -0.708950253085109\n",
      "Evaluating List(c)=Vector(0.2150939941406253)\n",
      "Querying /home/user/indices/index_wt2g/data.properties for 26 queries\n",
      "Got for 25974 results total\n",
      "We have 47506 qrels\n",
      "List(c)=Vector(0.2150939941406253) => -0.7093859764645523\n",
      "Best parameters Vector(0.21520996093750025) with eval -0.7094020242217248\n",
      "0.21520996093750025\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "props = Map(terrier.home -> /home/user/projects/tr5/Terrier, termpipelines -> \"\")\n",
       "indexref = /home/user/indices/index_wt2g/data.properties\n",
       "allTopicsList = List((401,foreign minorities germany), (402,behavioral genetics), (403,osteoporosis), (404,ireland peace talks), (405,cosmic events), (406,parkinson s disease), (407,poaching wildlife preserves), (408,tropical storms), (409,legal pan am 103), (410,schengen agreement), (411,salvaging shipwreck treasure), (412,airport security), (...\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "lastException: Throwable = null\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "List((401,foreign minorities germany), (402,behavioral genetics), (403,osteoporosis), (404,ireland peace talks), (405,cosmic events), (406,parkinson s disease), (407,poaching wildlife preserves), (408,tropical storms), (409,legal pan am 103), (410,schengen agreement), (411,salvaging shipwreck treasure), (412,airport security), (413,steel production), (414,cuba sugar exports), (415,drugs golden triangle), (416,three gorges project), (417,creativity), (418,quilts income), (419,recycle automobile tires), (420,carbon monoxide poisoning), (421,industrial waste disposal), (422,art stolen forged), (423,milosevic mirjana markovic), (424,suicides), (425,counterfeiting money), (426,law enforcement dogs), (427,uv damage eyes), (428,declining birth rates), (429,legionnaires disease), (430,killer bee attacks), (431,robotic technology), (432,profiling motorists police), (433,greek philosophy stoicism), (434,estonia economy), (435,curbing population growth), (436,railway accidents), (437,deregulation gas electric), (438,tourism increase), (439,inventions scientific discoveries), (440,child labor), (441,lyme disease), (442,heroic acts), (443,u s investment africa), (444,supercritical fluids), (445,women clergy), (446,tourists violence), (447,stirling engine), (448,ship losses), (449,antibiotics ineffectiveness), (450,king hussein peace))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import org.apache.spark.ml._\n",
    "import org.apache.spark.sql._\n",
    "import org.terrier.spark.ml._\n",
    "import org.terrier.spark.TopicSource\n",
    "import org.terrier.querying.IndexRef\n",
    "\n",
    "val props = Map(\"terrier.home\" -> terrierHome, \"termpipelines\"->\"\")\n",
    "val indexref = IndexRef.of(wt2gIndex)\n",
    "\n",
    "TopicSource.configureTerrier(props)\n",
    "\n",
    "val allTopicsList = TopicSource.extractTRECTopics(topicsFile).toList\n",
    "val Array(trTopics, teTopics) = allTopicsList.toDF(\"qid\", \"query\").randomSplit(Array(0.5,0.5), 130882)\n",
    "\n",
    "val tuner = new ArbitraryParameterTrainingEstimator()\n",
    "tuner.setTerrierProperties(props)\n",
    "tuner.setIndexReference(indexref)\n",
    "tuner.set(tuner.measure, Measure.NDCG)\n",
    "tuner.set(tuner.measureCutoff, 1000)\n",
    "tuner.set(tuner.paramName, Seq(\"c\"))\n",
    "tuner.setSampleModel(\"BM25\")\n",
    "tuner.set(tuner.paramValueInitial, Seq(0.25d))\n",
    "tuner.set(tuner.paramValueMin, 0d)\n",
    "tuner.set(tuner.paramValueMax, 1d)\n",
    "tuner.set(tuner.measureTol, 1e-4)\n",
    "tuner.set(tuner.optMaxIter, 100)\n",
    "tuner.setQrelsFile(qrelsFile)\n",
    "val model = tuner.fit(trTopics)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets see what parameter value it got?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.21520996093750025\n"
     ]
    }
   ],
   "source": [
    "println(model.propertySettings.get(\"c\").get)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Apache Toree - Scala",
   "language": "scala",
   "name": "apache_toree_scala"
  },
  "language_info": {
   "codemirror_mode": "text/x-scala",
   "file_extension": ".scala",
   "mimetype": "text/x-scala",
   "name": "scala",
   "pygments_lexer": "scala",
   "version": "2.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}