afirm2020_query.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/castorini/anserini-notebooks-afirm2020/blob/master/afirm2020_query.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "URfNSXe6L8sH"
      },
      "source": [
        "# Querying\n",
        "\n",
        "In this exercise, we are going to first interactively query the index and then produce a TREC run with [Pyserini](https://github.com/castorini/pyserini), the Python interface to Anserini."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "m1y2srQHdOA3"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hoP7jph2d78z"
      },
      "source": [
        "Install Python dependencies (again - remember that each notebook instantiates a virtual machine of its own):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IFGfQRYFMO2A",
        "colab": {}
      },
      "source": [
        "!pip install pyjnius==1.2.1\n",
        "!pip install pyserini\n",
        "\n",
        "import os\n",
        "os.environ['JAVA_HOME'] = '/usr/lib/jvm/java-11-openjdk-amd64'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "z0HQz3QLd-9_"
      },
      "source": [
        "Fix known issue with pyjnius (see [this explanation](https://github.com/castorini/pyserini/blob/master/README.md#known-issues) for details):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "eKT2WWp2eEJy",
        "colab": {}
      },
      "source": [
        "!mkdir -p /usr/lib/jvm/java-1.11.0-openjdk-amd64/jre/lib/amd64/server/\n",
        "!ln -s /usr/lib/jvm/java-1.11.0-openjdk-amd64/lib/server/libjvm.so /usr/lib/jvm/java-1.11.0-openjdk-amd64/jre/lib/amd64/server/libjvm.so"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "lnqt7w1sU4Dv"
      },
      "source": [
        "We have already made the Anserini jar that we built in the previous exercise available in a Google bucket.\n",
        "Let's pull it directly:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "h4xSZLdsVS3-",
        "colab": {}
      },
      "source": [
        "!gsutil -m cp gs://sigir2020/anserini-0.9.2-fatjar.jar ."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1Wcr7BpfVgSg"
      },
      "source": [
        "Let's point Pyserini to the Anserini jar that we have just pulled:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "VuAwdnkpViBH",
        "colab": {}
      },
      "source": [
        "os.environ['ANSERINI_CLASSPATH'] = '.'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "rsEU-U3ketRL"
      },
      "source": [
        "## Interactive Querying"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "zK1D24DDTWcO"
      },
      "source": [
        "We will need the index for the querying experments in this exercise.\n",
        "Because Colab notebooks don't share data among themselves, i.e., each session runs on its own, we have to pull the pre-built index form GCS."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "HU6Vyp4PTY7p",
        "colab": {}
      },
      "source": [
        "!gsutil -m cp -r gs://sigir2020/indexes ."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "wokA-RljLy0V",
        "colab": {}
      },
      "source": [
        "from pyserini.search import pysearch\n",
        "import itertools"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "FO8CncEOcbjq"
      },
      "source": [
        "First, let's see grab the queries that are defined for our collection:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "oY0Ls72U3GSc",
        "colab": {}
      },
      "source": [
        "!mkdir data\n",
        "!gsutil -m cp gs://sigir2020/queries.dev.small.tsv data/queries.dev.small.tsv"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "4xMlOOqhe5xp"
      },
      "source": [
        "The hits data structure holds the docid, the retrieval score, as well as the document content.\n",
        "Let's look at the top 10 passages for the query `south african football teams`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IpsnrIaoMYg_",
        "colab": {}
      },
      "source": [
        "from IPython.core.display import display, HTML\n",
        "\n",
        "searcher = pysearch.SimpleSearcher('indexes/lucene-index.msmarco-passage.pos+docvectors+rawdocs')\n",
        "interactive_hits = searcher.search('south african football teams')\n",
        "\n",
        "for i in range(0, 10):\n",
        "    print('Rank: {} | Passage ID: {} | BM25 Score: {}'.format(i+1, interactive_hits[i].docid, interactive_hits[i].score))\n",
        "    display(HTML('<div style=\"font-family: Times New Roman; padding-bottom:10px\">' + interactive_hits[i].raw + '</div>'))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ogrmRRfobshT"
      },
      "source": [
        "The above example uses default parameters.\n",
        "Let's try setting tuned parameters for this collection:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "XIP34ZtsCaB5",
        "colab": {}
      },
      "source": [
        "searcher.set_bm25_similarity(0.82, 0.68)\n",
        "interactive_hits_tuned = searcher.search('south african football teams')\n",
        "\n",
        "for i in range(0, 10):\n",
        "    print('Rank: {} | Passage ID: {} | BM25 Score: {}'.format(i+1, interactive_hits_tuned[i].docid, interactive_hits_tuned[i].score))\n",
        "    display(HTML('<div style=\"font-family: Times New Roman; padding-bottom:10px\">' + interactive_hits[i].raw + '</div>'))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "tJpEXdsx3uBJ"
      },
      "source": [
        "**Exercise #1:**\n",
        "Compare the rankings with and without tuned parameters.\n",
        "Add a new cell to query the index with a different query of your choice, both with untuned and tuned parameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2g5YcAjkgA78",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mdZeRTf3ClAR"
      },
      "source": [
        "Note how the ranking has changed.\n",
        "We can also enable RM3 query expansion to see if it helps with our collection:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "LH3EVKaTMcgg",
        "colab": {}
      },
      "source": [
        "searcher.set_rm3_reranker(10, 10, 0.5)\n",
        "interactive_hits_tuned_rm3 = searcher.search('south african football teams')\n",
        "\n",
        "for i in range(0, 10):\n",
        "    print('Rank: {} | Passage ID: {} | BM25 Score: {}'.format(i+1, interactive_hits_tuned_rm3[i].docid, interactive_hits_tuned_rm3[i].score))\n",
        "    display(HTML('<div style=\"font-family: Times New Roman; padding-bottom:10px\">' + interactive_hits_tuned_rm3[i].raw + '</div>'))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "fHX_l-5jKZ3P"
      },
      "source": [
        "## Batch Retrieval"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "k08tEtsmDCyv"
      },
      "source": [
        "Previously we interactively queried the index.\n",
        "However, in a typical experimental setting, you would evaluate over a larger number of queries to test different information needs.\n",
        "\n",
        "Let's begin by constructing the dev queries and corresponding query IDs:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "RSlbbOMecd2d",
        "colab": {}
      },
      "source": [
        "topics = {}\n",
        "with open('data/queries.dev.small.tsv') as file:\n",
        "    for line in file:\n",
        "       id, q = line.strip().split('\\t')\n",
        "       topics[int(id)] = q\n",
        "\n",
        "print('{} queries total'.format(len(topics)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "LpSWU35Bd9Bp",
        "colab": {}
      },
      "source": [
        "queries = list(topics.values())\n",
        "qids = list([str(t) for t in topics.keys()])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mtYICC936IMU"
      },
      "source": [
        "**Exercise #2:**\n",
        "We have previously looked at these queries in the last activity.\n",
        "Again find the queries that contain `football`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "fW6cX15t5cGd",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "F37KoOfsfCSe"
      },
      "source": [
        "Now, let's run all the queries from the dev set.\n",
        "For the sake of speed, let's again only retrieve the top 5 documents for each query.\n",
        "Note that this step may still take a while."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "C74qogSKfFtD",
        "colab": {}
      },
      "source": [
        "searcher = pysearch.SimpleSearcher('indexes/lucene-index.msmarco-passage.pos+docvectors+rawdocs')\n",
        "bm25_hits = searcher.batch_search(queries, qids, k=5)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0HTohrkIO5N2"
      },
      "source": [
        "Note that the above runs batch retrieval with untuned BM25.\n",
        "We can repeat with tuned parameters, just like we did for the interactive queries:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "R6clfMWkD3ZU",
        "colab": {}
      },
      "source": [
        "searcher.set_bm25_similarity(0.82, 0.68)\n",
        "bm25_hits_tuned = searcher.batch_search(queries, qids, k=5)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mNMa1DKsD1fc"
      },
      "source": [
        "Now let's repeat with RM3 query expansion:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "j9o_IXRTk5mf",
        "colab": {}
      },
      "source": [
        "searcher.set_rm3_reranker(10, 10, 0.5)\n",
        "bm25_hits_tuned_rm3 = searcher.batch_search(queries, qids, k=5)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RZbakkgXiPSO",
        "colab_type": "text"
      },
      "source": [
        "**Exercise #3:**\n",
        "Produce a run for untuned BM25 with RM3."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vZvRqPFAgCXw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6J-CuGlz9hjc"
      },
      "source": [
        "**Exercise #4:**\n",
        "So far we have downloaded and retrieved the top passages for the dev queries.\n",
        "Now pull the eval queries and repeat the process for eval queries."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "7npsIcEU91Ji",
        "colab": {}
      },
      "source": [
        "!gsutil -m cp gs://sigir2020/queries.eval.small.tsv data/queries.eval.small.tsv"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kRxS0rzG--tB",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "tU1jjvYfKYgA"
      },
      "source": [
        "## Evaluation\n",
        "\n",
        "A crucial component of information retrieval research is evaluation and metrics.\n",
        "The most common tool used to achieve this goal is `trec_eval` developed by [NIST](https://www.nist.gov/).\n",
        "\n",
        "`trec_eval` defines a number of standard retrieval measures, the details of which can be seen [here](http://www.rafaelglater.com/en/post/learn-how-to-use-trec_eval-to-evaluate-your-information-retrieval-system).\n",
        "\n",
        "### TREC Format\n",
        "\n",
        "`trec_eval` requires the runs from various experiments to be expressed in a standard TREC format:\n",
        "\n",
        "`query_id iter docno rank similarity run_id` delimited by spaces\n",
        "\n",
        "- `query_id`: query ID\n",
        "- `iter`: constant, often either 0 or Q0 - required but ignored by `trec_eval`\n",
        "- `docno`: string values that uniquely identify a document in the collection\n",
        "- `rank`: integer, often zero indexed\n",
        "- `similarity`: float value that represents the similarity of the document to the query specified by `query_id`\n",
        "- `run_id`: string that identifies runs, used to keep track of different experiments - also ignored by `trec_eval`\n",
        "\n",
        "Evaluation also requires the ground truth in the form of relevance judgements in the qrels file.\n",
        "The qrels file follows the following format:\n",
        "\n",
        "`query_id iter docno label`\n",
        "\n",
        "- `label`: binary code (0 for not relevant and 1 for relevant)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GnhjuvsQhrVB"
      },
      "source": [
        "Convert the hits for both BM25 (tuned and untuned) and BM25+RM3 runs into the TREC format:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "P_I_F6K9kW3L",
        "colab": {}
      },
      "source": [
        "def convert_to_trec_run(experiment, run_dict):\n",
        "  with open('run.{}.txt'.format(experiment), 'w') as run_file:\n",
        "    for qid in run_dict:\n",
        "      for rank, doc in enumerate(run_dict[qid]):\n",
        "        run_file.write('{} Q0 {} {} {} {}\\n'.format(qid, doc.docid, rank, doc.score, experiment))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "nDPIwvy5iqMK",
        "colab": {}
      },
      "source": [
        "convert_to_trec_run('msmarco_passage_dev_bm25', bm25_hits)\n",
        "convert_to_trec_run('msmarco_passage_dev_bm25_tuned', bm25_hits_tuned)\n",
        "convert_to_trec_run('msmarco_passage_dev_bm25_tuned_rm3', bm25_hits_tuned_rm3)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "NtKO5XXWnWZi"
      },
      "source": [
        "Let's pull `trec_eval` and the qrels file:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "SQWfpv1gnYaf",
        "colab": {}
      },
      "source": [
        "!gsutil -m cp -r gs://sigir2020/trec_eval.9.0.4 .\n",
        "!chmod -R +x trec_eval.9.0.4/\n",
        "!gsutil -m cp gs://sigir2020/qrels.dev.small.tsv data/qrels.dev.small.tsv"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "J8dRwjl3aGsn"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MmWLvAl7iqbz"
      },
      "source": [
        "Now that we have our runs in the TREC format, we can evaluate them with `trec_eval`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "5QWvGHiXVx0V",
        "colab": {}
      },
      "source": [
        "!head -5 run.msmarco_passage_dev_bm25.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "PPNjxxLWO0X-",
        "colab": {}
      },
      "source": [
        "!chmod -R +x trec_eval.9.0.4/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-iQOW2wkgcyV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!trec_eval.9.0.4/trec_eval -m ndcg_cut.20 -c -m recall.1000 -c data/qrels.dev.small.tsv run.msmarco_passage_dev_bm25.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "XXWy1WHZGq_Y",
        "colab": {}
      },
      "source": [
        "!trec_eval.9.0.4/trec_eval -m ndcg_cut.20 -c -m recall.1000 -c data/qrels.dev.small.tsv run.msmarco_passage_dev_bm25_tuned.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xQV_2o-OoDkv",
        "colab": {}
      },
      "source": [
        "!trec_eval.9.0.4/trec_eval -m ndcg_cut.20 -c -m recall.1000 -c data/qrels.dev.small.tsv run.msmarco_passage_dev_bm25_tuned_rm3.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "DdX1XQfKG7QB"
      },
      "source": [
        "**Exercise #5:**\n",
        "What can you infer based on these result?"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "afirm2020_query.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}