pyserini_robust04_demo.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "pyserini-robust04-demo.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/castorini/anserini-notebooks/blob/master/pyserini_robust04_demo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vUpmSFflwzR1"
      },
      "source": [
        "# Pyserini Demo on Robust04"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-2PzgU4YxIl3"
      },
      "source": [
        "This notebook provides a brief overview of how to use [Pyserini](http://pyserini.io/), the Python interface to [Anserini](http://anserini.io), to search the collection from the TREC 2004 Robust Track.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XR8bFJc0DgaE"
      },
      "source": [
        "## Installation\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f2vNd7bpJlDZ"
      },
      "source": [
        "Install Python dependencies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E_lt0-pXJia0"
      },
      "source": [
        "%%capture\n",
        "!pip install pyserini==0.12.0\n",
        "\n",
        "import os\n",
        "os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-11-openjdk-amd64\""
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RJ2MrEUuEk4D"
      },
      "source": [
        "## SimpleSearcher Usage"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NkD0kKxW9mHP"
      },
      "source": [
        "You can use the `SimpleSearcher` to search over an index. We can initialize the searcher with a pre-built index, which Pyserini will automatically download:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zVoAZvuAI_la",
        "outputId": "d97e030e-231d-4281-d502-77f9a5f9b9d0"
      },
      "source": [
        "from pyserini.search import SimpleSearcher\n",
        "\n",
        "searcher = SimpleSearcher.from_prebuilt_index('robust04')"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\rindex-robust04-20191213.tar.gz: 0.00B [00:00, ?B/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Attempting to initialize pre-built index robust04.\n",
            "Downloading index at https://git.uwaterloo.ca/jimmylin/anserini-indexes/raw/master/index-robust04-20191213.tar.gz...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "index-robust04-20191213.tar.gz: 1.70GB [00:28, 64.2MB/s]                            \n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Extracting /root/.cache/pyserini/indexes/index-robust04-20191213.tar.gz into /root/.cache/pyserini/indexes/index-robust04-20191213.15f3d001489c97849a010b0a4734d018...\n",
            "Initializing robust04...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N6xHyonHJDKy"
      },
      "source": [
        "Now we can search:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yFZlcqEX0t1f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4b5193e1-e648-4e7a-8240-fec940a3313f"
      },
      "source": [
        "hits = searcher.search('black bear attacks')\n",
        "\n",
        "# Prints the first 10 hits\n",
        "for i in range(0, 10):\n",
        "    print(f'{i+1:2} {hits[i].docid:15} {hits[i].score:.5f}')"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            " 1 LA092790-0015   7.06680\n",
            " 2 LA081689-0039   6.89020\n",
            " 3 FBIS4-16530     6.61630\n",
            " 4 LA102589-0076   6.46450\n",
            " 5 FT932-15491     6.25090\n",
            " 6 FBIS3-12276     6.24630\n",
            " 7 LA091090-0085   6.17030\n",
            " 8 FT922-13519     6.04270\n",
            " 9 LA052790-0205   5.94060\n",
            "10 LA103089-0041   5.90650\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xw6EjwPgPB_b"
      },
      "source": [
        "The `hits` data structure holds the `docid`, the retrieval score, as well as the document content:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lGLZhQgANy3I",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 539
        },
        "outputId": "f04003ef-b8f0-4f26-fc38-f9d85880ca20"
      },
      "source": [
        "from IPython.core.display import display, HTML\n",
        "display(HTML('<div style=\"font-family: Times New Roman; padding-bottom:10px\">' + hits[0].raw + '</div>'))"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div style=\"font-family: Times New Roman; padding-bottom:10px\"><DATE>\n",
              "<P>\n",
              "September 27, 1990, Thursday, Ventura County Edition\n",
              "</P>\n",
              "</DATE>\n",
              "<HEADLINE>\n",
              "<P>\n",
              "HUNGRY WILDLIFE STRAYING INTO SUBURBS;\n",
              "</P>\n",
              "<P>\n",
              "DROUGHT: FOUR DRY YEARS HAVE PARCHED NATIVE VEGETATION, FORCING BOBCATS, BEARS,\n",
              "MOUNTAIN LIONS, DEER AND COYOTES TO FORAGE CLOSER TO INHABITED AREAS.\n",
              "</P>\n",
              "</HEADLINE>\n",
              "<TEXT>\n",
              "<P>\n",
              "Hungry bobcats, bears and mountain lions -- unable to find food in Ventura\n",
              "County's drought-parched forests -- are being pushed out of their natural\n",
              "habitats to scavenge in rural communities, game officials said Wednesday.\n",
              "</P>\n",
              "<P>\n",
              "Two weeks ago, a black bear ripped the door off a trailer home in Rose Valley\n",
              "just north of Ojai. And within the past month, there have been several reports\n",
              "of mountain lions eating livestock near Los Padres National Forest. Several\n",
              "bobcats have been reported near houses in the Ojai Valley.\n",
              "</P>\n",
              "<P>\n",
              "Authorities say that over the past two years they have received twice the\n",
              "complaints -- about 20 a month -- of wild animals in populated areas. The\n",
              "drought is now in its fourth year in California.\n",
              "</P>\n",
              "<P>\n",
              "\"We've been having more and more conflicts with animals,\" said Capt. Roger\n",
              "Reese, with the state Department of Fish and Game. \"The fact is, it's very dry\n",
              "out there, and there just isn't a lot of food and water for them.\"\n",
              "</P>\n",
              "<P>\n",
              "Animal control officials say they are advising residents in rural areas to be\n",
              "aware of the problem. But so far, no one has been attacked by the wild animals,\n",
              "although there have been such attacks reported elsewhere in Southern\n",
              "California, authorities said.\n",
              "</P>\n",
              "<P>\n",
              "Coyotes have been running amok, officials said. Virtually all parts of the\n",
              "county except beach areas probably have been visited at one time or another by\n",
              "coyotes, said Kathy Jenks, director of the Ventura County Department of Animal\n",
              "Regulation.\n",
              "</P>\n",
              "<P>\n",
              "In Ventura, coyotes are often seen in Grant Park above City Hall, and in Arroyo\n",
              "Verde Park in the Ondulando district on the east side, Jenks said.\n",
              "</P>\n",
              "<P>\n",
              "Elsewhere, coyotes have been seen on streets in Thousand Oaks, Moorpark and\n",
              "Simi Valley. The rural, foothill developments are especially vulnerable, she\n",
              "said.\n",
              "</P>\n",
              "<P>\n",
              "Jenks said she advises residents to keep small pets inside, especially at\n",
              "night.\n",
              "</P>\n",
              "<P>\n",
              "There have even been a few cases in which brazen coyotes have attacked family\n",
              "animals in back yards, and a large number of house cats are disappearing,\n",
              "officials said.\n",
              "</P>\n",
              "<P>\n",
              "\"The common house cat is like a fancy feast for a coyote,\" she said. \"They're\n",
              "hungry, they're thirsty and they're coming down out of the hills.\"\n",
              "</P>\n",
              "<P>\n",
              "There have been a few reports of deer grazing in people's yards, Reese said.\n",
              "</P>\n",
              "<P>\n",
              "Traditionally, September is the worst month for wildlife, authorities said.\n",
              "</P>\n",
              "<P>\n",
              "\"It is usually the driest month,\" Reese said. \"And a lot of animals that have\n",
              "been raised in the spring leave their parents and go in search of food.\"\n",
              "</P>\n",
              "<P>\n",
              "More animals are expected to leave their natural habitats if the drought\n",
              "continues, officials said.\n",
              "</P>\n",
              "<P>\n",
              "\"I tell people who call that if we didn't have the big cats and the coyotes, we\n",
              "would be overrun by rodents,\" Jenks said. \"I would much rather hear a coyote in\n",
              "the distance than have roof rats.\"\n",
              "</P>\n",
              "<P>\n",
              "Don DeBusschere, who lives on a 45-acre walnut orchard in Happy Valley near\n",
              "Ojai, said his family has grown accustomed to wild animals. DeBusschere said he\n",
              "has seen scores of coyotes, several deer and a black bear.\n",
              "</P>\n",
              "<P>\n",
              "Recently, he said, two bobcats have moved into the trees on the edge of his\n",
              "property.\n",
              "</P>\n",
              "<P>\n",
              "\"They're not out to get humans,\" DeBusschere said. \"They're just trying to make\n",
              "a living off the land.\"\n",
              "</P>\n",
              "</TEXT></div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JOLbQXmPOvbJ"
      },
      "source": [
        "Here's how you can configure search options, such as BM25 parameters and using relevance feedback."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qHKYXSUGN76K",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "96e8f70c-6041-4bff-c2e7-af18a7b4f211"
      },
      "source": [
        "searcher.set_bm25(0.9, 0.4)\n",
        "searcher.set_rm3(10, 10, 0.5)\n",
        "\n",
        "hits2 = searcher.search('black bear attacks')\n",
        "\n",
        "# Prints the first 10 hits\n",
        "for i in range(0, 10):\n",
        "    print(f'{i+1:2} {hits2[i].docid:15} {hits2[i].score:.5f}')"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            " 1 LA092790-0015   1.97260\n",
            " 2 FR940902-1-00057 1.78400\n",
            " 3 FR940603-0-00087 1.77910\n",
            " 4 LA102589-0076   1.77360\n",
            " 5 FR940902-1-00061 1.76380\n",
            " 6 LA081689-0039   1.76310\n",
            " 7 FR940902-1-00070 1.73970\n",
            " 8 FR940902-1-00069 1.73420\n",
            " 9 LA091090-0085   1.73220\n",
            "10 FR940603-0-00102 1.72650\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Vgo98wnPUQO"
      },
      "source": [
        "Note that the results have changed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_aSGvvu57R6g"
      },
      "source": [
        "## IndexReaderUtils Usage"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "emoSTga_7fOB"
      },
      "source": [
        "The `IndexReaderUtils` class provides various methods to read the index directly. For example, we can fetch a raw document from the index given its `docid`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N5ApT1YG71mz",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 590
        },
        "outputId": "f90128cb-d1a9-48c5-ecca-ad0b8376bfef"
      },
      "source": [
        "from pyserini.index import IndexReader\n",
        "\n",
        "reader = IndexReader.from_prebuilt_index('robust04')\n",
        "\n",
        "doc = reader.doc('LA092790-0015').raw()\n",
        "display(HTML('<div style=\"font-family: Times New Roman; padding-bottom:10px\">' + doc + '</div>'))"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Attempting to initialize pre-built index robust04.\n",
            "/root/.cache/pyserini/indexes/index-robust04-20191213.15f3d001489c97849a010b0a4734d018 already exists, skipping download.\n",
            "Initializing robust04...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div style=\"font-family: Times New Roman; padding-bottom:10px\"><DATE>\n",
              "<P>\n",
              "September 27, 1990, Thursday, Ventura County Edition\n",
              "</P>\n",
              "</DATE>\n",
              "<HEADLINE>\n",
              "<P>\n",
              "HUNGRY WILDLIFE STRAYING INTO SUBURBS;\n",
              "</P>\n",
              "<P>\n",
              "DROUGHT: FOUR DRY YEARS HAVE PARCHED NATIVE VEGETATION, FORCING BOBCATS, BEARS,\n",
              "MOUNTAIN LIONS, DEER AND COYOTES TO FORAGE CLOSER TO INHABITED AREAS.\n",
              "</P>\n",
              "</HEADLINE>\n",
              "<TEXT>\n",
              "<P>\n",
              "Hungry bobcats, bears and mountain lions -- unable to find food in Ventura\n",
              "County's drought-parched forests -- are being pushed out of their natural\n",
              "habitats to scavenge in rural communities, game officials said Wednesday.\n",
              "</P>\n",
              "<P>\n",
              "Two weeks ago, a black bear ripped the door off a trailer home in Rose Valley\n",
              "just north of Ojai. And within the past month, there have been several reports\n",
              "of mountain lions eating livestock near Los Padres National Forest. Several\n",
              "bobcats have been reported near houses in the Ojai Valley.\n",
              "</P>\n",
              "<P>\n",
              "Authorities say that over the past two years they have received twice the\n",
              "complaints -- about 20 a month -- of wild animals in populated areas. The\n",
              "drought is now in its fourth year in California.\n",
              "</P>\n",
              "<P>\n",
              "\"We've been having more and more conflicts with animals,\" said Capt. Roger\n",
              "Reese, with the state Department of Fish and Game. \"The fact is, it's very dry\n",
              "out there, and there just isn't a lot of food and water for them.\"\n",
              "</P>\n",
              "<P>\n",
              "Animal control officials say they are advising residents in rural areas to be\n",
              "aware of the problem. But so far, no one has been attacked by the wild animals,\n",
              "although there have been such attacks reported elsewhere in Southern\n",
              "California, authorities said.\n",
              "</P>\n",
              "<P>\n",
              "Coyotes have been running amok, officials said. Virtually all parts of the\n",
              "county except beach areas probably have been visited at one time or another by\n",
              "coyotes, said Kathy Jenks, director of the Ventura County Department of Animal\n",
              "Regulation.\n",
              "</P>\n",
              "<P>\n",
              "In Ventura, coyotes are often seen in Grant Park above City Hall, and in Arroyo\n",
              "Verde Park in the Ondulando district on the east side, Jenks said.\n",
              "</P>\n",
              "<P>\n",
              "Elsewhere, coyotes have been seen on streets in Thousand Oaks, Moorpark and\n",
              "Simi Valley. The rural, foothill developments are especially vulnerable, she\n",
              "said.\n",
              "</P>\n",
              "<P>\n",
              "Jenks said she advises residents to keep small pets inside, especially at\n",
              "night.\n",
              "</P>\n",
              "<P>\n",
              "There have even been a few cases in which brazen coyotes have attacked family\n",
              "animals in back yards, and a large number of house cats are disappearing,\n",
              "officials said.\n",
              "</P>\n",
              "<P>\n",
              "\"The common house cat is like a fancy feast for a coyote,\" she said. \"They're\n",
              "hungry, they're thirsty and they're coming down out of the hills.\"\n",
              "</P>\n",
              "<P>\n",
              "There have been a few reports of deer grazing in people's yards, Reese said.\n",
              "</P>\n",
              "<P>\n",
              "Traditionally, September is the worst month for wildlife, authorities said.\n",
              "</P>\n",
              "<P>\n",
              "\"It is usually the driest month,\" Reese said. \"And a lot of animals that have\n",
              "been raised in the spring leave their parents and go in search of food.\"\n",
              "</P>\n",
              "<P>\n",
              "More animals are expected to leave their natural habitats if the drought\n",
              "continues, officials said.\n",
              "</P>\n",
              "<P>\n",
              "\"I tell people who call that if we didn't have the big cats and the coyotes, we\n",
              "would be overrun by rodents,\" Jenks said. \"I would much rather hear a coyote in\n",
              "the distance than have roof rats.\"\n",
              "</P>\n",
              "<P>\n",
              "Don DeBusschere, who lives on a 45-acre walnut orchard in Happy Valley near\n",
              "Ojai, said his family has grown accustomed to wild animals. DeBusschere said he\n",
              "has seen scores of coyotes, several deer and a black bear.\n",
              "</P>\n",
              "<P>\n",
              "Recently, he said, two bobcats have moved into the trees on the edge of his\n",
              "property.\n",
              "</P>\n",
              "<P>\n",
              "\"They're not out to get humans,\" DeBusschere said. \"They're just trying to make\n",
              "a living off the land.\"\n",
              "</P>\n",
              "</TEXT></div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1RMfMgWt8bgm"
      },
      "source": [
        "Note that the result is exactly the same as displaying the hit contents above. Given the raw text, we can obtain its analyzed form (i.e., tokenized, stemmed, stopwords removed, etc.). Here we show the first ten tokens:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZgNGM65F6m5h",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4905770a-5a1a-4655-bd27-ad2ba6d316b0"
      },
      "source": [
        "analyzed = reader.analyze(doc)\n",
        "analyzed[0:10]"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['date',\n",
              " 'p',\n",
              " 'septemb',\n",
              " '27',\n",
              " '1990',\n",
              " 'thursdai',\n",
              " 'ventura',\n",
              " 'counti',\n",
              " 'edit',\n",
              " 'p']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z5OUXedN89Yf"
      },
      "source": [
        "The index also stores the raw document vector, which we can obtain as a Python dictionary of analyzed terms to counts (i.e., term frequency).\n",
        "For brevity, we only look at terms that appear more than once:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mMtneeJw8HDI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "93292331-b837-41fc-c6c4-d12f7933ead3"
      },
      "source": [
        "doc_vector = reader.get_document_vector('LA092790-0015')\n",
        "{ k: v for (k, v) in doc_vector.items() if v >1 }"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'advis': 2,\n",
              " 'anim': 9,\n",
              " 'area': 4,\n",
              " 'attack': 3,\n",
              " 'author': 3,\n",
              " 'bear': 4,\n",
              " 'been': 11,\n",
              " 'black': 2,\n",
              " 'bobcat': 4,\n",
              " 'california': 2,\n",
              " 'cat': 3,\n",
              " 'counti': 4,\n",
              " 'coyot': 10,\n",
              " 'debusscher': 3,\n",
              " 'deer': 3,\n",
              " 'depart': 2,\n",
              " 'drought': 4,\n",
              " 'dry': 2,\n",
              " 'elsewher': 2,\n",
              " 'especi': 2,\n",
              " 'famili': 2,\n",
              " 'few': 2,\n",
              " 'food': 3,\n",
              " 'forest': 2,\n",
              " 'game': 2,\n",
              " 'ha': 3,\n",
              " 'habitat': 2,\n",
              " 'have': 16,\n",
              " 'he': 2,\n",
              " 'hi': 2,\n",
              " 'hous': 3,\n",
              " 'hungri': 3,\n",
              " 'i': 2,\n",
              " 'jenk': 4,\n",
              " 'just': 3,\n",
              " 'leav': 2,\n",
              " 'lion': 3,\n",
              " 'live': 2,\n",
              " 'lot': 2,\n",
              " 'month': 4,\n",
              " 'more': 3,\n",
              " 'mountain': 3,\n",
              " 'natur': 2,\n",
              " 'near': 3,\n",
              " 'off': 2,\n",
              " 'offici': 5,\n",
              " 'ojai': 3,\n",
              " 'on': 2,\n",
              " 'out': 4,\n",
              " 'parch': 2,\n",
              " 'park': 2,\n",
              " 'past': 2,\n",
              " 'peopl': 2,\n",
              " 'rees': 3,\n",
              " 'report': 4,\n",
              " 'resid': 2,\n",
              " 'rural': 3,\n",
              " 'sai': 2,\n",
              " 'said': 19,\n",
              " 'seen': 3,\n",
              " 'septemb': 2,\n",
              " 'sever': 3,\n",
              " 'she': 3,\n",
              " \"they'r\": 5,\n",
              " 'two': 3,\n",
              " 'vallei': 4,\n",
              " 'ventura': 4,\n",
              " 'we': 2,\n",
              " 'who': 2,\n",
              " 'wild': 3,\n",
              " 'wildlif': 2,\n",
              " 'would': 2,\n",
              " 'yard': 2,\n",
              " 'year': 3}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    }
  ]
}