afirm2020_index.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/castorini/anserini-notebooks-afirm2020/blob/master/afirm2020_index.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "p4JoAhnmDwlF"
      },
      "source": [
        "# Indexing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wxJfO72PD2Re"
      },
      "source": [
        "In this activity, we are going to index the [MS MARCO](http://www.msmarco.org/) passage collection and explore some features of the index."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "46p_d_BWahwD"
      },
      "source": [
        "## Setup\n",
        "\n",
        "We are going to use the open-source [Anserini](https://github.com/castorini/anserini) information retrieval toolkit to run the experiments.\n",
        "Anserini provides an easy-to-use interface over the popular [Apache Lucene](https://lucene.apache.org/) search library to facilitate rapid experimentation.\n",
        "\n",
        "Here we clone the repository from GitHub but won't be dealing with other facets of Git. If you want to dive deeper into Git, here are some good resources:\n",
        "\n",
        "- [Visual Git](http://marklodato.github.io/visual-git-guide/index-en.html)\n",
        "- [Think Like (a) Git](http://think-like-a-git.net/)\n",
        "- [Git from the bottom up](http://ftp.newartisans.com/pub/git.from.bottom.up.pdf)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "SFIMhczyD0yD",
        "colab": {}
      },
      "source": [
        "!git clone https://github.com/castorini/anserini.git\n",
        "!cd anserini && git checkout ad5ba1c76196436f8a0e28efdb69960d4873efe3"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "NtZfTTOqZh8C"
      },
      "source": [
        "Set up Java 11:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "VLHfZ3PPZjIw",
        "colab": {}
      },
      "source": [
        "!apt-get install -y openjdk-11-jdk-headless\n",
        "%env JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mHNC0K6mfu3I"
      },
      "source": [
        "After cloning the Anserini repo, we should build it using Maven.\n",
        "Please note that Maven has rather verbose logging, but so you only output the final lines that indicate whether or not the build succeeded."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "YI0TsF0sf3qe",
        "colab": {}
      },
      "source": [
        "# Install Maven\n",
        "!apt-get install -q maven\n",
        "\n",
        "# Build Anserini\n",
        "!cd anserini && mvn clean package appassembler:assemble | grep \"BUILD\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "fH8QRjXXaN7R"
      },
      "source": [
        "Now is the time to build the `trec_eval` tool (more on this later):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "32qqBId1gIfd",
        "colab": {}
      },
      "source": [
        "!cd anserini/eval && tar xvfz trec_eval.9.0.4.tar.gz && cd trec_eval.9.0.4 && make"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sOxCxt9wc9OL"
      },
      "source": [
        "## Data Preparation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Y2-J2uyKfh2D"
      },
      "source": [
        "MS MARCO (MicroSoft MAchine Reading COmprehension) is a large-scale dataset that defines many tasks from question answering to ranking.\n",
        "Here we focus on the collection designed for passage re-ranking.\n",
        "\n",
        "This collection is composed of the top 1000 most relevant passages for each query, as retrieved by BM25.\n",
        "\n",
        "First, create a directory named `data/msmarco_passage` to hold the collection. Next, download the MS MARCO passage collection from the Google bucket dedicated to this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P50lUocfojwP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!mkdir -p data/msmarco_passage\n",
        "!gsutil -m cp gs://sigir2020/collection.tsv data/msmarco_passage/collection.tsv"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7tvrnDt_W36P"
      },
      "source": [
        "Let's start exploring the passages in the collection:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "B7PNkiCkXEek",
        "colab": {}
      },
      "source": [
        "file=\"data/msmarco_passage/collection.tsv\"\n",
        "!wc -l $file\n",
        "!head -5 $file"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "zx6gJbW9XZt-"
      },
      "source": [
        "`collection.tsv` contains 8841823 passages, each labeled with a unique passage ID."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "k8EwBpcoYN9G"
      },
      "source": [
        "**Exercise #1:**\n",
        "Find how many passages contain the phrase \"South Africa\" using `grep`, `wc` and the pipe operator `|`.\n",
        "Also output the first 10 passages."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "CgajKG7OYOXQ",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_q7IcAxZZZdS"
      },
      "source": [
        "Now let's look at the development queries.\n",
        "We will be using `queries/qrels.dev.small.tsv` for demonstration purposes, which constitutes a smaller subset of the queries (6,980 queries compared to the 101,093 queries of the original)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JD6M75dGSDsr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!gsutil -m cp gs://sigir2020/qrels.dev.small.tsv data/msmarco_passage/qrels.dev.small.tsv\n",
        "!gsutil -m cp gs://sigir2020/queries.dev.small.tsv data/msmarco_passage/queries.dev.small.tsv"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "VDTUAaI5Zln6",
        "colab": {}
      },
      "source": [
        "!wc -l data/msmarco_passage/queries.dev.small.tsv"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "p4GKPn_eabkm"
      },
      "source": [
        "**Exercise #2:**\n",
        "Output the top 3 queries that contain the term \"football\" and count the total number of words in them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "FM7paOITbuyx",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dAakebUMc2Hr"
      },
      "source": [
        "The final important component of the collection is the qrels files:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "jBtmYJu1c_mc",
        "colab": {}
      },
      "source": [
        "!head -5 data/msmarco_passage/qrels.dev.small.tsv"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hoxreCJ9dHJu"
      },
      "source": [
        "**Exercise #3:**\n",
        "Get the query ID of the query `average pay for nfl football players (39908)` from the previous exercise and see how many passages it is associated with."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "-IzufgMcdRPl",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6eQO468kdwxW"
      },
      "source": [
        "**Exercise #4:**\n",
        "Check the first passage that the query is associated with from the `collection.tsv` file.\n",
        "Does it make sense?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3f6R5cfNdnTw",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "yGxMRIVmgKWP"
      },
      "source": [
        "As you can see, the original MS MARCO collection is a tab-separated values (TSV) file.\n",
        "We need to convert the collection into the jsonl format that can be processed by Anserini.\n",
        "jsonl files contain JSON object per line."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "wwl0o9R1ggZo",
        "colab": {}
      },
      "source": [
        "!cd anserini && python ./src/main/python/msmarco/convert_collection_to_jsonl.py \\\n",
        " --collection_path ../data/msmarco_passage/collection.tsv --output_folder ../data/msmarco_passage/collection_jsonl"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ap421Se6gqa7"
      },
      "source": [
        "The above command should generate 9 jsonl files in our `data/msmarco_passage/collection_jsonl` directory, each with 1M lines (except for the last one, which should have 841,823 lines)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xZR9OEVFgzpy",
        "colab": {}
      },
      "source": [
        "!wc -l data/msmarco_passage/collection_jsonl/*"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "tFmhUkUOexI-"
      },
      "source": [
        "jsonl files are JSON files with keys `id` and `contents`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Xd62bv8bexUm",
        "colab": {}
      },
      "source": [
        "!head -5 data/msmarco_passage/collection_jsonl/docs00.json"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "S7_kBEkxivcs"
      },
      "source": [
        "Let's remove the original files to make room for the index.\n",
        "Check the contents of `data/msmarco_passage` before and after.\n",
        "Alternatively, you can look at your filesystem in the menu on the left by clicking the bottom icon."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "A7zunsLHiyMh",
        "colab": {}
      },
      "source": [
        "!ls data/msmarco_passage\n",
        "!rm data/msmarco_passage/*.tsv\n",
        "!ls data/msmarco_passage\n",
        "!rm -rf sample_data"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "OnEa9PuhEFgT"
      },
      "source": [
        "## Indexing\n",
        "\n",
        "Some common indexing options with Anserini:\n",
        "\n",
        "- `input`: Path to collection\n",
        "- `threads`: Number of threads to run\n",
        "- `collection`: Type of Anserini Collection, e.g., LuceneDocumentGenerator, TweetGenerator (subclass of LuceneDocumentGenerator for TREC Microblog)\n",
        "- `index`: Path to index output\n",
        "- `storePositions`: Boolean flag to store positions\n",
        "- `storeDocvectors`: Boolean flag to store document vbectors\n",
        "- `storeRawDocs`: Boolean flag to store raw document text\n",
        "- `keepStopwords`: Boolean flag to keep stopwords (False by default) \n",
        "- `stemmer`: Stemmer to use ([Porter](http://snowball.tartarus.org/algorithms/porter/stemmer.html) by default)\n",
        "\n",
        "We now have everything in place to index the collection.\n",
        "The process takes about 10 minutes; in the meantime, feel free to ask questions or help others while you wait."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "EG0p1FrCEGh-",
        "colab": {}
      },
      "source": [
        "!cd anserini && sh target/appassembler/bin/IndexCollection -collection JsonCollection -input ../data/msmarco_passage/collection_jsonl \\\n",
        " -index ../indexes/lucene-index.msmarco-passage.pos+docvectors+rawdocs -generator DefaultLuceneDocumentGenerator -threads 9 \\\n",
        " -storePositions -storeDocvectors -storeRaw"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "EXDsGo30a7Pg"
      },
      "source": [
        "Check the size of the index at the specified destination:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kwuGDOW-a97r",
        "colab": {}
      },
      "source": [
        "!du -h indexes/lucene-index.msmarco-passage.pos+docvectors+rawdocs"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "U3F97ppoz7GH"
      },
      "source": [
        "## Explore the Index"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "I3y-nap2bBFX"
      },
      "source": [
        "We can explore the index with [Pyserini](https://github.com/castorini/pyserini), the Python interface to Anserini."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "i7mfRPK8mlKv"
      },
      "source": [
        "### Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "3AgSkzr0mobG"
      },
      "source": [
        "Install Python dependencies:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "CBlm_2LwmqNS",
        "colab": {}
      },
      "source": [
        "!pip install pyjnius==1.2.1\n",
        "!pip install pyserini==0.9.2.0\n",
        "\n",
        "import os\n",
        "os.environ['JAVA_HOME'] = '/usr/lib/jvm/java-11-openjdk-amd64'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qebCOu3vmtVc"
      },
      "source": [
        "Fix known issue with pyjnius (see [this explanation](https://github.com/castorini/pyserini/blob/master/README.md#known-issues) for details):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "V-9Ij-VEmy0Z",
        "colab": {}
      },
      "source": [
        "!mkdir -p /usr/lib/jvm/java-1.11.0-openjdk-amd64/jre/lib/amd64/server/\n",
        "!ln -s /usr/lib/jvm/java-1.11.0-openjdk-amd64/lib/server/libjvm.so /usr/lib/jvm/java-1.11.0-openjdk-amd64/jre/lib/amd64/server/libjvm.so"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "clT0MAQ_HaU8"
      },
      "source": [
        "Let's point Pyserini to the Anserini jar that we built earlier:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "liucF_AqHda1",
        "colab": {}
      },
      "source": [
        "os.environ['ANSERINI_CLASSPATH'] = 'anserini/target'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "JZz5f1CuHODJ",
        "colab": {}
      },
      "source": [
        "from pyserini.index import pyutils\n",
        "\n",
        "index_utils = pyutils.IndexReaderUtils('indexes/lucene-index.msmarco-passage.pos+docvectors+rawdocs')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2qDr_6ITIys1"
      },
      "source": [
        "Collection frequency corresponds to the total number of times a term appears in the index.\n",
        "Document frequency, as the name implies, refers to the number of documents that contains the term.\n",
        "\n",
        "For example, consider a toy index that looks like:\n",
        "\n",
        "```\n",
        "Document 1: \"here is some text here is some more text\"\n",
        "Document 2: \"more texts\"\n",
        "Document 3: \"here is a test\"\n",
        "```\n",
        "\n",
        "The collection frequency of the term `text` is 3 (2 times in Document 1 and once in Document 2).\n",
        "However, its document frequency is 2.\n",
        "Intuitively, document frequency is always equal to or less than collection frequency."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6skB5TMlHTgf"
      },
      "source": [
        "Let's choose a term, say, `played`.\n",
        "We can now compute the collection and document frequencies of the term:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "O5AmhPq0Hd07",
        "colab": {}
      },
      "source": [
        "term = 'played'\n",
        "\n",
        "stemmed_form = index_utils.analyze(term)\n",
        "collection_freq, doc_freq = index_utils.get_term_counts(term)\n",
        "\n",
        "print('Stemmed form: {stemmed_form}\\nCollection frequency: {collection_freq}\\nDocument frequency: {doc_freq}'.format(stemmed_form=stemmed_form, collection_freq=collection_freq, doc_freq=doc_freq))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "QRqFcgDMnEnP"
      },
      "source": [
        "**Exercise #5:**\n",
        "Get the same statistics for the term `playing`.\n",
        "Compare the collection and document frequencies to that of the term `played`.\n",
        "Why are they different/the same?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "EFyle6m4nJun",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "3-mb5NWoHgtn"
      },
      "source": [
        "In simple terms, we can think of the index as a dictionary of terms each of which is a postings list.\n",
        "A postings list includes a list of document IDs that contains a given term, and optionally the number of occurrences in that particular document.\n",
        "Because we also stored the positions while indexing the collection, we can also access the positions at which the term appears.\n",
        "\n",
        "Let's get the postings list for the term `rondebosch`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iaYSPtJfHm8l",
        "colab": {}
      },
      "source": [
        "postings_list = index_utils.get_postings_list('rondebosch')\n",
        "\n",
        "for posting in postings_list:\n",
        "  print('Document ID: {docid} | Term frequency: {term_freq} | Positions: {pos}'.format(docid=posting.docid, term_freq=posting.tf, pos=','.join([str(p) for p in posting.positions])))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "vWH51Cmfp4Ls"
      },
      "source": [
        "**Exercise #6:**\n",
        "Note that tokens are stemmed prior to indexing.\n",
        "For example, both `played` and `playing` would share the same postings list.\n",
        "Confirm this by building the postings list for `playing`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MgEp3nBnc_i0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "m6q8zNrQH2Bb"
      },
      "source": [
        "Let's get its document vector of two documents:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "23T_OfPRHxGk",
        "colab": {}
      },
      "source": [
        "doc_vector1 = index_utils.get_document_vector('2803')\n",
        "doc_vector2 = index_utils.get_document_vector('3983')\n",
        "\n",
        "print(doc_vector1)\n",
        "print(doc_vector2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "XFtWIIQoqbs9"
      },
      "source": [
        "Let's view the original passages:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E5O0urUqZzc2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "raw_doc1 = index_utils.doc_raw('2803')\n",
        "raw_doc2 = index_utils.doc_raw('3983')\n",
        "\n",
        "print(raw_doc1)\n",
        "print(raw_doc2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "FLwxKl9yOEJE"
      },
      "source": [
        "The document vector gives a succinct representation of the overall document.\n",
        "We can use the respective representations of two documents to judge their similarity."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "7HWZtyiBOSHx",
        "colab": {}
      },
      "source": [
        "import math\n",
        "\n",
        "def dot_prod(doc1, doc2):\n",
        "  tokens1 = set(doc1.keys())\n",
        "  tokens2 = set(doc2.keys())\n",
        "  all_tokens = list(tokens1 & tokens2)  # Get common tokens (otherwise different sized dicts)\n",
        "  return sum(doc1[t] * doc2[t] for t in all_tokens)\n",
        "\n",
        "def cosine_similarity(doc1, doc2):\n",
        "  return dot_prod(doc1, doc2) / (math.sqrt(dot_prod(doc1, doc1)) * dot_prod(doc2, doc2))\n",
        "\n",
        "cosine_similarity(doc_vector1, doc_vector2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0ws2OsICu5tM"
      },
      "source": [
        "**Exercise #7:**\n",
        "Pick different pairs of documents, and compute the cosine similarity between them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HFKV7BdRdArr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "name": "afirm2020_index.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}