extractive_qa_haystack.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/patzacher/extractive_qa/blob/main/extractive_qa_haystack.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gHBuehT_Ge3D"
      },
      "source": [
        "# Extractive QA System Using Python and Haystack\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oPCJAtEgWH9h"
      },
      "source": [
        "##Overview\n",
        "\n",
        "Haystack is an end-to-end open-source framework for creating Question-Answering models. Haystack has three primary components: the DocumentStore, Retriever, and Reader.\n",
        "\n",
        "1. DocumentStore: This is exactly what it sounds like. The DocumentStore stores text documents and their meta data. Documents are typically split into smaller units (e.g., paragraphs) before indexing to enable higher accuracy and granularity to answers.\n",
        "\n",
        "2. Retriever: These are fast and simple algorithms to identify candidate passages from a large collection of documents. It allows a set of k-candidate documents to be sent to the Reader. In general, the Retriever helps narrow the scope for the Reader, which will then perform a thorough search of the top-k documents for the best answer.\n",
        "\n",
        "3. Reader: Takes passages of text as input and returns top-k answers with their corresponding confidence scores (range 0-1). Readers are powerful models that are able to make a full search in the selected documents with the aim of finding the right answer.\n",
        "\n",
        "The DocumentStore, Retriever, and Reader are connected using a querying pipeline. Querying pipelines are used to receive a query from the user and produce a result."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yk-MoMiuGe3H"
      },
      "source": [
        "\n",
        "## Preparing the Colab Environment\n",
        "\n",
        "- [Enable GPU Runtime](https://docs.haystack.deepset.ai/docs/enabling-gpu-acceleration#enabling-the-gpu-in-colab)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YLwWtfnaGe3I"
      },
      "source": [
        "## Install Packages\n",
        "\n",
        "Install Haystack and other required packages:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZuyiNhlxGe3I"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "\n",
        "pip install --upgrade pip\n",
        "pip install farm-haystack[colab,preprocessing,elasticsearch,inference]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VKXyFU7LGe3K"
      },
      "source": [
        "Configure Haystack's logging level:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MqquSsg4Ge3K"
      },
      "outputs": [],
      "source": [
        "import logging\n",
        "\n",
        "logging.basicConfig(format=\"%(levelname)s - %(name)s -  %(message)s\", level=logging.DEBUG)\n",
        "logging.getLogger(\"haystack\").setLevel(logging.DEBUG)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "etiROvfEvOJK"
      },
      "source": [
        "## Import Packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "SSMe_Mz8vUS2"
      },
      "outputs": [],
      "source": [
        "from google.colab import files\n",
        "\n",
        "import requests\n",
        "import re\n",
        "import os"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EQ_pjPsgGe3K"
      },
      "source": [
        "## Initialize the ElasticsearchDocumentStore\n",
        "\n",
        "A DocumentStore stores the documents that the question-answering system uses to find answers to questions. Here, we're using the [`ElasticsearchDocumentStore`](https://docs.haystack.deepset.ai/reference/document-store-api#module-elasticsearch) which connects to a running Elasticsearch service. It's a fast and scalable text-focused storage option. This service runs independently from Haystack and persists even after the Haystack program has finished running. To learn more about the DocumentStore and the different types of external databases that we support, see [DocumentStore](https://docs.haystack.deepset.ai/docs/document_store).\n",
        "\n",
        "As an aside, Elasticsearch is an open-source, distributed search and analytics engine designed for scalability, real-time searching, and data analysis. Among other things, Elasticsearch can index and search large volumes of text data quickly and efficiently."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JriRkIzFGe3L"
      },
      "source": [
        "1. Download, extract, and set the permissions for the Elasticsearch installation image:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TB1BOcHqGe3L"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "\n",
        "# Use `wget` utility to quietly (-q) download the Elasticsearch archive from\n",
        "# this URL\n",
        "wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q\n",
        "\n",
        "# After downloading, use `tar` to extract contents (-x extract, -z archive is\n",
        "# compressed with gzip, -f specifies file name)\n",
        "tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz\n",
        "\n",
        "# Change ownership and group ownership (`chown`) of extracted files and\n",
        "# directories to `daemon`, a non-priveleged user and group name for running\n",
        "# Elastic search in a secure manner.\n",
        "chown -R daemon:daemon elasticsearch-7.9.2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aVuV-JI8Ge3L"
      },
      "source": [
        "2. Start the server:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k2Wtrtg9Ge3L"
      },
      "outputs": [],
      "source": [
        "%%bash --bg\n",
        "\n",
        "# Start Elasticsearch server as the `daemon` user.\n",
        "sudo -u daemon -- elasticsearch-7.9.2/bin/elasticsearch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qssTnmXtGe3L"
      },
      "source": [
        "If Docker is available in your environment (Colab notebooks do not support Docker), you can also start Elasticsearch using Docker. You can do this manually, or using our [`launch_es()`](https://docs.haystack.deepset.ai/reference/utils-api#module-doc_store) utility function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4I5Wr28JGe3L"
      },
      "outputs": [],
      "source": [
        "# from haystack.utils import launch_es\n",
        "\n",
        "# launch_es()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xH6x4JgMGe3M"
      },
      "source": [
        "3. Wait 30 seconds for the server to fully start up:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qIEyRnVgGe3M"
      },
      "outputs": [],
      "source": [
        "import time\n",
        "\n",
        "time.sleep(30)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UUVPj5t9Ge3M"
      },
      "source": [
        "4. Initialize the ElasticsearchDocumentStore:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NqS3F9KiGe3M",
        "outputId": "a5c1c987-9773-4df0-c080-793e315ded19"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:haystack.telemetry:Haystack sends anonymous usage data to understand the actual usage and steer dev efforts towards features that are most meaningful to users. You can opt-out at anytime by manually setting the environment variable HAYSTACK_TELEMETRY_ENABLED as described for different operating systems in the [documentation page](https://docs.haystack.deepset.ai/docs/telemetry#how-can-i-opt-out). More information at [Telemetry](https://docs.haystack.deepset.ai/docs/telemetry).\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "from haystack.document_stores import ElasticsearchDocumentStore\n",
        "\n",
        "# Get the host where Elasticsearch is running, default to localhost\n",
        "host = os.environ.get(\"ELASTICSEARCH_HOST\", \"localhost\")\n",
        "\n",
        "# Configure ElasticsearchDocumentStore for accessing and storing documents.\n",
        "document_store = ElasticsearchDocumentStore(host=host, username=\"\", password=\"\", index=\"document\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y1P1fsVWGe3M"
      },
      "source": [
        "ElasticsearchDocumentStore is up and running and ready to store the Documents."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5v7j65x4Ge3M"
      },
      "source": [
        "## Upload Files\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FfILuDP-3DOh"
      },
      "source": [
        "####**Option 1:** Upload Previously Scraped Data\n",
        "\n",
        "Step 1.\n",
        "\n",
        "If you have text data that you would like to use, upload the files here."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CXLUyGk2iwZ0"
      },
      "outputs": [],
      "source": [
        "from google.colab import files\n",
        "\n",
        "# Create new directory for scraped content.\n",
        "os.mkdir(\"/content/Data\")\n",
        "\n",
        "# Switch to new directory.\n",
        "os.chdir(\"/content/Data\")\n",
        "\n",
        "files.upload()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZZ2J9Fgsi1hu"
      },
      "source": [
        "Step 2.\n",
        "\n",
        "If you have a meta data file, you can set `meta_file` from meta_file.txt for use in the indexing pipeline. The meta data file must be a list of dictionaries with entries in the same order as the documents contained in your `Data` directory. If this doesn't apply, skip this step, but remember to switch back to the parent directory (\"/content/\")."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cM5cGJxKbz6W"
      },
      "outputs": [],
      "source": [
        "meta_file_path = \"/content/Data/meta_file.txt\"\n",
        "\n",
        "# Read meta_file.txt\n",
        "with open(meta_file_path, \"r\") as file:\n",
        "  meta_data = file.read()\n",
        "\n",
        "# Split the input string into items based on the newline characters\n",
        "meta_items = meta_data.strip().split('\\n\\n')\n",
        "\n",
        "# Initialize a list to store dictionaries\n",
        "meta_file = []\n",
        "\n",
        "# Iterate through the items and create the list of dictionaries contained in\n",
        "# `meta_file`.\n",
        "for item in meta_items:\n",
        "    item_lines = item.split('\\n')\n",
        "    item_dict = {}\n",
        "\n",
        "    for line in item_lines:\n",
        "        key, value = line.split(': ', 1)\n",
        "        item_dict[key] = value\n",
        "\n",
        "    meta_file.append(item_dict)\n",
        "\n",
        "\n",
        "# Set directory where documents are located.\n",
        "doc_dir = \"/content/Data\"\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Switch back to parent directory.\n",
        "os.chdir(\"/content/\")"
      ],
      "metadata": {
        "id": "S8bTPtcuP_Ag"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jk77XNf53GJd"
      },
      "source": [
        "####**Option 2: Example Dataset from Haystack Tutorial**\n",
        "\n",
        "Download 517 articles from the Game of Thrones Wikipedia. You can find them in *data/build_a_scalable_question_answering_system* as a set of *.txt* files."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I_HJ0yNKGe3M"
      },
      "outputs": [],
      "source": [
        "from haystack.utils import fetch_archive_from_http\n",
        "\n",
        "doc_dir = \"data/build_a_scalable_question_answering_system\"\n",
        "\n",
        "fetch_archive_from_http(\n",
        "    url=\"https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt3.zip\",\n",
        "    output_dir=doc_dir,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5WL7qV72Ge3M"
      },
      "source": [
        "## Index Documents with a Pipeline\n",
        "\n",
        "Indexing pipelines prepare the files for search. The main objective here is to convert files (.txt, in our case) into Haystack Documents, so they can be saved in a DocumentStore. Our indexing pipeline will have three nodes:\n",
        "\n",
        "1. `TextConverter`, which turns `.txt` files into Haystack `Document` objects and sends to the `PreProcessor`.\n",
        "2. `PreProcessor`, which cleans and splits the text within a `Document` and sends to the `DocumentStore`.\n",
        "3. `DocumentStore` is the database that stores text and meta data and provides them to the Retriever at query time. Our `ElasticsearchDocumentStore` has already been initialized.\n",
        "\n",
        "Once we combine these nodes into a pipeline, the pipeline will ingest `.txt` file paths, preprocess them, and write them into the DocumentStore.\n",
        "\n",
        "Note: More nodes are available for our indexing pipeline as needed. For example, a `FileClassifier` can be added as the first node to classify files into text, PDF, Markdown, docx, and HTML files and route them to the appropriate `FileConverter`. Also, a `DocumentClassifier` could be used to attach a classification label each Document's meta data (e.g., sentiment labels like \"positive\", \"negative\").\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-7ozcoDCy0pM"
      },
      "source": [
        "####**Option 1:**\n",
        "\n",
        "Step 1.\n",
        "\n",
        "Manually apply the converter(s) to each file. If we are only using one type of file (in this case, .txt), then we can specify the type of file converter we want to use.\n",
        "\n",
        "Next, intialize the preprocessor with recommended values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q99YBCeNGe3N",
        "outputId": "1c40772c-2c29-4292-a6c7-59d62e1c8c73"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        }
      ],
      "source": [
        "from haystack import Pipeline\n",
        "from haystack.nodes import TextConverter, PreProcessor\n",
        "\n",
        "indexing_pipeline = Pipeline() # Initialize the indexing pipeline\n",
        "text_converter = TextConverter() # Reads text from .txt file\n",
        "                                 # Sends to preprocessor\n",
        "preprocessor = PreProcessor(\n",
        "    clean_whitespace=True, # Remove whitespace at start/end of each line in text\n",
        "    clean_header_footer=True, # Remove repeated header/footer\n",
        "    clean_empty_lines=True, # Normalize 3+ empty lines to 2 empty lines\n",
        "    split_by=\"word\", # Unit to split document by\n",
        "    split_length=100, # Max number of units per document (Recommended value)\n",
        "    split_overlap=10, # Overlap between adjacent documents\n",
        "    split_respect_sentence_boundary=True, # Doc boundaries preserve sentences\n",
        "    max_chars_check = 1000 # Some docs have very long passages so we need to split them\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RHpxSnPjGe3N"
      },
      "source": [
        "To learn more about the parameters of the `PreProcessor`, see [Usage](https://docs.haystack.deepset.ai/docs/preprocessor#usage).\n",
        "\n",
        "[Document splitting](https://docs.haystack.deepset.ai/docs/optimization#document-length) is important for your question answering system's performance. If you halve the length of your documents, you will halve the workload placed on your Retriever. Depending on the type of Retriever used, the maximum number of words will vary (between 100 - 500 words).\n",
        "\n",
        "Our current pipeline uses a dense retriever, which have more restrictive guidelines for sentence length. We have to ensure that documents are not longer than the retriever's maximium input length (256 tokens). As such, decent performance has been found with documents around 100 words long (see [Optimization - Document Length](https://docs.haystack.deepset.ai/docs/optimization#document-length) for more details)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1hF85uMzGe3N"
      },
      "source": [
        "Step 2.\n",
        "\n",
        "Add the nodes into an indexing pipeline. You should provide the `name` or `name`s of preceding nodes as the `input` argument. Note that in an indexing pipeline, the input to the first node is `File`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UX4FpYjbGe3N"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "indexing_pipeline.add_node(component=text_converter, name=\"TextConverter\", inputs=[\"File\"])\n",
        "indexing_pipeline.add_node(component=preprocessor, name=\"PreProcessor\", inputs=[\"TextConverter\"])\n",
        "indexing_pipeline.add_node(component=document_store, name=\"DocumentStore\", inputs=[\"PreProcessor\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dWvY4UngGe3N"
      },
      "source": [
        "Step 3.\n",
        "\n",
        "Run the indexing pipeline to write the text data into the DocumentStore. We can add metadata to our files using the `meta` argument in the `indexing_pipeline.run_batch` command. For example, we can include the title and URL of a document and return them for additional context when the user asks a question.\n",
        "\n",
        "Note that `meta_file` has to be a list of dictionaries, the same length as `files_to_index`. Also, we are alphabetically sorting these arguments because the entries in `meta` must be in the same order as `file_paths`. If you are not using a `meta_file` remove that argument from the `run_batch` command."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XLc1sONQGe3N"
      },
      "outputs": [],
      "source": [
        "# Specify the files we want to send to the DocumentStore.\n",
        "files_to_index = [\n",
        "    os.path.join(doc_dir, f)\n",
        "    for f in os.listdir(doc_dir)\n",
        "    if f != \"meta_file.txt\" # Don't include metadata file because we read that\n",
        "]                           # separately.\n",
        "\n",
        "# Run our indexing pipeline to convert files, preprocess, and store them.\n",
        "indexing_pipeline.run_batch(file_paths=sorted(files_to_index),\n",
        "                            meta=sorted(meta_file, key=lambda x: x['Title']))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z0ZXYIsZaq26"
      },
      "source": [
        "####**Option 2:**\n",
        "Haystack has a convenience function that will automatically apply the right converter to each file in a directory instead of having to specify a converter (i.e., for pdf, docx, txt). See [Better Retrieval via Embedding Retrieval](https://haystack.deepset.ai/tutorials/06_better_retrieval_via_embedding_retrieval) tutorial for an example."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ELdfRVORXCJw"
      },
      "outputs": [],
      "source": [
        "from haystack.utils import convert_files_to_docs\n",
        "\n",
        "all_docs = convert_files_to_docs(dir_path=doc_dir)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FUwMq-nhGe3O"
      },
      "source": [
        "## Initializing the Retriever\n",
        "\n",
        "Now that the Documents are in the DocumentStore, let's initialize the nodes we want to use in our query pipeline. First, a Retreiver.\n",
        "\n",
        "In a query pipeline, the Retriever takes a query as input and checks it against the documents contained in the DocumentStore. It scores each document for its relevance to the query and returns the top candidates (top-k documents) to the Reader. The Reader will then perform the more complex task of question-answering using transformer-based language models (if using a dense retriever).\n",
        "\n",
        "Two (out of many) Retriever options are the **BM25Retriever** (no GPU needed) and an **EmbeddingRetriever** with Sentence Transformers models (recommended if we have a GPU available). The BM25Retriever is a *sparse* retriever while the EmbeddingRetriever is *dense*.\n",
        "\n",
        "Sparse methods operate by looking for shared keywords between the document and the query. Dense approaches perform better than sparse counterparts, but are computationally more expensive. The models used by the EmbeddingRetriever are trained to embed similar sentences close to each other in a shared embedding space.\n",
        "\n",
        "A starting model for a dense Retriever is the `multi-qa-mpnet-base-dot-v1` as it was tuned for semantic search (i.e., given a query, it can find relevant passages). It was trained on a large and diverse set of question/answer pairs. Downside is that it is one of the larger models (420 MB), while a smaller, similar option might be the `multi-qa-MiniLM-L6-cos-V1` (80 MB). Here we have to consider model size with performance, as the smaller model generally has poorer performance.\n",
        "\n",
        "For more Retriever options, see [Retriever](https://docs.haystack.deepset.ai/docs/retriever).\n",
        "\n",
        "For model info, see [Pretrained Models](https://www.sbert.net/docs/pretrained_models.html#).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kVaayXdhbK6v"
      },
      "source": [
        "####**Option 1: EmbeddingRetriever**\n",
        "\n",
        "Let's use the `multi-qa-mpnet-base-dot-v1` model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tRYrloYybr-7"
      },
      "outputs": [],
      "source": [
        "from haystack.nodes import EmbeddingRetriever\n",
        "\n",
        "retriever = EmbeddingRetriever(\n",
        "    document_store=document_store, embedding_model=\"sentence-transformers/multi-qa-mpnet-base-dot-v1\"\n",
        ")\n",
        "# Important:\n",
        "# Now that we initialized the Retriever, we need to call update_embeddings() to iterate over all\n",
        "# previously indexed documents and update their embedding representation.\n",
        "# While this can be a time consuming operation (depending on the corpus size), it only needs to be done once.\n",
        "# At query time, we only need to embed the query and compare it to the existing document embeddings, which is very fast.\n",
        "document_store.update_embeddings(retriever)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M2Z9GaiLbNRF"
      },
      "source": [
        "####**Option 2: BM25 Retriever**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hlp4VChQGe3O"
      },
      "outputs": [],
      "source": [
        "from haystack.nodes import BM25Retriever\n",
        "\n",
        "retriever = BM25Retriever(document_store=document_store)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C7E-XiJyl4pZ"
      },
      "source": [
        "##Route Documents##\n",
        "\n",
        "Now that the Retriever has been initialized, we can move on specifying our approach to routing documents. We can use the EmbeddingRetriever to retrieve both texts and tables. To do question-answering on these documents, we need to route the \"text\" documents to a FARMReader and \"table\" documents to a TableReader. Then we need to join the answers coming from the two Readers to a single list of answers.\n",
        "\n",
        "To read more about this process, see [Pipeline for QA on Combination of Text and Tables](https://haystack.deepset.ai/tutorials/15_tableqa) including how to evaluate the pipeline and how to add tables from PDFs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Srr7O0JhGe3O"
      },
      "source": [
        "## Initializing the Reader\n",
        "\n",
        "Our query pipeline also needs a Reader, so we'll initialize it next. A Reader scans the texts it received from the Retriever and extracts the top answer candidates. Readers are based on powerful deep learning models but are much slower than Retrievers at processing the same amount of text. This is due to the model complexity (e.g., number of parameters), but also the difficulty of the task. Readers must process the text within the selected documents to extract the answer to a question, which involves fine-grained language understanding and reasoning.\n",
        "\n",
        "We'll use a FARMReader with a base-sized RoBERTa question answering model called [`deepset/roberta-base-squad2`](https://huggingface.co/deepset/roberta-base-squad2). It's a good all-round model to start with and has been trained on QA pairs, including unanswerable questions, for the task of question-answering.\n",
        "\n",
        "See [Models](https://docs.haystack.deepset.ai/docs/reader#models) for more options."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KsMW8rq2Ge3O"
      },
      "outputs": [],
      "source": [
        "from haystack.nodes import FARMReader, TableReader, RouteDocuments, JoinAnswers\n",
        "\n",
        "text_reader = FARMReader(model_name_or_path=\"deepset/roberta-base-squad2\", context_window_size=300, use_gpu=True)\n",
        "table_reader = TableReader(model_name_or_path=\"deepset/tapas-large-nq-hn-reader\")\n",
        "route_documents = RouteDocuments()\n",
        "join_answers = JoinAnswers()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CdSctDS1Ge3O"
      },
      "source": [
        "## Creating the Retriever-Reader Pipeline\n",
        "\n",
        "You can combine the Reader and Retriever in a querying pipeline using the `Pipeline` class. The combination of the two speeds up processing because the Reader only processes the Documents that it received from the Retriever.\n",
        "\n",
        "To speed things up, Haystack comes with a few predefined pipelines. One of them is the `ExtractiveQAPipeline` that combines a retriever and a reader to answer questions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TfR2irHeGe3S"
      },
      "source": [
        "**Option 1: Manually Define a Pipeline**\n",
        "\n",
        "*We'll use this option if we expect some answers to be contained within tables.\n",
        "\n",
        "Initialize the `Pipeline` object and add the Retriever and Reader as nodes. You should provide the `name` or `name`s of preceding nodes as the input argument. Note that in a querying pipeline, the input to the first node is `Query`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vbHee5AWGe3S"
      },
      "outputs": [],
      "source": [
        "from haystack import Pipeline\n",
        "\n",
        "text_table_qa_pipeline = Pipeline()\n",
        "text_table_qa_pipeline.add_node(component=retriever, name=\"EmbeddingRetriever\", inputs=[\"Query\"])\n",
        "text_table_qa_pipeline.add_node(component=route_documents, name=\"RouteDocuments\", inputs=[\"EmbeddingRetriever\"])\n",
        "text_table_qa_pipeline.add_node(component=text_reader, name=\"TextReader\", inputs=[\"RouteDocuments.output_1\"])\n",
        "text_table_qa_pipeline.add_node(component=table_reader, name=\"TableReader\", inputs=[\"RouteDocuments.output_2\"])\n",
        "text_table_qa_pipeline.add_node(component=join_answers, name=\"JoinAnswers\", inputs=[\"TextReader\", \"TableReader\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sELy0Byw8zFM"
      },
      "source": [
        "**Option 2: Predefined Pipeline**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mirGmx8V8yce"
      },
      "outputs": [],
      "source": [
        "from haystack.pipelines import ExtractiveQAPipeline\n",
        "\n",
        "pipe = ExtractiveQAPipeline(text_reader, retriever)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4sGnTV3KGe3T"
      },
      "source": [
        "That's it! The pipeline is ready to answer questions!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cOKl5WS8Ge3T"
      },
      "source": [
        "## Ask your Question\n",
        "\n",
        "1. Use the pipeline's `run()` method to ask a question. The query argument is where you type your question. Additionally, you can set the number of documents you want the Reader and Retriever to return using the `top-k` parameter. The `top-k` parameter in both the Retriever and Reader determine how many results they return and is a trade-off between speed and accuracy. Specifically, Retriever top-k dictates how many retrieved documents are passed on to the Reader, while Reader top-k determines how many answer candidates to show. Haystack recommends using a Retriever top-k = 10 for decent overall performance.\n",
        "\n",
        "To learn more about setting arguments, see [Arguments](https://docs.haystack.deepset.ai/docs/pipelines#arguments).\n",
        "\n",
        "To read more about the `top-k` parameter, see [Choosing the Right top-k Values](https://docs.haystack.deepset.ai/docs/optimization#choosing-the-right-top-k-values).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "t4-K7x0XGe3T"
      },
      "outputs": [],
      "source": [
        "question = \"what is haystack?\"\n",
        "\n",
        "# Wrap prediction pipeline in a try/except statement to prevent errors from\n",
        "# impeding operation.\n",
        "try:\n",
        "  prediction = text_table_qa_pipeline.run(\n",
        "          query = question,\n",
        "          params = {\"EmbeddingRetriever\": {\"top_k\" : 10},\n",
        "                    \"TableReader\": {\"top_k\" : 2},\n",
        "                    \"TextReader\": {\"top_k\" : 2}}\n",
        "          )\n",
        "except:\n",
        "  prediction = [] # If we run into an error, return an empty list."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ka3FgfhH8jfl"
      },
      "source": [
        "##Filter by Score Threshold\n",
        "First, check if the prediction pipeline returned an answer (of type 'dict'). If it did, use a score threshold to filter documents so that the only answers returned are greater than the threshold. Also include a condition to return a default answer if no answers are returned that meet our threshold.\n",
        "\n",
        "If the prediction pipeline ran into an error and returned an empty list, use the default answer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e_m-3RkP6WYZ"
      },
      "outputs": [],
      "source": [
        "if isinstance(prediction, dict):\n",
        "  score_threshold = 0.1\n",
        "  filtered_documents = [doc for doc in prediction['answers'] if doc.score > score_threshold]\n",
        "\n",
        "  if not filtered_documents:\n",
        "        default_answer = {\"answer\": \"Sorry, I don't have an answer for that. Try asking your question in a different way\", \"score\": 0.0}\n",
        "        filtered_documents = [default_answer]\n",
        "\n",
        "else:\n",
        "  default_answer = {\"answer\": \"Sorry, I don't have an answer for that. Try asking your question in a different way.\", \"score\": 0.0}\n",
        "  filtered_documents = [default_answer]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QhW2hwiSGe3U"
      },
      "source": [
        "2. Print out the answers the pipeline returns:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tvOJ9jT4EQoH"
      },
      "outputs": [],
      "source": [
        "from haystack.schema import Answer\n",
        "\n",
        "# Set a hyperlink format for answers.\n",
        "hyperlink_format = '<a href=\"{link}\">{text}</a>'\n",
        "\n",
        "# Check if filtered_documents is a Haystack Answer object.\n",
        "# If so, print the answers. If not, print the default\n",
        "# answer.\n",
        "for answer in filtered_documents:\n",
        "    if isinstance(answer, Answer):\n",
        "        print('The suggested answer is:',\n",
        "              '\"',\n",
        "              answer.answer,\n",
        "              '\"',\n",
        "              'with {} percent probability.'.format(round((answer.score)*100)),\n",
        "              '\\n\\n',\n",
        "              'See here for more information related to this answer: ',\n",
        "              hyperlink_format.format(link = answer.meta['Link'], text = answer.meta['Title']),\n",
        "              '\\n\\n',\n",
        "              'Context for this answer: ',\n",
        "              answer.context,\n",
        "              '\\n\\n',\n",
        "              'Document ID: ',\n",
        "              answer.document_ids,\n",
        "              '\\n\\n')\n",
        "\n",
        "    else:\n",
        "        print(answer['answer'])\n",
        "\n",
        "#print(filtered_documents) # Print all filtered answers and related info"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "Owpyrz_ZGe3U"
      },
      "source": [
        "## Improvements/Extras\n",
        "\n",
        "Improve the performance of the Reader, by [fine-tuning](https://haystack.deepset.ai/tutorials/02_finetune_a_model_on_your_data)."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "z0ZXYIsZaq26",
        "M2Z9GaiLbNRF"
      ],
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.12"
    },
    "vscode": {
      "interpreter": {
        "hash": "85ea2c107d7945555de8e73270cf8a4d668bafec7aac344fa62e3415dc7bf5ec"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}