examples/52_Build_RAG_pipelines_with_txtai.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Build RAG pipelines with txtai\n",
        "\n",
        "Large Language Models (LLMs) have completely dominated the AI and machine learning space in 2023. The results have been amazing and the public imagination is almost endless.\n",
        "\n",
        "While LLMs have been impressive, they are not problem free. The biggest challenge is with hallucinations. Hallucinations is the term for when a LLM generates output that is factually incorrect. The alarming part of this is that on a cursory glance, it actually sounds like good content. The default behavior of LLMs is to produce plausible answers even when no plausible answer exists. LLMs are not great at saying I don't know.\n",
        "\n",
        "Retrieval augmented generation (RAG) helps reduce the risk of hallucinations by limiting the context in which a LLM can generate answers. This is typically done with a vector search query that hydrates a prompt with a relevant context. RAG is one of the most practical and production-ready use cases for *Generative AI*. It's so popular now, that some are creating their entire companies around it.\n",
        "\n",
        "[txtai](https://github.com/neuml/txtai) has long had question-answering pipelines, which employ the same process of retrieving a relevant context. LLMs are now the preferred approach for analyzing that context and RAG pipelines are one of the main features of txtai. One of the other main features of txtai is that it's a vector database! You can build your prompts and limit your context all with one library. Hence the phrase *all-in-one embeddings database*.\n",
        "\n",
        "This notebook shows how to build RAG pipelines with txtai."
      ],
      "metadata": {
        "id": "VGeVB8M41jqW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Install dependencies\n",
        "\n",
        "Install `txtai` and all dependencies. Since this notebook is using optional pipelines, we need to install the pipeline extras package."
      ],
      "metadata": {
        "id": "ZQrHIw351lwE"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "R0AqRP7v1hdr"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!pip install git+https://github.com/neuml/txtai#egg=txtai[pipeline] autoawq\n",
        "\n",
        "# Get test data\n",
        "!wget -N https://github.com/neuml/txtai/releases/download/v6.2.0/tests.tar.gz\n",
        "!tar -xvzf tests.tar.gz\n",
        "\n",
        "# Install NLTK\n",
        "import nltk\n",
        "nltk.download('punkt')"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Start with the basics\n",
        "\n",
        "Let's jump right in and start with a simple LLM pipeline. The [LLM pipeline](https://neuml.github.io/txtai/pipeline/text/llm/) supports loading local LLM models from the [Hugging Face Hub](https://huggingface.co/models).\n",
        "\n",
        "For those using LLM API services (OpenAI, Cohere, etc), this call can easily be replaced with an API call."
      ],
      "metadata": {
        "id": "xmPN8RDF1pXd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from txtai.pipeline import LLM\n",
        "\n",
        "# Create LLM\n",
        "llm = LLM(\"TheBloke/Mistral-7B-OpenOrca-AWQ\")"
      ],
      "metadata": {
        "id": "XZ7vPBIs1rGZ"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, we'll load a document to query. The [Textractor pipeline](https://neuml.github.io/txtai/pipeline/data/textractor/) has support for extracting text from common document formats (docx, pdf, xlsx)."
      ],
      "metadata": {
        "id": "9rmTWMxAH3Vx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from txtai.pipeline import Textractor\n",
        "\n",
        "# Create Textractor\n",
        "textractor = Textractor()\n",
        "text = textractor(\"txtai/document.docx\")\n",
        "print(text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nifStGtOHuyc",
        "outputId": "5a4010e0-75f9-4095-a24c-cd4c859847d0"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "txtai – the all-in-one embeddings database\n",
            "txtai is an all-in-one embeddings database for semantic search, LLM orchestration and language model workflows.\n",
            "\n",
            "Summary of txtai features:\n",
            "· Vector search with SQL, object storage, topic modeling\n",
            "· Create embeddings for text, documents, audio, images and video\n",
            "· Pipelines powered by language models that run LLM prompts\n",
            "· Workflows to join pipelines together and aggregate business logic\n",
            "· Build with Python or YAML. API bindings available for JavaScript, Java, Rust and Go.\n",
            "· Run local or scale out with container orchestration\n",
            "\n",
            "Examples\n",
            "List of example notebooks.\n",
            "|Notebook|Description|\n",
            "|---|---|\n",
            "|Introducing txtai |Overview of the functionality provided by txtai|\n",
            "|Similarity search with images|Embed images and text into the same space for search|\n",
            "|Build a QA database|Question matching with semantic search|\n",
            "|Semantic Graphs|Explore topics, data connectivity and run network analysis|\n",
            "\n",
            "Install\n",
            "The easiest way to install is via pip and PyPI\n",
            "pip install txtai\n",
            "Python 3.8+ is supported. Using a Python virtual environment is recommended.\n",
            "See the detailed install instructions for more information covering optional dependencies, environment specific prerequisites, installing from source, conda support and how to run with containers.\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Model guide\n",
            "The following shows a list of suggested models.\n",
            "|Component|Model(s)|\n",
            "|---|---|\n",
            "|Embeddings|all-MiniLM-L6-v2|\n",
            "||E5-base-v2|\n",
            "|Image Captions|BLIP|\n",
            "|Labels - Zero Shot|BART-Large-MNLI|\n",
            "|Labels - Fixed|Fine-tune with training pipeline|\n",
            "|Large Language Model (LLM)|Flan T5 XL|\n",
            "||Mistral 7B OpenOrca|\n",
            "|Summarization|DistilBART|\n",
            "|Text-to-Speech|ESPnet JETS|\n",
            "|Transcription|Whisper|\n",
            "|Translation|OPUS Model Series|\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we'll define a simple LLM pipeline. It takes a question and context (which in this case is the whole file), creates a prompt and runs it with the LLM."
      ],
      "metadata": {
        "id": "2jkamwgdIgEp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def execute(question, text):\n",
        "  prompt = f\"\"\"<|im_start|>system\n",
        "  You are a friendly assistant. You answer questions from users.<|im_end|>\n",
        "  <|im_start|>user\n",
        "  Answer the following question using only the context below. Only include information specifically discussed.\n",
        "\n",
        "  question: {question}\n",
        "  context: {text} <|im_end|>\n",
        "  <|im_start|>assistant\n",
        "  \"\"\"\n",
        "\n",
        "  return llm(prompt, maxlength=4096, pad_token_id=32000)\n",
        "\n",
        "execute(\"Tell me about txtai in one sentence\", text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 59
        },
        "id": "9HU6C0OIIAKn",
        "outputId": "f9d556c4-cd7a-4774-ef62-1f3fff90aa47"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Txtai is an all-in-one embeddings database for semantic search, LLM orchestration, and language model workflows, offering features such as vector search, pipeline creation, workflow management, and API bindings for various programming languages.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "execute(\"What model does txtai recommend for transcription?\", text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 39
        },
        "id": "xxF7ajCPJP5_",
        "outputId": "0e6f6dbb-c784-4841-fe3f-82754ef478eb"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'The model that txtai recommends for transcription is Whisper.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "execute(\"I don't know anything about txtai, what would be the best thing to read?\", text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 97
        },
        "id": "AKmmTqsnJa5X",
        "outputId": "834bf3ee-b7ed-4e38-e2ef-d5950f99ed9a"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'The best thing to read to learn about txtai is the \"Introducing txtai\" notebook, which provides an overview of the functionality provided by txtai. This notebook covers various features such as vector search with SQL, object storage, topic modeling, creating embeddings for text, documents, audio, images, and video, and running language model workflows. Additionally, you can explore other example notebooks like \"Similarity search with images,\" \"Build a QA database,\" and \"Semantic Graphs\" to learn more about specific use cases and features. To install txtai, use pip and PyPI with Python 3.8+, and follow the detailed install instructions for more information on optional dependencies and environment-specific prerequisites.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "If this is the first time you've seen *Generative AI*, then these statements are 🤯. Even if you've been in the space a while, it's still amazing how much a language model can understand and the high level of quality in it's answers.\n",
        "\n",
        "While this use case is fun, lets try to scale it to a larger set of documents."
      ],
      "metadata": {
        "id": "WaVeEHrIMpFr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Build a RAG pipeline with vector search\n",
        "\n",
        "Let's say we have a large number of documents, hundreds/thousands etc. We can't just put all those documents into a single prompt, we'll run out of GPU memory fast!\n",
        "\n",
        "This is where retrieval augmented generation enters the picture. We can use a query step that finds the best candidates to add to the prompt.\n",
        "\n",
        "Typically, this candidate query uses vector search but it can be anything that runs a search and returns results. In fact, many complex production systems have customized retrieval pipelines that feed a context into LLM prompts.\n",
        "\n",
        "The first step in building our RAG pipeline is creating the knowledge store. In this case, it's a vector database of file content. The files will be split into paragraphs with each paragraph stored as a separate row."
      ],
      "metadata": {
        "id": "viVVft59NbKv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "\n",
        "from txtai import Embeddings\n",
        "\n",
        "def stream(path):\n",
        "  for f in sorted(os.listdir(path)):\n",
        "    fpath = os.path.join(path, f)\n",
        "\n",
        "    # Only accept documents\n",
        "    if f.endswith((\"docx\", \"xlsx\", \"pdf\")):\n",
        "      print(f\"Indexing {fpath}\")\n",
        "      for paragraph in textractor(fpath):\n",
        "        yield paragraph\n",
        "\n",
        "# Document text extraction, split into paragraphs\n",
        "textractor = Textractor(paragraphs=True)\n",
        "\n",
        "# Vector Database\n",
        "embeddings = Embeddings(content=True)\n",
        "embeddings.index(stream(\"txtai\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ipmsmtN1NahT",
        "outputId": "64733e1f-fb7b-4a2d-bf02-8930478a8ee8"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Indexing txtai/article.pdf\n",
            "Indexing txtai/document.docx\n",
            "Indexing txtai/document.pdf\n",
            "Indexing txtai/spreadsheet.xlsx\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The next step is defining the RAG pipeline. This pipeline takes the input question, runs a vector search and builds a context using the search results. The context is then inserted into a prompt template and run with the LLM."
      ],
      "metadata": {
        "id": "ASlmAaR3nBPN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def context(question):\n",
        "  context =  \"\\n\".join(x[\"text\"] for x in embeddings.search(question))\n",
        "  return context\n",
        "\n",
        "def rag(question):\n",
        "  return execute(question, context(question))\n",
        "\n",
        "rag(\"What model does txtai recommend for image captioning?\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 39
        },
        "id": "_-9SW6r4P5ha",
        "outputId": "6a7bcd69-bcd0-4f6e-81c1-e32c323b3ffb"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Based on the provided context, txtai recommends the model \"BLIP\" for image captioning.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result = rag(\"When was the BLIP model added for image captioning?\")\n",
        "print(result)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NbQhSunPQtB0",
        "outputId": "de0caf04-4cdb-48e8-aadf-37283be9909a"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The BLIP model was added for image captioning on 2022-03-17.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As we can see, the result is similar to what we had before without vector search. The difference is that we only used a relevant portion of the documents to generate the answer.\n",
        "\n",
        "As we discussed before, this is important when dealing with large volumes of data. Not all of the data can be added to a LLM prompt. Additionally, having only the most relevant context helps the LLM generate higher quality answers."
      ],
      "metadata": {
        "id": "D6HW-3GtnTFl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Citations for LLMs\n",
        "\n",
        "A healthy level of skepticism should be applied to answers generated by AI. We're far from the day where we can blindly trust answers from an AI model.\n",
        "\n",
        "txtai has a couple approaches for generating citations. The basic approach is to take the answer and search the vector database for the closest match."
      ],
      "metadata": {
        "id": "FZ-yPC-xiUqa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for x in embeddings.search(result):\n",
        "  print(x[\"text\"])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "obttSg_dSFT5",
        "outputId": "c7ae7675-6959-4bcb-ad06-065ea8609c31"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "E5-base-v2\n",
            "Image Captions BLIP\n",
            "Labels - Zero Shot BART-Large-MNLI\n",
            "Model Guide\n",
            "|Component |Model(s)|Date Added|\n",
            "|---|---|---|\n",
            "|Embeddings |all-MiniLM-L6-v2|2022-04-15|\n",
            "|Image Captions |BLIP|2022-03-17|\n",
            "|Labels - Zero Shot |BART-Large-MNLI|2022-01-01|\n",
            "|Large Language Model (LLM) |Mistral 7B OpenOrca|2023-10-01|\n",
            "|Summarization |DistilBART|2021-02-22|\n",
            "|Text-to-Speech |ESPnet JETS|2022-08-01|\n",
            "|Transcription |Whisper|2022-08-01|\n",
            "|Translation |OPUS Model Series|2021-04-06|\n",
            "&\"Times New Roman,Regular\"&12&A\n",
            "Notebook Description\n",
            "Introducing txtai Overview of the functionality provided by txtai\n",
            "Similarity search with \n",
            "images Embed images and text into the same space for search\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "While the basic approach above works in this case, txtai has a more robust pipeline to handle citations and references.\n",
        "\n",
        "The Extractor pipeline is defined below. An Extractor pipeline works in the same way as a LLM + Vector Search pipeline, except it has special logic for generating citations. This pipeline takes the answers and compares it to the context passed to the LLM to determine the most likely reference."
      ],
      "metadata": {
        "id": "MDcB7GCWY6TO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from txtai.pipeline import Extractor\n",
        "\n",
        "# Extractor prompt\n",
        "def prompt(question):\n",
        "  return [{\n",
        "    \"query\": question,\n",
        "    \"question\": f\"\"\"\n",
        "Answer the following question using only the context below. Only include information specifically discussed.\n",
        "\n",
        "question: {question}\n",
        "context:\n",
        "\"\"\"\n",
        "}]\n",
        "\n",
        "# Create LLM with system prompt template\n",
        "llm = LLM(\"TheBloke/Mistral-7B-OpenOrca-AWQ\", template=\"\"\"<|im_start|>system\n",
        "You are a friendly assistant. You answer questions from users.<|im_end|>\n",
        "<|im_start|>user\n",
        "{text} <|im_end|>\n",
        "<|im_start|>assistant\n",
        "\"\"\")\n",
        "\n",
        "# Create extractor instance\n",
        "extractor = Extractor(embeddings, llm, output=\"reference\")"
      ],
      "metadata": {
        "id": "Lm6gg85_Y7ot"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "result = extractor(prompt(\"What version of Python is supported?\"), maxlength=4096, pad_token_id=32000)[0]\n",
        "print(\"ANSWER:\", result[\"answer\"])\n",
        "print(\"CITATION:\", embeddings.search(\"select id, text from txtai where id = :id\", limit=1, parameters={\"id\": result[\"reference\"]}))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4pOfE5paZatH",
        "outputId": "2bed2de5-22ff-4f7b-dba5-41b8e4cc6c75"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ANSWER: Python 3.8+ is supported. Using a Python virtual environment is recommended. The easiest way to install is via pip and PyPI.\n",
            "CITATION: [{'id': '24', 'text': 'Python 3.8+ is supported. Using a Python virtual environment is recommended.'}]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "And as we can see, not only is the answer to the statement shown, the extractor pipeline also provides a citation. This step is crucial in any line of work where answers must be verified (which is most lines of work)."
      ],
      "metadata": {
        "id": "vHdE2Q59jNnF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Wrapping up\n",
        "\n",
        "This notebook introduced retrieval augmented generation (RAG), explained why we need it and showed the options available for running RAG pipelines with txtai.\n",
        "\n",
        "The advantages of building RAG pipelines with txtai are:\n",
        "\n",
        "- **All-in-one database** - one library can handle LLM inference and vector search retrieval\n",
        "- **Generating citations** - generating answers is useful but referencing where those answers came from is crucial in gaining the trust of users\n",
        "- **Simple yet powerful** - building pipelines can be done in a small amount of Python. Options are available to build pipelines in YAML and/or run through the API"
      ],
      "metadata": {
        "id": "oPwgCgBc2Er2"
      }
    }
  ]
}