notebooks/using-vectara-with-llamaindex.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "adf7d63d",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/vectara/example-notebooks/blob/main/notebooks/using-vectara-with-llamaindex.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "397bea86",
   "metadata": {},
   "source": [
    "# Vectara and LlamaIndex"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db0855d0",
   "metadata": {},
   "source": [
    "## About Vectara\n",
    "\n",
    "[Vectara](https://vectara.com/) is the trusted GenAI and semantic search platform that provides an easy-to-use API for document indexing and querying. \n",
    "\n",
    "Vectara provides an end-to-end managed service for Retrieval Augmented Generation or [RAG](https://vectara.com/grounded-generation/), which includes:\n",
    "\n",
    "1. A way to extract text from document files and chunk them into sentences.\n",
    "\n",
    "2. The state-of-the-art [Boomerang](https://vectara.com/how-boomerang-takes-retrieval-augmented-generation-to-the-next-level-via-grounded-generation/) embeddings model. Each text chunk is encoded into a vector embedding using Boomerang, and stored in the Vectara internal knowledge (vector+text) store. Thus, when using Vectara with LlamaIndex you do not need to call a separate embedder model - this happens automatically within the Vectara backend.\n",
    "\n",
    "3. A query service that automatically encodes the query into embedding, and retrieves the most relevant text segments (including support for [Hybrid Search](https://docs.vectara.com/docs/api-reference/search-apis/lexical-matching) and [MMR](https://vectara.com/get-diverse-results-and-comprehensive-summaries-with-vectaras-mmr-reranker/))\n",
    "\n",
    "4. An option to create [generative summary](https://docs.vectara.com/docs/learn/grounded-generation/grounded-generation-overview), based on the retrieved documents, including citations.\n",
    "\n",
    "See the [Vectara API documentation](https://docs.vectara.com/docs/) for more information on how to use the API.\n",
    "\n",
    "The main benefits for using Vectara for a RAG application are:\n",
    "* **Easy to use**: Vectara provides an end-to-end, fully functional, highly scalable and robust RAG pipeline, so as a user you don't have to code up these pieces and maintain them over time.\n",
    "* **Scalable and Secure**: building GenAI applications may seem easy at first, but the DIY approach can become overwhelming beyond simple examples. Vectara provides instant scalablility to millions of documents, while maintaing data security and privacy, as well as latency SLAs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b33079b",
   "metadata": {},
   "source": [
    "## About LlamaIndex\n",
    "\n",
    "LlamaIndex is a \"data framework\" to help you build LLM apps:\n",
    "\n",
    "1. It includes **data connectors** to ingest your existing data sources and data formats (APIs, PDFs, docs, SQL, etc.)\n",
    "2. It provides ways to **structure your data** (indices, graphs) so that this data can be easily used with LLMs.\n",
    "3. It provides an **advanced retrieval/query interface over your data**: Feed in any LLM input prompt, get back retrieved context and knowledge-augmented output.\n",
    "\n",
    "LlamaIndex's high-level API allows beginner users to use LlamaIndex to ingest and query their data in just a few lines of code, whereas its lower-level APIs allow advanced users to customize and extend any module (data connectors, indices, retrievers, query engines, reranking modules), to fit their needs.\n",
    "\n",
    "Vectara is implemented in LlamaIndex as a [Managed Service](https://docs.llamaindex.ai/en/stable/community/integrations/managed_indices.html#vectara), abstracting all of Vectara's powerful API so they are easily integrated into LlamaIndex.\n",
    "\n",
    "In this notebook, we will demonstrate some of the great ways you can use Vectara together with LlamaIndex."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfe2497c",
   "metadata": {},
   "source": [
    "## Getting Started\n",
    "\n",
    "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙.\n",
    "\n",
    "To get started with Vectara, [sign up](https://vectara.com/integrations/llamaindex) (if you haven't already) and follow our [quickstart](https://docs.vectara.com/docs/quickstart) guide to create a corpus and an API key. \n",
    "\n",
    "Once you have these, you can provide them as environment variables, which will be used by the LlamaIndex code later on:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6019e01a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install -U llama-index llama-index-indices-managed-vectara arxiv\n",
    "\n",
    "import os\n",
    "os.environ['VECTARA_API_KEY'] = \"<YOUR_VECTARA_API_KEY>\"\n",
    "os.environ['VECTARA_CORPUS_ID'] = \"<YOUR_VECTARA_CORPUS_ID>\"\n",
    "os.environ['VECTARA_CUSTOMER_ID'] = \"<YOUR_VECTARA_CUSTOMER_ID>\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7010b1d-d1bb-4f08-9309-a328bb4ea396",
   "metadata": {},
   "source": [
    "## Loading Data Into Vectara\n",
    "\n",
    "As mentioned above, Vectara is a RAG managed service, and in many cases data may be uploaded to the index ahead of time (e.g. by using [Airbyte](https://docs.airbyte.com/integrations/destinations/vectara), directly via Vectara's [indexing API](https://docs.vectara.com/docs/api-reference/indexing-apis/indexing) or using tools like [vectara-ingest](https://github.com/vectara/vectara-ingest)), but another easy way is via the VectaraIndex constructor: `from_documents()`.\n",
    "\n",
    "For this notebook we will assume the Vectara corpus is empty and will load PDF documents from Arxiv, using Python's [arxiv](https://github.com/lukasschwab/arxiv.py) library. We would pull in data from the top papers related to \"climate change\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "40947545",
   "metadata": {},
   "outputs": [],
   "source": [
    "import arxiv\n",
    "\n",
    "client = arxiv.Client()\n",
    "search = arxiv.Search(\n",
    "  query = \"(ti:embedding model) OR (ti:sentence embedding)\",\n",
    "  max_results = 100,\n",
    "  sort_by = arxiv.SortCriterion.Relevance\n",
    ")\n",
    "papers = list(client.results(search))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0ae7bd09-6569-48cc-8c54-8bf491c0e656",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['http://arxiv.org/abs/2402.14776v2',\n",
       " 'http://arxiv.org/abs/2007.01852v2',\n",
       " 'http://arxiv.org/abs/1910.13291v1',\n",
       " 'http://arxiv.org/abs/2104.06719v1',\n",
       " 'http://arxiv.org/abs/1511.08198v3']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[p.entry_id for p in papers][:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c163ade",
   "metadata": {},
   "source": [
    "Next, download the Arxiv paper, and upload them into Vectara using the `add_file()`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c154dd4b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File temp/1909.03104v2.Efficient_Sentence_Embedding_using_Discrete_Cosine_Transform.pdf failed to load with error HTTP Error 404: Not Found\n"
     ]
    }
   ],
   "source": [
    "import shutil\n",
    "from llama_index.indices.managed.vectara import VectaraIndex\n",
    "\n",
    "data_folder = 'temp'\n",
    "os.makedirs(data_folder, exist_ok=True)\n",
    "\n",
    "# Create Vectara Index\n",
    "index = VectaraIndex()\n",
    "\n",
    "# Upload content ofr all papers\n",
    "for paper in papers:\n",
    "    try:\n",
    "        paper_fname = paper.download_pdf(data_folder)\n",
    "    except Exception as e:\n",
    "        print(f\"File {paper_fname} failed to load with error {e}\")\n",
    "        continue\n",
    "    metadata = {\n",
    "        'url': paper.pdf_url,\n",
    "        'title': paper.title,\n",
    "        'author': str(paper.authors[0]),\n",
    "        'published': str(paper.published.date())\n",
    "    }\n",
    "    index.insert_file(file_path=paper_fname, metadata=metadata)\n",
    "\n",
    "shutil.rmtree(data_folder)\n",
    "del papers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d94ea571",
   "metadata": {},
   "source": [
    "Two important things to note here:\n",
    "1. Vectara processes each file uploaded on the backend, and performs appropriate chunking. So you don't need to apply any local processing, or choose a chunking strategy. \n",
    "2. We have used the fields `url`, `title`, `author`, and `published` as metadata fields (where author is the first author if there are many, just to simplify). You will need to make sure those fields are defined in your Vectara corpus as [filterable metadata fields](https://docs.vectara.com/docs/learn/metadata-search-filtering/filter-overview) to ensure we can filter by them in query time.\n",
    "\n",
    "So that's it for upload. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ee4473a-094f-4d0a-a825-e1213db07240",
   "metadata": {},
   "source": [
    "## Querying with the VectaraIndex\n",
    "We can now ask questions using the `VectaraIndex` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "bb174ec3",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"What is sentence embedding?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "21facbac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence embedding is a method used in natural language processing that represents sentences in a numerical format. It involves the use of models like SimCSE-BERT/SRoBERTa/ST5 to convert sentences into a form that can be processed by machine learning algorithms. This process often involves the use of LSTM (Long Short-Term Memory) encoders and decoders. The LSTM encoder architecture with max-pooling is commonly used for all encoders, and LSTM decoders are used for SDAE and NMT. The resulting sentence embeddings can then be used in various applications, such as in a softmax inference classifier [1][2][3].\n"
     ]
    }
   ],
   "source": [
    "query_engine = index.as_query_engine(\n",
    "    summary_enabled=True, summary_num_results=5,\n",
    "    summary_response_lang=\"en\",\n",
    "    summary_prompt_name=\"vectara-summary-ext-24-05-med\"\n",
    ")\n",
    "res = query_engine.query(query)\n",
    "print(res.response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52878fd2",
   "metadata": {},
   "source": [
    "Note that the response here is fully generated by Vectara. There is no additional LLM involved (or API key you need to setup). The response also includes citations (marked in square brackets), which provide links to references used to generate this response by Vectara. \n",
    "<br>\n",
    "The `res` object includes the actual response to the user query, but also has the citations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3b110a6a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 'http://arxiv.org/pdf/2305.03010v1'),\n",
       " (1, 'http://arxiv.org/pdf/1904.05542v1'),\n",
       " (2, 'http://arxiv.org/pdf/1904.05542v1'),\n",
       " (3, 'http://arxiv.org/pdf/1904.05542v1'),\n",
       " (4, 'http://arxiv.org/pdf/1904.05542v1'),\n",
       " (5, 'http://arxiv.org/pdf/1904.05542v1'),\n",
       " (6, 'http://arxiv.org/pdf/1904.05542v1'),\n",
       " (7, 'http://arxiv.org/pdf/1904.05542v1'),\n",
       " (8, 'http://arxiv.org/pdf/2206.02690v3'),\n",
       " (9, 'http://arxiv.org/pdf/2404.03921v2')]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(inx, n.node.metadata['url']) for inx,n in enumerate(res.source_nodes)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74cce7c7-c664-4001-8c84-f80c5d499796",
   "metadata": {},
   "source": [
    "## Using Streaming\n",
    "\n",
    "You can also stream the Vectara response simply by specifying `streaming=True`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "abd30027-d975-4440-a1c2-66d1bd30bc29",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence embedding refers to the process of converting a sentence into a fixed-dimensional vector representation. This is achieved through various models like SimCSE-BERT, SRoBERTa, and ST5 using techniques such as LSTM encoders and decoders. The embeddings capture the semantic meaning of the sentence, allowing for tasks like entailment, contradiction, and neutral classification. Different architectures like bidirectional LSTM and softmax inference classifiers are utilized in sentence embedding models to encode sentences efficiently."
     ]
    }
   ],
   "source": [
    "query_engine = index.as_query_engine(summary_enabled=True, streaming=True)\n",
    "res = query_engine.query(query)\n",
    "\n",
    "# print streamed output chunk by chunk\n",
    "for chunk in res.response_gen:\n",
    "    print(chunk.delta or \"\", end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b2cd598-ce73-4c9e-9aa6-5d8604fd6c14",
   "metadata": {},
   "source": [
    "## Reranking\n",
    "\n",
    "Vectara supports two types of [reranking](https://docs.vectara.com/docs/api-reference/search-apis/reranking). The first one is called \n",
    "[max-marginal-relevance](https://docs.vectara.com/docs/api-reference/search-apis/reranking#maximal-marginal-relevance-mmr-reranker) or MMR and provides a reranking that can promote diversity in results at the cost of relevance. The other reranker, called Slingshot, is an ML reranker that increases accuracy of results ranking and is available to Vectara Scale customers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e49914a",
   "metadata": {},
   "source": [
    " Let's see an example of how to use MMR: We will run the same query but this time we will use MMR where `mmr_diversity_bias=0.3` provides a tradeoff between relevance and diversity (0.0 is full relevance, 1.0 is only diversity):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "72832e45",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence embedding is a technique in Natural Language Processing (NLP) that involves representing sentences as vectors in n-dimensional space to encode their meanings. Various models like ELMo, BERT, and SBERT-WK are used to compute sentence embeddings by averaging LSTM outputs or word representations. These embeddings are crucial for tasks like document classification and sentiment analysis. The goal is to preserve the original sentence meanings effectively in the embedded vectors. Sentence embedding plays a vital role in NLP applications such as search engines and question-and-answer platforms, contributing to significant breakthroughs in the field [2] [3] [4] [7].\n"
     ]
    }
   ],
   "source": [
    "query_engine = index.as_query_engine(\n",
    "    similarity_top_k=5,\n",
    "    reranker=\"mmr\",\n",
    "    rerank_k=50,\n",
    "    mmr_diversity_bias=0.3,\n",
    ")\n",
    "response = query_engine.query(query)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "17020d34-5176-4910-bb88-5c5685513804",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 'http://arxiv.org/pdf/2305.03010v1'),\n",
       " (1, 'http://arxiv.org/pdf/1808.05505v3'),\n",
       " (2, 'http://arxiv.org/pdf/2206.02690v3'),\n",
       " (3, 'http://arxiv.org/pdf/2002.09620v2'),\n",
       " (4, 'http://arxiv.org/pdf/2210.06432v3')]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(inx, n.node.metadata['url']) for inx,n in enumerate(response.source_nodes)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53e76fd1",
   "metadata": {},
   "source": [
    "As you can see, the results are now reranked in a way that provides more diversity instead of maximizing pure relevance. This in turn results in a different set of chunks used to generate the response."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e863b914",
   "metadata": {},
   "source": [
    "So far we've used Vectara's internal summarization capability, which is the best way for most users.\n",
    "\n",
    "You can still use Llama-Index's standard VectorStore `as_query_engine()` method, in which case Vectara's summarization won't be used, and you would be using an external LLM (like OpenAI's GPT-4 or similar) and a custom prompt from LlamaIndex to generate the summary. For this option just set `summary_enabled=False`\n",
    "\n",
    "For this you would need to specify your own OpenAI API key in the environment:\n",
    "\n",
    "> `os.environ['OPENAI_API_KEY'] = '<YOUR_OPENAI_API_KEY>`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7b0a49d1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence embedding is a term used for representing words or sentences in a text that encodes the meaning of the word or the sentence in n-dimensional space. It involves mapping text data into vectors that can be a set of real numbers, allowing the encoded text to be processed and understood by machines. This representation is crucial in various applications such as search engines, expert systems, and question-and-answer platforms, where the proximity of vectors in the space can indicate the similarity of meanings between words or sentences.\n"
     ]
    }
   ],
   "source": [
    "from llama_index.llms.openai import OpenAI\n",
    "llm = OpenAI(model=\"gpt-4-turbo\", temperature=0)\n",
    "\n",
    "query_engine = index.as_query_engine(\n",
    "    similarity_top_k=5,\n",
    "    summary_enabled=False,\n",
    "    llm=llm\n",
    ")\n",
    "response = query_engine.query(query)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ebad3cf-b09e-4af5-be3d-ba2a3e3e8d73",
   "metadata": {},
   "source": [
    "## Using Vectara Chat\n",
    "\n",
    "Vectara now fully supports Chat in its platform, where the chat history is maintained by Vectara and so you don't have to worry about keeping history and integrating it with your RAG pipeline. \n",
    "\n",
    "To use it simple call `as_chat_engine()`.\n",
    "\n",
    "(Chat mode always uses Vectara's summarization so you don't have to explicitly specify `summary_enabled=True` like before)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "42af08f5-dc13-4991-96f4-fdd294b0ce97",
   "metadata": {},
   "outputs": [],
   "source": [
    "ce = index.as_chat_engine()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "7e3368be-46a3-4b2f-9589-de3383c65201",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: What is a sentence embedding model?\n",
      "\n",
      "Response: A sentence embedding model is a method that represents input sentences as fixed-dimensional vectors, regardless of sentence length. These models have shown significant enhancements in various NLP tasks like information retrieval, question answering, and machine translation. They are trained to adapt to specific domains by fine-tuning on synthesized sentence pairs before further fine-tuning on labeled data, leading to improved performance. Additionally, sentence embedding models can be tailored for cross-lingual applications by aligning at both sentence and token levels, enhancing their versatility and effectiveness in capturing semantic relationships within and across languages [4][3][5].\n",
      "\n",
      "Question: What are some known models?\n",
      "\n",
      "Response: Some examples of well-known sentence embedding models include BERT (both BERT-Large and BERT-Base), ELMO + Attn, GenSen, DSE (with different alpha values), FastSent, Quick-Thought, and models like InferSent (InferSentV1 and InferSentV2), Universal Sentence Encoder (USE-DAN and USE-Transformer), RoBERTa, XLNet, and Sentence-BERT. These models utilize various techniques such as using the average of all output layers or the CLS token from BERT for sentence embedding. Additionally, some models incorporate domain-specific knowledge through adapters to enhance semantic textual similarity tasks [1], [2], [3], [6].\n",
      "\n",
      "Question: How are they different than token embedding models\n",
      "\n",
      "Response: Sentence embedding models differ from token embedding models in the way they generate representations. While token embedding models focus on individual words or tokens, sentence embedding models aim to capture the overall meaning of a sentence. Sentence embedding models typically pool word or token embeddings, often by averaging them, and then fine-tune this process to create representative embeddings for various tasks. On the other hand, token embedding models disperse token representations across the embedding space. Additionally, sentence embedding models can be trained for both sentence-level alignment and token-level alignment, providing a more comprehensive understanding of the text. Overall, the key distinction lies in the level of abstraction and context that each type of model captures: token embeddings focus on individual elements, while sentence embeddings aim to encapsulate the meaning of entire sentences [1][2][6].\n",
      "\n"
     ]
    }
   ],
   "source": [
    "questions = [\n",
    "    'What is a sentence embedding model?',\n",
    "    'What are some known models?',\n",
    "    'How are they different than token embedding models'\n",
    "]\n",
    "\n",
    "for q in questions:\n",
    "    print(f\"Question: {q}\\n\")\n",
    "    response = ce.chat(q).response\n",
    "    print(f\"Response: {response}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e111b1fa-5efc-4ee6-99f0-21c73dbd5433",
   "metadata": {},
   "source": [
    "Of course streaming works as well with Chat:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c8b24622-bc73-4bee-929e-377f06335a80",
   "metadata": {},
   "outputs": [],
   "source": [
    "ce = index.as_chat_engine(streaming=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c8f16016-be7c-411f-aafd-b03d8816d21d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The entity behind SBERT is a team that extended Sentence-BERT (SBERT) based on transformer models like BERT. SBERT utilizes a siamese network structure and pre-trained weights from BERT for efficient training of suitable sentence embeddings methods. The team achieved state-of-the-art performance for various sentence embedding tasks using SBERT, which applies mean pooling on the output. Additionally, they used XLM-R as a pre-trained network on 100 languages in their experiments. The team's work demonstrates improvements in supervised sentence embedding methods, showcasing the effectiveness of SBERT in enhancing performance. However, the specific names or affiliations of the individuals behind SBERT are not explicitly mentioned in the search results provided."
     ]
    }
   ],
   "source": [
    "response = ce.stream_chat(\"who is behind SBERT?\")\n",
    "for chunk in response.chat_stream:\n",
    "    print(chunk.delta or \"\", end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a01ad93e-0692-4ad5-99a1-329788252b03",
   "metadata": {},
   "source": [
    "# Advanced RAG with Vectara and LLamaIndex\n",
    "\n",
    "## Agentic RAG\n",
    "\n",
    "LlamaIndex provides various agent implementations such as ChainOfThough or React.\n",
    "\n",
    "To use these with Vectara, you would need to use an external LLM as the driver of the agent resoning, and in this example we will be using OpenAI's GPT4o (for this to work, please make sure you have `OPENAI_API_KEY` defined in your environment)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "92019803-7aaa-4199-a322-692d03b374c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.agent import ReActAgent\n",
    "from llama_index.core.tools import QueryEngineTool, ToolMetadata\n",
    "\n",
    "vectara_tool = QueryEngineTool(\n",
    "    query_engine=index.as_query_engine(\n",
    "        summary_enabled=True,\n",
    "        summary_num_results=5,\n",
    "        summary_response_lang=\"en\",\n",
    "        summary_prompt_name=\"vectara-summary-ext-24-05-large\",\n",
    "    ),\n",
    "    metadata=ToolMetadata(\n",
    "        name=\"Vectara\",\n",
    "        description=\"Vectara tool that can answer Questions about Embedding Models, NLP, and related topics.\",\n",
    "    ),\n",
    ")\n",
    "agent = ReActAgent.from_tools(\n",
    "    tools=[vectara_tool],\n",
    "    llm=llm,\n",
    "    context=\"\"\"\n",
    "        You are a helpful chatbot that answers any user questions around embedding models in NLP using the Vectara tool.\n",
    "        You break down complex questions into simpler ones and use the vectara tool to answer every question or sub-question.\n",
    "        You use the Vectara tool to help answer the user question.\n",
    "    \"\"\",\n",
    "    verbose=True,\n",
    "    max_iterations=20\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c0ae3ea3-76f9-4898-a4bf-1788abb5d53c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;5;200mThought: The current language of the user is: English. I need to use a tool to help me answer the question.\n",
      "Action: Vectara\n",
      "Action Input: {'input': 'What are sentence embedding models?'}\n",
      "\u001b[0m\u001b[1;3;34mObservation: Sentence embedding models are designed to convert sentences into numerical representations that capture their semantic meanings. These models are typically trained to achieve state-of-the-art representations for tasks like semantic textual similarity. They can be adapted for specific domains by incorporating domain-specific knowledge [2]. Examples of sentence embedding models include Sentence-BERT, SimCSE-BERT/SimCSE-RoBERTa, Sentence-T5, and MP-Net [3]. These models are useful in various applications including information retrieval [4].\n",
      "\u001b[0m\u001b[1;3;38;5;200mThought: The user asked for details about the best sentence embedding models, their creators, and publication years. I need to gather this information next.\n",
      "Action: Vectara\n",
      "Action Input: {'input': 'What are the best sentence embedding models and who created each model and in what year was the paper published?'}\n",
      "\u001b[0m\u001b[1;3;34mObservation: The search results provide information on several notable sentence embedding models, their creators, and publication years. GenSen, created by Subramanian et al., was published in 2018 and is highlighted for its performance [1], [5]. InferSent, another model mentioned, was developed by Conneau et al. and discussed in their 2017 publication [4]. These models are noted for their effectiveness in various natural language processing tasks. However, the search results do not provide comprehensive details on all top sentence embedding models, such as BERT or ELMO, nor do they cover the complete landscape of sentence embedding technologies.\n",
      "\u001b[0m\u001b[1;3;38;5;200mThought: The user also requested a comparison of the architecture, training procedures, and performance of these models. I need to gather this comparative information next.\n",
      "Action: Vectara\n",
      "Action Input: {'input': 'Compare and contrast the architecture, training procedure and performance of sentence embedding models like GenSen, InferSent, BERT, and ELMO.'}\n",
      "\u001b[0m\u001b[1;3;34mObservation: The search results provide a comparative overview of the architecture, training procedures, and performance of sentence embedding models like GenSen, InferSent, BERT, and ELMO.\n",
      "\n",
      "**Architecture**: InferSent primarily uses LSTM architectures, while BERT and its variants rely on the Transformer architecture. ELMO uses a language model that captures context and meaning [1], [4].\n",
      "\n",
      "**Training Procedure**: BERT is trained using a masked language model approach. GenSen and InferSent training details are less explicit in the provided results, but GenSen is noted for its performance in multi-task settings [3]. ELMO's training involves language models to enhance context understanding [1].\n",
      "\n",
      "**Performance**: The performance of these models varies depending on the tasks. BERT does not perform exceptionally well on Semantic Textual Similarity (STS) tasks compared to other models like GenSen, which shows better performance in sentence embedding tasks [2], [3]. ELMO's performance benefits from its context-aware training but struggles when applied to a broad set of downstream tasks [1].\n",
      "\n",
      "Overall, each model has strengths and weaknesses depending on the specific application and the nature of the task at hand.\n",
      "\u001b[0m\u001b[1;3;38;5;200mThought: I can answer without using any more tools. I'll use the user's language to answer.\n",
      "Answer: Sentence embedding models are designed to convert sentences into numerical representations that capture their semantic meanings. Notable models include GenSen, created by Subramanian et al. in 2018, and InferSent by Conneau et al. from 2017. These models are crucial in various NLP tasks.\n",
      "\n",
      "In terms of architecture, InferSent uses LSTM architectures, BERT and its variants employ the Transformer architecture, and ELMO utilizes a language model to capture context. Regarding training, BERT is trained with a masked language model approach, while ELMO's training involves language models to enhance context understanding. The training details for GenSen and InferSent are less explicit but note GenSen's performance in multi-task settings.\n",
      "\n",
      "Performance-wise, BERT, while robust, does not excel in Semantic Textual Similarity (STS) tasks as much as GenSen, which shows better performance in sentence embedding tasks. ELMO benefits from its context-aware training but may struggle in a broader range of downstream tasks.\n",
      "\n",
      "Each model has its strengths and weaknesses, making them suitable for different applications and tasks.\n",
      "\u001b[0mSentence embedding models are designed to convert sentences into numerical representations that capture their semantic meanings. Notable models include GenSen, created by Subramanian et al. in 2018, and InferSent by Conneau et al. from 2017. These models are crucial in various NLP tasks.\n",
      "\n",
      "In terms of architecture, InferSent uses LSTM architectures, BERT and its variants employ the Transformer architecture, and ELMO utilizes a language model to capture context. Regarding training, BERT is trained with a masked language model approach, while ELMO's training involves language models to enhance context understanding. The training details for GenSen and InferSent are less explicit but note GenSen's performance in multi-task settings.\n",
      "\n",
      "Performance-wise, BERT, while robust, does not excel in Semantic Textual Similarity (STS) tasks as much as GenSen, which shows better performance in sentence embedding tasks. ELMO benefits from its context-aware training but may struggle in a broader range of downstream tasks.\n",
      "\n",
      "Each model has its strengths and weaknesses, making them suitable for different applications and tasks.\n"
     ]
    }
   ],
   "source": [
    "question = \"\"\"\n",
    "    What are the sentence embedding models? \n",
    "    what are the best sentence embedding models, who created each model and in what year was the paper published?\n",
    "    Compare and contrast their architecture, training procedure and performance\n",
    "\"\"\"\n",
    "\n",
    "print(agent.chat(question).response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f43963c4",
   "metadata": {},
   "source": [
    "## Using Auto Retriever with Vectara\n",
    "\n",
    "LlamaIndex's auto-retriever functionality is really cool. \n",
    "It is most useful when you have metadata fields (like in our case of papers from Arxiv), and would like a query that references a metadata field to be automatically interpreted in the right way.\n",
    "\n",
    "For example, if I ask \"what is a paper about climate change risks published after 2020\", the auto-retriever would (behind the scences) interpret ths into a query \"what is a paper about climate change risks\" along with a filter condition of \"published > 2020\"\n",
    "\n",
    "Let's see how this works with the Vectara Index.\n",
    "First, we have to define a `VectorStoreInfo` structure that defines the meta data fields the auto-retriever knows about to do its job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "24cc7017",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.vector_stores.types import MetadataInfo, VectorStoreInfo\n",
    "\n",
    "vector_store_info = VectorStoreInfo(\n",
    "    content_info=\"information about a paper\",\n",
    "    metadata_info=[\n",
    "        MetadataInfo(\n",
    "            name=\"published\",\n",
    "            description=\"The date the paper was published\",\n",
    "            type=\"string\",\n",
    "        ),\n",
    "        MetadataInfo(\n",
    "            name=\"author\",\n",
    "            description=\"The author of the paper\",\n",
    "            type=\"string\",\n",
    "        ),\n",
    "        MetadataInfo(\n",
    "            name=\"title\",\n",
    "            description=\"The title of the papers\",\n",
    "            type=\"string\",\n",
    "        ),\n",
    "        MetadataInfo(\n",
    "            name=\"url\",\n",
    "            description=\"The URL for this paper\",\n",
    "            type=\"string\",\n",
    "        ),\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23712d46",
   "metadata": {},
   "source": [
    "Auto-retrieval is implemented before calling Vectara as a query transformation. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "585f0333",
   "metadata": {},
   "source": [
    "Now we can define the `VectaraAutoRetriever`, which can perform auto-retrieval using Vectara:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "92de30c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using query str: What is sentence embedding?\n",
      "Using implicit filters: [('published', '<', '2019')]\n",
      "final filter string: (doc.published < '2019')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('2018-08-16',\n",
       "  'This\\nproblem can be alleviated by obtaining more of para-\\nphrase sentence pairs. Conclusion Sentence embedding is one of the most important text\\nprocessing techniques in NLP. To date,  various sen-\\ntence embedding models have been proposed and have\\nyielded good performances in document classification\\nand sentiment analysis tasks. However, the fundamen-\\ntal ability of sentence embedding methods, i.e., how\\neffectively the meanings of the original sentences are\\npreserved  in  the  embedded  vectors,  cannot  be  fully\\nevaluated through such indirect methods.'),\n",
       " ('2018-08-16',\n",
       "  'Paraphrase Thought:  Sentence Embedding Module Imitating\\n                        Human Language Recognition Myeongjun Jang 1 Abstract\\nSentence embedding is an important research\\ntopic in natural language processing. It is es-\\nsential to generate a good embedding vector\\nthat  fully  reflects  the  semantic  meaning  of\\na sentence in order to achieve an enhanced\\nperformance  for  various  natural  language\\nprocessing  tasks,   such  as  machine  trans-\\nlation  and  document  classification. Thus\\nfar, various sentence embedding models have\\nbeen proposed, and their feasibility has been\\ndemonstrated through good performances on\\ntasks following embedding, such as sentiment\\nanalysis  and  sentence  classification.'),\n",
       " ('2018-08-16',\n",
       "  'Pilsung Kang 1 Introduction Sentence embedding, which transforms sentences into\\nlow-dimensional  vector  values  reflecting  their  mean-\\nings,  is a highly important task in natural language\\nprocessing  (NLP). By  mapping  unstructured  text\\ndata into a certain form of structured representation,\\nthe embedding vector can enhance the performances\\nof  various  NLP  tasks,  such  as  machine  translation\\n(Artetxe et al., 2017; Lee et al., 2016; Zhao & Zhang,\\n2016), document classification (Conneau et al., 2017b;\\nZhou et al., 2016), and sentence matching (Wan et al.,\\n2016). As sentence embedding plays an import role\\nin NLP, various methods (Kiros et al., 2015; Pagliar-\\ndini et al., 2017; Hill et al., 2016; Arora et al., 2017;\\nConneau  et  al.,  2017a;  Chen,  2017)  have  been  pro-\\nposed since the advent of the Doc2vec method (Le &\\nMikolov, 2014).'),\n",
       " ('2016-06-15',\n",
       "  '2    Siamese CBOW\\n\\nWe present the Siamese Continuous Bag of Words\\n(CBOW)  model,  a  neural  network  for  efﬁcient\\nestimation of high-quality sentence embeddings. Quality should manifest itself in embeddings of\\nsemantically close sentences being similar to one\\nanother, and embeddings of semantically different\\nsentences being dissimilar. An efﬁcient and sur-\\nprisingly successful way of computing a sentence\\nembedding is to average the embeddings of its\\nconstituent words. Recent work uses pre-trained\\nword embeddings (such as word2vec and GloVe)\\nfor this task, which are not optimized for sentence\\nrepresentations. Following these approaches, we\\ncompute sentence embeddings by averaging word\\nembeddings, but we optimize word embeddings\\ndirectly for the purpose of being averaged.'),\n",
       " ('2017-03-09',\n",
       "  'Instead of using a vector, we use a 2-D matrix\\nto represent the embedding, with each row of the matrix attending on a different\\npart of the sentence. We also propose a self-attention mechanism and a special\\nregularization term for the model. As a side effect, the embedding comes with an\\neasy way of visualizing what speciﬁc parts of the sentence are encoded into the\\nembedding. We evaluate our model on 3 different tasks: author proﬁling, senti-\\nment classiﬁcation and textual entailment. Results show that our model yields a\\nsigniﬁcant performance gain compared to other sentence embedding methods in\\nall of the 3 tasks.'),\n",
       " ('2018-06-03',\n",
       "  'We use a one-hot\\nvector representation for every word and obtain a word embedding ci for each word using a Temporal CNN (Zhang et al., 2015; Palangi et al., 2016) module that we parameterize through a function G(Xi; We)\\nwhere We are the weights of the temporal CNN. Now this word embedding is fed to an LSTM-based\\nencoder which provides encoding features of the sentence. We use LSTM (Hochreiter and Schmidhuber,\\n1997) due to its capability of capturing long term memory (Palangi et al., 2016). As the words are\\npropagated through the network, the network collects more and more semantic information about the\\nsentence.'),\n",
       " ('2017-03-07',\n",
       "  'Con-\\nceptually, the model can be interpreted as a natu-\\nral extension of the word-contexts from C-BOW\\n(Mikolov et al., 2013b,a) to a larger sentence con-\\ntext, with the sentence words being speciﬁcally\\noptimized towards additive combination over the\\nsentence, by means of the unsupervised objective\\nfunction. Formally, we learn a source (or context) embed-\\nding vw and target embedding uw for each word w\\nin the vocabulary, with embedding dimension h\\nand k  = jVj as in (1). The sentence embedding\\nis deﬁned as the average of the source word em-\\nbeddings of its constituent words, as in (2). We\\naugment this model furthermore by also learning\\nsource embeddings for not only unigrams but also\\nn-grams present in each sentence, and averaging\\nthe n-gram embeddings along with the words, i.e.,\\nthe sentence embedding vS for S is modeled as X'),\n",
       " ('2018-02-12',\n",
       "  'Background Background Sentence Embeddings Introduction\\nA hallmark of human intelligence is compositionality:  the\\nability,  in  the words  of  von  Humboldt,  to “make  inﬁnite\\nuse of ﬁnite means.” The failure of neural network models\\nto achieve compositionality has been a recurring (and con-\\ntroversial) theme in cognitive science (Fodor & Pylyshyn,\\n1988; Gershman & Tenenbaum, 2015; Lake & Baroni, 2017).'),\n",
       " ('2017-03-09',\n",
       "  '(See Figure 3a and 3b). The second way of visualization can be achieved by summing up over all the annotation vectors,\\nand then normalizing the resulting weight vector to sum up to 1. Since it sums up all aspects of\\nsemantics of a sentence, it yields a general view of what the embedding mostly focuses on. We can\\nﬁgure out which words the embedding takes into account a lot, and which ones are skipped by the\\nembedding. See Figure 3c and 3d.'),\n",
       " ('2018-09-17',\n",
       "  'Word  Embeddings    We  rely  on  GloVe  (Pen-\\nnington et al., 2014) embeddings of size 300 to\\ncreate a dense, low-dimension vector representa-\\ntion of a sentence.6  We average all word vectors\\nof a sentence, representing it by kind of a centroid\\nword—a simple method shown to be effective for\\nseveral tasks (Wieting et al., 2016). Sentence Embeddings    Bags of words and aver-\\nage word embeddings lose sequence information,\\nwhich intuitively should help for (directed) com-\\nparison extraction. Sentence embeddings aim to\\nlearn representations for spans of text instead of\\nsingle words by taking sequence information into\\naccount. Several methods like FastSent (Hill et al.,\\n2016) or SkipTought (Kiros et al., 2015) have been\\nproposed to create sentence embeddings. We use\\nInferSent (Conneau et al., 2017) that learns sen-\\ntence embeddings similar to word embeddings.')]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from llama_index.indices.managed.vectara import VectaraAutoRetriever\n",
    "\n",
    "retriever = VectaraAutoRetriever(\n",
    "    index,\n",
    "    vector_store_info=vector_store_info,\n",
    "    llm=llm,\n",
    "    verbose=True\n",
    ")\n",
    "res = retriever.retrieve(\"What is sentence embedding, based on papers before 2019?\")\n",
    "[(r.metadata['published'], r.text) for r in res]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b7450bc",
   "metadata": {},
   "source": [
    "As you can see, the Auto Retriever was able to translate the natural language text into a shorter query and a proper condition (in this case `doc.published < 2019`).\n",
    "\n",
    "We can also of course ask a question directly: we use the `VectaraQueryEngine` which can work with the `VectaraAutoRetriever` directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "327ffbce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using query str: What is sentence embedding?\n",
      "Using implicit filters: [('published', '<', '2019')]\n",
      "final filter string: (doc.published < '2019')\n",
      "Sentence embedding is a crucial technique in Natural Language Processing (NLP) that involves transforming sentences into low-dimensional vector representations that capture their semantic meanings. Various models have been developed to create these embeddings, aiming to preserve the original sentence meanings effectively. These embeddings play a vital role in enhancing the performance of NLP tasks like machine translation, document classification, sentiment analysis, and more. Different methods have been proposed to optimize sentence embeddings, such as averaging word embeddings or using self-attention mechanisms. The goal is to ensure that semantically similar sentences have similar embeddings, while dissimilar sentences have distinct embeddings. Overall, sentence embedding is fundamental for improving the efficiency and accuracy of various NLP applications. [1], [2], [3], [4], [5].\n"
     ]
    }
   ],
   "source": [
    "from llama_index.indices.managed.vectara.query import VectaraQueryEngine\n",
    "from llama_index.indices.managed.vectara import VectaraAutoRetriever\n",
    "\n",
    "ar = VectaraAutoRetriever(\n",
    "    index,\n",
    "    vector_store_info=vector_store_info,\n",
    "    llm=llm,\n",
    "    summary_enabled=True,\n",
    "    summary_num_results=5,\n",
    "    verbose=True\n",
    ")\n",
    "\n",
    "query_engine = VectaraQueryEngine(retriever=ar)\n",
    "response = query_engine.query(\"What is sentence embedding, based on papers before 2019?\")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dba5faef-9fe9-4acc-a55d-ad7c379697fd",
   "metadata": {},
   "source": [
    "## Advanced querying with QueryFusionRetriever\n",
    "\n",
    "The QueryFusion [Retriever](https://docs.llamaindex.ai/en/stable/examples/retrievers/reciprocal_rerank_fusion.html#reciprocal-rerank-fusion-retriever) is an advanced query mechanism whereby the original query is pre-processed to generate N variations. Each of these rephrased queries is then run against the Vectara engine and rank-fusion is used to combine the best results. \n",
    "\n",
    "Let's see this in action:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "6c662bc3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SBERT, or Sentence-BERT, is a modification of the pre-trained BERT network that uses a siamese and triplet network structure to derive semantically meaningful sentence embeddings that can be compared using cosine similarity. This approach effectively makes it a dual-encoder architecture, where each encoder processes one sentence independently, and the embeddings are then compared.\n"
     ]
    }
   ],
   "source": [
    "query = \"is SBERT a dual encoder? what type of DL architecture does it use?\"\n",
    "query_engine = index.as_query_engine(\n",
    "    similarity_top_k=3,\n",
    "    summary_enabled=False,\n",
    "    llm=llm,\n",
    ")\n",
    "response = query_engine.query(query)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "896e8b7e-c028-4ec6-bb7c-0c05dfb86321",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated queries:\n",
      "1. What is SBERT and how does it differ from traditional encoder models?\n",
      "2. Comparison of SBERT with other dual encoder architectures in deep learning.\n",
      "3. Advantages and disadvantages of using SBERT as a dual encoder in natural language processing tasks.\n",
      "4. How does SBERT utilize deep learning architecture to achieve superior performance in sentence embeddings?\n",
      "SBERT is not a dual encoder. It uses a Bi-Encoder architecture.\n"
     ]
    }
   ],
   "source": [
    "from llama_index.core.retrievers import QueryFusionRetriever\n",
    "from llama_index.core.query_engine import RetrieverQueryEngine\n",
    "import nest_asyncio\n",
    "\n",
    "rf_retriever = QueryFusionRetriever(\n",
    "    [index.as_retriever(similarity_top_k=2)],\n",
    "    similarity_top_k=2,\n",
    "    num_queries=5,  # this includes the origianl query; set this to 1 to disable query generation\n",
    "    mode=\"reciprocal_rerank\",\n",
    "    use_async=True,\n",
    "    verbose=True,\n",
    ")\n",
    "\n",
    "nest_asyncio.apply()     # apply nested async to run in a notebook\n",
    "query_engine = RetrieverQueryEngine.from_args(rf_retriever)\n",
    "response = query_engine.query(query)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1a6b3fd",
   "metadata": {},
   "source": [
    "We can see how the QueryFusionRetriever created additional query variations (they are displayed since we used `verbose=True`) and then the overall response includes the results fused together. This is very helpful in this case because the QueryFusionRetriever creates sub-questions that inquire about the specific architecture of SBERT which is relevant context to answering this question properly."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98c98981",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this notebook we've seen various examples for using Vectara with LlamaIndex, which provides the following benefits:\n",
    "* Vectara provides a complete RAG pipeline, so you don't have to deal with a lot of the details around data ingestion: pre-processing, chunking, embedding, etc. Instead all these steps are handled automatically and efficiently in Vectara. \n",
    "* Being a platform, Vectara uses its own internal Embedding model (Boomerang), its own vector storage, and calls the LLM for summarization, so you don't have to maintain separate API keys and relationships with additional vendors or install other products.\n",
    "* Vectara is built for large scale GenAI applications, and with the tools provided by LlamaIndex like Auto Retrieval and Query Fusion, you can easily build and test advanced RAG applications at enteprise scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c950620-9618-4d2c-9597-58ec5adae393",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}