trulens_eval/examples/expositional/frameworks/llama_index/llama_index_retrievalquality.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Measuring Retrieval Quality\n",
    "\n",
    "There are a variety of ways we can measure retrieval quality from LLM-based evaluations to embedding similarity. In this example, we will explore the different methods available.\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/truera/trulens/blob/main/trulens_eval/examples/expositional/frameworks/llama_index/llama_index_retrievalquality.ipynb)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "### Install dependencies\n",
    "Let's install some of the dependencies for this notebook if we don't have them already"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ! pip install trulens_eval==0.24.0 llama_index==0.10.11 html2text>=2020.1.16"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add API keys\n",
    "For this quickstart, you will need Open AI and Huggingface keys. The OpenAI key is used for embeddings and GPT, and the Huggingface key is used for evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"...\"\n",
    "os.environ[\"HUGGINGFACE_API_KEY\"] = \"...\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import from LlamaIndex and TruLens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from trulens_eval import Feedback, Tru, TruLlama\n",
    "from trulens_eval.feedback import Embeddings\n",
    "from trulens_eval.feedback.provider.openai import OpenAI\n",
    "\n",
    "tru = Tru()\n",
    "tru.reset_database()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Simple LLM Application\n",
    "\n",
    "This example uses LlamaIndex which internally uses an OpenAI LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import VectorStoreIndex\n",
    "from llama_index.legacy import ServiceContext\n",
    "from llama_index.readers.web import SimpleWebPageReader\n",
    "\n",
    "documents = SimpleWebPageReader(\n",
    "    html_to_text=True\n",
    ").load_data([\"http://paulgraham.com/worked.html\"])\n",
    "\n",
    "from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n",
    "\n",
    "embed_model = HuggingFaceEmbeddings(model_name = \"sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2\")\n",
    "service_context = ServiceContext.from_defaults(embed_model = embed_model)\n",
    "\n",
    "index = VectorStoreIndex.from_documents(documents, service_context = service_context)\n",
    "\n",
    "query_engine = index.as_query_engine(top_k = 5)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Send your first request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = query_engine.query(\"What did the author do growing up?\")\n",
    "print(response)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize Feedback Function(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Initialize provider class\n",
    "openai = OpenAI()\n",
    "\n",
    "# Question/statement relevance between question and each context chunk.\n",
    "f_qs_relevance = Feedback(openai.qs_relevance).on_input().on(\n",
    "    TruLlama.select_source_nodes().node.text\n",
    "    ).aggregate(np.mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f_embed = Embeddings(embed_model=embed_model)\n",
    "\n",
    "f_embed_dist = Feedback(f_embed.cosine_distance).on_input().on(\n",
    "    TruLlama.select_source_nodes().node.text\n",
    "    ).aggregate(np.mean)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instrument app for logging with TruLens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tru_query_engine_recorder = TruLlama(query_engine,\n",
    "    app_id='LlamaIndex_App1',\n",
    "    feedbacks=[f_qs_relevance, f_embed_dist])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# or as context manager\n",
    "with tru_query_engine_recorder as recording:\n",
    "    query_engine.query(\"What did the author do growing up?\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore in a Dashboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tru.run_dashboard() # open a local streamlit app to explore\n",
    "\n",
    "# tru.stop_dashboard() # stop if needed"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you can run `trulens-eval` from a command line in the same folder to start the dashboard."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: Feedback functions evaluated in the deferred manner can be seen in the \"Progress\" page of the TruLens dashboard."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Or view results directly in your notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tru.get_records_and_feedback(app_ids=[])[0] # pass an empty list of app_ids to get all"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.11.4 ('agents')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "vscode": {
   "interpreter": {
    "hash": "7d153714b979d5e6d08dd8ec90712dd93bff2c9b6c1f0c118169738af3430cd4"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}