trulens_eval/examples/expositional/frameworks/llama_index/llama_index_complex_evals.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "X6-q-gTUaZU7"
      },
      "source": [
        "# Advanced Evaluation Methods\n",
        "\n",
        "In this notebook, we will level up our evaluation using chain of thought reasoning. Chain of thought reasoning through interemediate steps improves LLM's ability to perform complex reasoning - and this includes evaluations. Even better, this reasoning is useful for us as humans to identify and understand new failure modes such as irrelevant retrieval or hallucination.\n",
        "\n",
        "Second, in this example we will leverage deferred evaluations. Deferred evaluations can be especially useful for cases such as sub-question queries where the structure of our serialized record can vary. By creating different options for context evaluation, we can use deferred evaluations to try both and use the one that matches the structure of the serialized record. Deferred evaluations can be run later, especially in off-peak times for your app.\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/truera/trulens/blob/main/trulens_eval/examples/expositional/frameworks/llama_index/llama_index_complex_evals.ipynb)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RTSzmVFIaffU",
        "outputId": "0fe81f4b-80c5-4811-fba3-49c45cac2d90"
      },
      "outputs": [],
      "source": [
        "# ! pip install trulens_eval==0.24.0 llama_index==0.10.11 sentence-transformers transformers pypdf gdown"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "SnwSnBkSaZU8"
      },
      "source": [
        "## Query Engine Construction"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IBfdyn3MaZU9"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import openai\n",
        "from trulens_eval import Feedback,  Tru, TruLlama, feedback, Select, FeedbackMode, OpenAI as fOpenAI\n",
        "\n",
        "from trulens_eval.feedback import Groundedness\n",
        "\n",
        "tru = Tru()\n",
        "\n",
        "tru.reset_database()\n",
        "\n",
        "os.environ[\"OPENAI_API_KEY\"] = \"...\"\n",
        "openai.api_key = os.environ[\"OPENAI_API_KEY\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Bh43sV1eaZU9",
        "outputId": "f356b401-d4c2-4496-da7c-9fb9fe4c9b6a"
      },
      "outputs": [],
      "source": [
        "!curl https://www.ipcc.ch/report/ar6/wg2/downloads/report/IPCC_AR6_WGII_Chapter03.pdf --output IPCC_AR6_WGII_Chapter03.pdf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wMvq1q8yaZU-"
      },
      "outputs": [],
      "source": [
        "from llama_index.core import SimpleDirectoryReader\n",
        "\n",
        "documents = SimpleDirectoryReader(\n",
        "    input_files=[\"./IPCC_AR6_WGII_Chapter03.pdf\"]\n",
        ").load_data()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# sentence-window index\n",
        "!gdown \"https://drive.google.com/uc?id=16pH4NETEs43dwJUvYnJ9Z-bsR9_krkrP\"\n",
        "!tar -xzf sentence_index.tar.gz"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sY8Oui4taZU-"
      },
      "outputs": [],
      "source": [
        "# Merge into a single large document rather than one document per-page\n",
        "from llama_index import Document\n",
        "\n",
        "document = Document(text=\"\\n\\n\".join([doc.text for doc in documents]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MkbaDRJCaZU_"
      },
      "outputs": [],
      "source": [
        "from llama_index.core import ServiceContext\n",
        "from llama_index.llms import OpenAI\n",
        "from llama_index.node_parser import SentenceWindowNodeParser\n",
        "\n",
        "# create the sentence window node parser w/ default settings\n",
        "node_parser = SentenceWindowNodeParser.from_defaults(\n",
        "    window_size=3,\n",
        "    window_metadata_key=\"window\",\n",
        "    original_text_metadata_key=\"original_text\",\n",
        ")\n",
        "\n",
        "llm = OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1)\n",
        "sentence_context = ServiceContext.from_defaults(\n",
        "    llm=llm,\n",
        "    embed_model=\"local:BAAI/bge-small-en-v1.5\",\n",
        "    node_parser=node_parser,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JQPRoF21aZU_"
      },
      "outputs": [],
      "source": [
        "from llama_index.core import VectorStoreIndex, StorageContext, load_index_from_storage\n",
        "\n",
        "if not os.path.exists(\"./sentence_index\"):\n",
        "    sentence_index = VectorStoreIndex.from_documents(\n",
        "        [document], service_context=sentence_context\n",
        "    )\n",
        "\n",
        "    sentence_index.storage_context.persist(persist_dir=\"./sentence_index\")\n",
        "else:\n",
        "    sentence_index = load_index_from_storage(\n",
        "        StorageContext.from_defaults(persist_dir=\"./sentence_index\"),\n",
        "        service_context=sentence_context\n",
        "    )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RAERQ_BeaZU_"
      },
      "outputs": [],
      "source": [
        "from llama_index.indices.postprocessor import (\n",
        "    MetadataReplacementPostProcessor,\n",
        "    SentenceTransformerRerank,\n",
        ")\n",
        "\n",
        "sentence_window_engine = sentence_index.as_query_engine(\n",
        "    similarity_top_k=6,\n",
        "    # the target key defaults to `window` to match the node_parser's default\n",
        "    node_postprocessors=[\n",
        "        MetadataReplacementPostProcessor(target_metadata_key=\"window\"),\n",
        "        SentenceTransformerRerank(top_n=2, model=\"BAAI/bge-reranker-base\"),\n",
        "    ],\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PCsOz-3ZaZVB"
      },
      "outputs": [],
      "source": [
        "from llama_index.tools import QueryEngineTool, ToolMetadata\n",
        "from llama_index.query_engine import SubQuestionQueryEngine\n",
        "\n",
        "sentence_sub_engine = SubQuestionQueryEngine.from_defaults(\n",
        "  [QueryEngineTool(\n",
        "    query_engine=sentence_window_engine,\n",
        "    metadata=ToolMetadata(name=\"climate_report\", description=\"Climate Report on Oceans.\")\n",
        "  )],\n",
        "  service_context=sentence_context,\n",
        "  verbose=False,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5KqV-IbQaZVB"
      },
      "outputs": [],
      "source": [
        "import nest_asyncio\n",
        "nest_asyncio.apply()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kXJBD4gfaZVC",
        "outputId": "b4ebd2f9-1768-47be-d0eb-8963f7076ecd"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "# Initialize OpenAI provider\n",
        "provider = fOpenAI()\n",
        "\n",
        "# Helpfulness\n",
        "f_helpfulness = Feedback(provider.helpfulness).on_output() \n",
        "\n",
        "# Question/answer relevance between overall question and answer.\n",
        "f_qa_relevance = Feedback(provider.relevance_with_cot_reasons).on_input_output()\n",
        "\n",
        "# Question/statement relevance between question and each context chunk with context reasoning.\n",
        "# The context is located in a different place for the sub questions so we need to define that feedback separately\n",
        "f_context_relevance_subquestions = (\n",
        "    Feedback(provider.context_relevance_with_cot_reasons)\n",
        "    .on_input()\n",
        "    .on(Select.Record.calls[0].rets.source_nodes[:].node.text)\n",
        "    .aggregate(np.mean)\n",
        ")\n",
        "\n",
        "f_context_relevance = (\n",
        "    Feedback(provider.context_relevance_with_cot_reasons)\n",
        "    .on_input()\n",
        "    .on(Select.Record.calls[0].args.prompt_args.context_str)\n",
        "    .aggregate(np.mean)\n",
        ")\n",
        "\n",
        "# Initialize groundedness\n",
        "# Groundedness with chain of thought reasoning\n",
        "# Similar to context relevance, we'll follow a strategy of definining it twice for the subquestions and overall question.\n",
        "f_groundedness_subquestions = (\n",
        "    Feedback(provider.groundedness_measure_with_cot_reasons)\n",
        "    .on(Select.Record.calls[0].rets.source_nodes[:].node.text.collect())\n",
        "    .on_output()\n",
        ")\n",
        "\n",
        "f_groundedness = (\n",
        "    Feedback(provider.groundedness_measure_with_cot_reasons)\n",
        "    .on(Select.Record.calls[0].args.prompt_args.context_str)\n",
        "    .on_output()\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KUDHInR-aZVC"
      },
      "outputs": [],
      "source": [
        "# We'll use the recorder in deferred mode so we can log all of the subquestions before starting eval.\n",
        "# This approach will give us smoother handling for the evals + more consistent logging at high volume.\n",
        "# In addition, for our two different qs relevance definitions, deferred mode can just take the one that evaluates.\n",
        "tru_recorder = TruLlama(\n",
        "    sentence_sub_engine,\n",
        "    app_id=\"App_1\",\n",
        "    feedbacks=[f_qa_relevance, f_context_relevance, f_context_relevance_subquestions, f_groundedness, f_groundedness_subquestions, f_helpfulness],\n",
        "    feedback_mode=FeedbackMode.DEFERRED\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dsA3ziw1aZVD"
      },
      "outputs": [],
      "source": [
        "questions = [\n",
        "  \"Based on the provided text, discuss the impact of human activities on the natural carbon dynamics of estuaries, shelf seas, and other intertidal and shallow-water habitats. Provide examples from the text to support your answer.\",\n",
        "  \"Analyze the combined effects of exploitation and multi-decadal climate fluctuations on global fisheries yields. How do these factors make it difficult to assess the impacts of global climate change on fisheries yields? Use specific examples from the text to support your analysis.\",\n",
        "  \"Based on the study by Gutiérrez-Rodríguez, A.G., et al., 2018, what potential benefits do seaweeds have in the field of medicine, specifically in relation to cancer treatment?\",\n",
        "  \"According to the research conducted by Haasnoot, M., et al., 2020, how does the uncertainty in Antarctic mass-loss impact the coastal adaptation strategy of the Netherlands?\",\n",
        "  \"Based on the context, explain how the decline in warm water coral reefs is projected to impact the services they provide to society, particularly in terms of coastal protection.\",\n",
        "  \"Tell me something about the intricacies of tying a tie.\",\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "01_P6TxaaZVD",
        "outputId": "4f03da5b-34a3-4d41-ee78-9c09bc97368e"
      },
      "outputs": [],
      "source": [
        "for question in questions:\n",
        "  with tru_recorder as recording:\n",
        "    sentence_sub_engine.query(question)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6Yp4_e4faZVD",
        "outputId": "d2ba9d2d-7e2a-46d2-8459-41ba3778eba3"
      },
      "outputs": [],
      "source": [
        "tru.run_dashboard()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Before we start the evaluator, note that we've logged all of the records including the sub-questions. However we haven't completed any evals yet.\n",
        "\n",
        "Start the evaluator to generate the feedback results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "tru.start_evaluator()"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "milvus",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.14"
    },
    "orig_nbformat": 4
  },
  "nbformat": 4,
  "nbformat_minor": 0
}