cookbook/evaluation_with_uptrain.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hIDdGw6hepiF"
      },
      "source": [
        "---\n",
        "description: This notebook demonstrates how to run UpTrain's evaluation metrics on the traces generated by Langfuse.\n",
        "category: Evaluation\n",
        "---\n",
        "\n",
        "# Evaluate Langfuse LLM Traces with UpTrain\n",
        "\n",
        "UpTrain's open-source library offers a series of evaluation metrics to assess LLM applications.\n",
        "\n",
        "This notebook demonstrates how to run UpTrain's evaluation metrics on the traces generated by Langfuse. In Langfuse you can then monitor these scores over time or use them to compare different experiments."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setup\n",
        "\n",
        "You can get your Langfuse API keys [here](https://cloud.langfuse.com/) and OpenAI API key [here](https://platform.openai.com/api-keys)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "%pip install langfuse datasets uptrain litellm openai --upgrade"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "R1nkKfEUepiH"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# get keys for your project from https://cloud.langfuse.com\n",
        "os.environ[\"LANGFUSE_PUBLIC_KEY\"] = \"pk-lf-...\"\n",
        "os.environ[\"LANGFUSE_SECRET_KEY\"] = \"sk-lf-...\"\n",
        "os.environ[\"LANGFUSE_HOST\"] = \"https://cloud.langfuse.com\" # for EU data region\n",
        "# os.environ[\"LANGFUSE_HOST\"] = \"https://us.cloud.langfuse.com\" # for US data region\n",
        "\n",
        "# your openai key\n",
        "os.environ[\"OPENAI_API_KEY\"] = \"\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6lfVmRQlepiI"
      },
      "source": [
        "## Sample Dataset\n",
        "\n",
        "We use this dataset to represent [traces](https://langfuse.com/docs/tracing) that you have logged to Langfuse. In a production environment, you would use your own data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N-wEyOQzepiI",
        "outputId": "48b3948c-22f1-4802-d5e7-2f3c6eb9689d"
      },
      "outputs": [],
      "source": [
        "data = [\n",
        "    {\n",
        "        \"question\": \"What are the symptoms of a heart attack?\",\n",
        "        \"context\": \"A heart attack, or myocardial infarction, occurs when the blood supply to the heart muscle is blocked. Chest pain is a good symptom of heart attack, though there are many others.\",\n",
        "        \"response\": \"Symptoms of a heart attack may include chest pain or discomfort, shortness of breath, nausea, lightheadedness, and pain or discomfort in one or both arms, the jaw, neck, or back.\"\n",
        "    },\n",
        "    {\n",
        "        \"question\": \"Can stress cause physical health problems?\",\n",
        "        \"context\": \"Stress is the body's response to challenges or threats. Yes, chronic stress can contribute to various physical health problems, including cardiovascular issues.\",\n",
        "        \"response\": \"Yes, chronic stress can contribute to various physical health problems, including cardiovascular issues, and a weakened immune system.\"\n",
        "    },\n",
        "    {\n",
        "        'question': \"What are the symptoms of a heart attack?\",\n",
        "        'context': \"A heart attack, or myocardial infarction, occurs when the blood supply to the heart muscle is blocked. Symptoms of a heart attack may include chest pain or discomfort, shortness of breath and nausea.\",\n",
        "        'response': \"Heart attack symptoms are usually just indigestion and can be relieved with antacids.\"\n",
        "    },\n",
        "    {\n",
        "        'question': \"Can stress cause physical health problems?\",\n",
        "        'context': \"Stress is the body's response to challenges or threats. Yes, chronic stress can contribute to various physical health problems, including cardiovascular issues.\",\n",
        "        'response': \"Stress is not real, it is just imaginary!\"\n",
        "    }\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Run Evaluations using UpTrain\n",
        "\n",
        "We have used the following 3 metrics from UpTrain's open-source library:\n",
        "\n",
        "1. [Context Relevance](https://docs.uptrain.ai/predefined-evaluations/context-awareness/context-relevance): Evaluates how relevant the retrieved context is to the question specified.\n",
        "\n",
        "2. [Factual Accuracy](https://docs.uptrain.ai/predefined-evaluations/context-awareness/factual-accuracy): Evaluates whether the response generated is factually correct and grounded by the provided context.\n",
        "\n",
        "3. [Response Completeness](https://docs.uptrain.ai/predefined-evaluations/response-quality/response-completeness): Evaluates whether the response has answered all the aspects of the question specified.\n",
        "\n",
        "You can look at the complete list of UpTrain's supported metrics [here](https://docs.uptrain.ai/predefined-evaluations/overview)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from uptrain import EvalLLM, Evals\n",
        "import json\n",
        "import pandas as pd\n",
        "\n",
        "eval_llm = EvalLLM(openai_api_key=os.environ[\"OPENAI_API_KEY\"])\n",
        "\n",
        "res = eval_llm.evaluate(\n",
        "    data = data,\n",
        "    checks = [Evals.CONTEXT_RELEVANCE, Evals.FACTUAL_ACCURACY, Evals.RESPONSE_COMPLETENESS]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Using Langfuse\n",
        "\n",
        "There are two main ways to run evaluations:\n",
        "\n",
        "1. Score each Trace (in development): This means you will run the UpTrain evaluations for each trace item.\n",
        "\n",
        "2. Score in Batches (in production): In this method we will simulate fetching production traces on a periodic basis to score them using the UpTrain evaluators. Often, you'll want to sample the traces instead of scoring all of them to control evaluation costs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Development: Score each trace while it's created"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from langfuse import Langfuse\n",
        " \n",
        "langfuse = Langfuse()\n",
        "\n",
        "langfuse.auth_check()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We mock the instrumentation of your application by using the sample dataset. See the [quickstart](https://langfuse.com/docs/get-started) to integrate Langfuse with your application."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# start a new trace when you get a question\n",
        "question = data[0]['question']\n",
        "trace = langfuse.trace(name = \"uptrain trace\")\n",
        "\n",
        "# retrieve the relevant chunks\n",
        "# chunks = get_similar_chunks(question)\n",
        "context = data[0]['context']\n",
        "# pass it as span\n",
        "trace.span(\n",
        "    name = \"retrieval\", input={'question': question}, output={'context': context}\n",
        ")\n",
        "\n",
        "# use llm to generate a answer with the chunks\n",
        "# answer = get_response_from_llm(question, chunks)\n",
        "response = data[0]['response']\n",
        "trace.span(\n",
        "    name = \"generation\", input={'question': question, 'context': context}, output={'response': response}\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We reuse the scores previously calculated for the traces in the sample dataset. In development, you would run the UpTrain evaluations for the single trace as it's created."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "trace.score(name='context_relevance', value=res[0]['score_context_relevance'])\n",
        "trace.score(name='factual_accuracy', value=res[0]['score_factual_accuracy'])\n",
        "trace.score(name='response_completeness', value=res[0]['score_response_completeness'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![UpTrain Evals on a single trace in Langfuse](https://langfuse.com/images/cookbook/uptrain-trace.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Production: Add scores to traces in batches"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To simulate a production environment, we will log our sample dataset to Langfuse."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "for interaction in data:\n",
        "    trace = langfuse.trace(name = \"uptrain batch\")\n",
        "    trace.span(\n",
        "        name = \"retrieval\",\n",
        "        input={'question': interaction['question']},\n",
        "        output={'context': interaction['context']}\n",
        "    )\n",
        "    trace.span(\n",
        "        name = \"generation\",\n",
        "        input={'question': interaction['question'], 'context': interaction['context']},\n",
        "        output={'response': interaction['response']}\n",
        "    )\n",
        " \n",
        "# await that Langfuse SDK has processed all events before trying to retrieve it in the next step\n",
        "langfuse.flush()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can now retrieve the traces like regular production data and evaluate them using UpTrain."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def get_traces(name=None, limit=10000, user_id=None):\n",
        "    all_data = []\n",
        "    page = 1\n",
        " \n",
        "    while True:\n",
        "        response = langfuse.client.trace.list(\n",
        "            name=name, page=page, user_id=user_id, order_by=None\n",
        "        )\n",
        "        if not response.data:\n",
        "            break\n",
        "        page += 1\n",
        "        all_data.extend(response.data)\n",
        "        if len(all_data) > limit:\n",
        "            break\n",
        " \n",
        "    return all_data[:limit]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Optional: create a random sample to reduce evaluation costs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from random import sample\n",
        " \n",
        "NUM_TRACES_TO_SAMPLE = 4\n",
        "traces = get_traces(name=\"uptrain batch\")\n",
        "traces_sample = sample(traces, NUM_TRACES_TO_SAMPLE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Convert the data into a dataset to be used for evaluation with UpTrain."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "evaluation_batch = {\n",
        "    \"question\": [],\n",
        "    \"context\": [],\n",
        "    \"response\": [],\n",
        "    \"trace_id\": [],\n",
        "}\n",
        " \n",
        "for t in traces_sample:\n",
        "    observations = [langfuse.client.observations.get(o) for o in t.observations]\n",
        "    for o in observations:\n",
        "        if o.name == 'retrieval':\n",
        "            question = o.input['question']\n",
        "            context = o.output['context']\n",
        "        if o.name=='generation':\n",
        "            answer = o.output['response']\n",
        "    evaluation_batch['question'].append(question)\n",
        "    evaluation_batch['context'].append(context)\n",
        "    evaluation_batch['response'].append(response)\n",
        "    evaluation_batch['trace_id'].append(t.id)\n",
        "\n",
        "\n",
        "data = [dict(zip(evaluation_batch,t)) for t in zip(*evaluation_batch.values())]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Evaluate the batch using UpTrain."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "\n",
        "res = eval_llm.evaluate(\n",
        "    data = data,\n",
        "    checks = [Evals.CONTEXT_RELEVANCE, Evals.FACTUAL_ACCURACY, Evals.RESPONSE_COMPLETENESS]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Add the `trace_id` back the the dataset as it was omitted in the previous step to be compatible with UpTrain."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df = pd.DataFrame(res)\n",
        "\n",
        "# add the langfuse trace_id to the result dataframe\n",
        "df[\"trace_id\"] = [d['trace_id'] for d in data]\n",
        "\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now that we have the evaluations, we can add them back to the traces in Langfuse as [scores](https://langfuse.com/docs/scores)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "for _, row in df.iterrows():\n",
        "    for metric_name in [\"context_relevance\", \"factual_accuracy\",\"response_completeness\"]:\n",
        "        langfuse.score(\n",
        "            name=metric_name,\n",
        "            value=row[\"score_\"+metric_name],\n",
        "            trace_id=row[\"trace_id\"]\n",
        "        )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In Langfuse, you can now see the scores for each trace and monitor them over time."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![UpTrain Evals on a list of traces in Langfuse](https://langfuse.com/images/cookbook/uptrain-batch.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.7"
    },
    "orig_nbformat": 4
  },
  "nbformat": 4,
  "nbformat_minor": 0
}