python-recipes/semantic-router/01_routing_optimization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cbba56a9",
   "metadata": {},
   "source": [
    "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n",
    "# Routing Optimization\n",
    "\n",
    "Implementing a semantic router is a great light weight way to add branching logic to your application without taking on additional LLM calls. However, it can be tough to determine the optimal distance threshold values for your routes to maximize performance. This guide will walk through:\n",
    "\n",
    "- how to configure a semantic router\n",
    "- how to optimize the distance thresholds for the routes\n",
    "- a comparison between performing similar logic with an LLM versus a router\n",
    "\n",
    "## Let's Begin!\n",
    "<a href=\"https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/semantic-router/01_routing_optimization.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19bdc2a5-2192-4f5f-bd6e-7c956fd0e230",
   "metadata": {},
   "source": [
    "# Setup\n",
    "\n",
    "## Install Packages\n",
    "> Threshold optimizers are scheduled for release with `0.5.0` so we will pull directly from that branch for the time being."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c620286e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install -q sentence-transformers ranx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "717284f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting git+https://github.com/redis/redis-vl-python.git@0.5.0\n",
      "  Cloning https://github.com/redis/redis-vl-python.git (to revision 0.5.0) to /private/var/folders/_g/rr4lnxxx1_z7m78lz89dhvsm0000gp/T/pip-req-build-54zjmrpr\n",
      "  Running command git clone --filter=blob:none --quiet https://github.com/redis/redis-vl-python.git /private/var/folders/_g/rr4lnxxx1_z7m78lz89dhvsm0000gp/T/pip-req-build-54zjmrpr\n",
      "  Running command git checkout -b 0.5.0 --track origin/0.5.0\n",
      "  Switched to a new branch '0.5.0'\n",
      "  branch '0.5.0' set up to track 'origin/0.5.0'.\n",
      "  Resolved https://github.com/redis/redis-vl-python.git to commit 3ca4c97baa9640d24feedd3bb3791cf95859367d\n",
      "  Installing build dependencies ... \u001b[?25ldone\n",
      "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n",
      "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: coloredlogs<16.0,>=15.0 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (15.0.1)\n",
      "Requirement already satisfied: ml-dtypes<0.5.0,>=0.4.0 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (0.4.1)\n",
      "Requirement already satisfied: numpy<2,>=1 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (1.26.4)\n",
      "Requirement already satisfied: pydantic<3,>=2 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (2.7.4)\n",
      "Requirement already satisfied: python-ulid<4.0.0,>=3.0.0 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (3.0.0)\n",
      "Requirement already satisfied: pyyaml<7.0,>=5.4 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (6.0.1)\n",
      "Requirement already satisfied: redis<6.0,>=5.0 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (5.0.5)\n",
      "Requirement already satisfied: tabulate<0.10.0,>=0.9.0 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (0.9.0)\n",
      "Requirement already satisfied: tenacity>=8.2.2 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (8.3.0)\n",
      "Requirement already satisfied: humanfriendly>=9.1 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from coloredlogs<16.0,>=15.0->redisvl==0.4.1) (10.0)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from pydantic<3,>=2->redisvl==0.4.1) (0.7.0)\n",
      "Requirement already satisfied: pydantic-core==2.18.4 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from pydantic<3,>=2->redisvl==0.4.1) (2.18.4)\n",
      "Requirement already satisfied: typing-extensions>=4.6.1 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from pydantic<3,>=2->redisvl==0.4.1) (4.12.2)\n",
      "Building wheels for collected packages: redisvl\n",
      "  Building wheel for redisvl (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for redisvl: filename=redisvl-0.4.1-py3-none-any.whl size=113401 sha256=973e3b34a10bf10547873947798f4c37f681a87bd1f53c7cf938f2b4bccd71a6\n",
      "  Stored in directory: /private/var/folders/_g/rr4lnxxx1_z7m78lz89dhvsm0000gp/T/pip-ephem-wheel-cache-wt36bttp/wheels/95/dc/1e/d8dc251e38989044675dae0b596a2dee10cbfdecac5c62ccdf\n",
      "Successfully built redisvl\n",
      "Installing collected packages: redisvl\n",
      "Successfully installed redisvl-0.4.1\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install git+https://github.com/redis/redis-vl-python.git@0.5.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1250544",
   "metadata": {},
   "source": [
    "### Grab data (if colab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76c1f678",
   "metadata": {},
   "outputs": [],
   "source": [
    "# NBVAL_SKIP\n",
    "!git clone https://github.com/redis-developer/redis-ai-resources.git temp_repo\n",
    "!mv temp_repo/python-recipes/semantic-router/resources .\n",
    "!rm -rf temp_repo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "323aec7f",
   "metadata": {},
   "source": [
    "## Run a Redis instance\n",
    "\n",
    "#### For Colab\n",
    "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cb85a99",
   "metadata": {},
   "outputs": [],
   "source": [
    "# NBVAL_SKIP\n",
    "%%sh\n",
    "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n",
    "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n",
    "sudo apt-get update  > /dev/null 2>&1\n",
    "sudo apt-get install redis-stack-server  > /dev/null 2>&1\n",
    "redis-stack-server --daemonize yes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c5dbaaf",
   "metadata": {},
   "source": [
    "#### For Alternative Environments\n",
    "There are many ways to get the necessary redis-stack instance running\n",
    "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n",
    "own version of Redis Enterprise running, that works too!\n",
    "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n",
    "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d4499ae",
   "metadata": {},
   "source": [
    "### Define the Redis Connection URL\n",
    "\n",
    "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "aefda1d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "# Replace values below with your own if using Redis Cloud instance\n",
    "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n",
    "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\")      # ex: 18374\n",
    "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\")  # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n",
    "\n",
    "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n",
    "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10f4cb85",
   "metadata": {},
   "source": [
    "# Routing with multiple routes\n",
    "\n",
    "## Define the Routes\n",
    "\n",
    "Below we define 3 different routes. One for `faq` (frequently asked questions), one for `general`, and\n",
    "another for `blocked`. Now for this example, the goal here is\n",
    "surely topic \"classification\". But you can create routes and references for\n",
    "almost anything.\n",
    "\n",
    "Each route has a set of references that cover the \"semantic surface area\" of the\n",
    "route. The incoming query from a user needs to be semantically similar to one or\n",
    "more of the references in order to \"match\" on the route. Note that each route can have it's own distinct `distance_threshold` that defines what is considered a match for the particular query. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "60ad280c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from redisvl.extensions.router import Route\n",
    "\n",
    "faq = Route(\n",
    "    name=\"faq\",\n",
    "    references=[\n",
    "        \"How do I reset my password?\",\n",
    "        \"Where can I view my order history?\",\n",
    "        \"How do I update my shipping address?\",\n",
    "        \"Where are my saved payment methods?\",\n",
    "        \"How do I change my email preferences?\",\n",
    "        \"How can I see my loyalty points balance?\",\n",
    "        \"Where do I find my digital receipts?\",\n",
    "        \"How do I enable two-factor authentication?\",\n",
    "        \"Can I change my username or email?\",\n",
    "        \"How do I manage my account settings?\"\n",
    "    ],\n",
    "    metadata={\"category\": \"account_management\", \"priority\": 1},\n",
    "    distance_threshold=0.5\n",
    ")\n",
    "\n",
    "general = Route(\n",
    "    name=\"general\",\n",
    "    references=[\n",
    "        \"I received the wrong item in my order, can you help?\",\n",
    "        \"Can you recommend products that match my specific needs?\",\n",
    "        \"The assembly instructions for my furniture are unclear\",\n",
    "        \"I need help finding a product with particular specifications\",\n",
    "        \"My order arrived damaged, what are my options?\",\n",
    "        \"Can you help me design a room with your products?\",\n",
    "        \"I'm looking for custom sizing options for this product\",\n",
    "        \"The item I received doesn't match the online description\",\n",
    "        \"I need advice on which model would work best for my situation\",\n",
    "        \"Can you help troubleshoot an issue with my recent purchase?\"\n",
    "    ],\n",
    "    metadata={\"category\": \"customer_service\", \"priority\": 2},\n",
    "    distance_threshold=0.5\n",
    ")\n",
    "\n",
    "blocked = Route(\n",
    "    name=\"blocked\",\n",
    "    references=[\n",
    "        \"What is your company's stance on the recent election?\",\n",
    "        \"Do you support liberal or conservative policies?\",\n",
    "        \"Can you tell me another customer's address?\",\n",
    "        \"What's your CEO's opinion on gun control?\",\n",
    "        \"I need personal information about one of your employees\",\n",
    "        \"How does your company vote on political issues?\",\n",
    "        \"Can you provide me with someone's credit card details?\",\n",
    "        \"What's your position on immigration reform?\",\n",
    "        \"I want to know where a specific customer lives\",\n",
    "        \"Does your company donate to political campaigns?\"\n",
    "    ],\n",
    "    metadata={\"category\": \"prohibited\", \"priority\": 3},\n",
    "    distance_threshold=0.5\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cdbcbff",
   "metadata": {},
   "source": [
    "## Initialize the SemanticRouter\n",
    "\n",
    "Like before the ``SemanticRouter`` class will automatically create an index within Redis upon initialization for the route references."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e80aaf84",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from redisvl.extensions.router import SemanticRouter\n",
    "\n",
    "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
    "\n",
    "# Initialize the SemanticRouter\n",
    "ecom_router = SemanticRouter(\n",
    "    name=\"ecom-router\",\n",
    "    routes=[faq, general, blocked],\n",
    "    redis_url=\"redis://localhost:6379\",\n",
    "    overwrite=True # Blow away any other routing index with this name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b199505",
   "metadata": {},
   "source": [
    "## View the created index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3caedb77",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Index Information:\n",
      "╭──────────────┬────────────────┬─────────────────┬─────────────────┬────────────╮\n",
      "│ Index Name   │ Storage Type   │ Prefixes        │ Index Options   │   Indexing │\n",
      "├──────────────┼────────────────┼─────────────────┼─────────────────┼────────────┤\n",
      "│ ecom-router  │ HASH           │ ['ecom-router'] │ []              │          0 │\n",
      "╰──────────────┴────────────────┴─────────────────┴─────────────────┴────────────╯\n",
      "Index Fields:\n",
      "╭────────────┬─────────────┬────────┬────────────────┬────────────────┬────────────────┬────────────────┬────────────────┬────────────────┬─────────────────┬────────────────╮\n",
      "│ Name       │ Attribute   │ Type   │ Field Option   │ Option Value   │ Field Option   │ Option Value   │ Field Option   │   Option Value │ Field Option    │ Option Value   │\n",
      "├────────────┼─────────────┼────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────────────────┼────────────────┤\n",
      "│ route_name │ route_name  │ TAG    │ SEPARATOR      │ ,              │                │                │                │                │                 │                │\n",
      "│ reference  │ reference   │ TEXT   │ WEIGHT         │ 1              │                │                │                │                │                 │                │\n",
      "│ vector     │ vector      │ VECTOR │ algorithm      │ FLAT           │ data_type      │ FLOAT32        │ dim            │            768 │ distance_metric │ COSINE         │\n",
      "╰────────────┴─────────────┴────────┴────────────────┴────────────────┴────────────────┴────────────────┴────────────────┴────────────────┴─────────────────┴────────────────╯\n"
     ]
    }
   ],
   "source": [
    "# look at the index specification created for the semantic router\n",
    "!rvl index info -i ecom-router"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8eb95dde",
   "metadata": {},
   "source": [
    "## Test it out\n",
    "\n",
    "The router will automatically vectorize the query perform a search to find the most related topic and return the related route match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5b0e3208",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RouteMatch(name='faq', distance=0.108501493931)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Query the router with a statement\n",
    "route_match = ecom_router(\"Whatup how do i reset my password?\")\n",
    "route_match"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7bc759b",
   "metadata": {},
   "source": [
    "## Optimize route distance thresholds with test data\n",
    "\n",
    "For optimization within redisvl you can create test data manually or make use of a model to generate some for you. In this case we will use a model to do it for us.\n",
    "\n",
    "Prompt for creating test data:\n",
    "> used claude sonnet 3.7 for generation of resource\n",
    "\n",
    "```txt\n",
    "You are a test data creation helper. \n",
    "\n",
    "Create test data of the form:\n",
    "\n",
    "{\n",
    "    \"query\": \"query about a topic\",\n",
    "    \"query_match\": \"topic-the-query-matches\"\n",
    "}\n",
    "\n",
    "The 3 available topics are: faq, general, and blocked. Generate many examples that map to these topics such that we can train a model to find the best thresholds for this classification task. Also make sure to include some examples that don't map to any of the topics to check the null case for these leave the query_match field empty.\n",
    "```\n",
    "\n",
    "The output of this call was saved to `./resources/test_data.json`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3c03a117",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "with open(\"resources/ecom_train_data.json\", \"r\") as f:\n",
    "    train_data = json.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d0c5c2a",
   "metadata": {},
   "source": [
    "## Run optimization with router"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "83d2a15c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Eval metric F1: start 0.867, end 0.911 \n",
      "Ending thresholds: {'faq': 0.6737373737373737, 'general': 0.28181818181818197, 'blocked': 0.6797979797979801}\n"
     ]
    }
   ],
   "source": [
    "from redisvl.utils.optimize import RouterThresholdOptimizer\n",
    "\n",
    "optimizer = RouterThresholdOptimizer(ecom_router, train_data)\n",
    "optimizer.optimize()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "343964ff",
   "metadata": {},
   "source": [
    "## Test classification against LLM\n",
    "\n",
    "Using the same prompt above we generated and stored another 20 questions to use as our `test_data` to compare against using an LLM model to perform this classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10c83f5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import getpass\n",
    "import time\n",
    "import numpy as np\n",
    "\n",
    "from openai import OpenAI\n",
    "\n",
    "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"False\"\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = os.getenv(\"OPENAI_API_KEY\") or getpass.getpass(\"Enter your OpenAI API key: \")\n",
    "\n",
    "client = OpenAI(api_key=os.environ[\"OPENAI_API_KEY\"])\n",
    "\n",
    "def ask_openai(question: str) -> str:\n",
    "    prompt = f\"\"\"\n",
    "    You are a classification bot. Your job is to classify the following query as either faq, general, blocked, or none. Return only the string label or an empty string if no match.\n",
    "\n",
    "    general is defined as request requiring customer service.\n",
    "    faq is defined as a request for commonly asked account questions.\n",
    "    blocked is defined as a request for prohibited information.\n",
    "\n",
    "    query: \"{question}\"\n",
    "    \"\"\"\n",
    "    response = client.responses.create(\n",
    "      model=\"gpt-4o-mini\",\n",
    "      input=prompt,\n",
    "    )\n",
    "    return response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "feb25546",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"resources/ecom_test_data.json\", \"r\") as f:\n",
    "    test_data = json.load(f)\n",
    "\n",
    "\n",
    "res = ask_openai(test_data[0][\"query\"])\n",
    "res.output_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "5ee72be1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ResponseUsage(input_tokens=117, output_tokens=2, output_tokens_details=OutputTokensDetails(reasoning_tokens=0), total_tokens=119, input_tokens_details={'cached_tokens': 0})"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "e5c921b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "INPUT_TOKEN_PRICE = (0.15 / 1_000_000)\n",
    "OUTPUT_TOKEN_PRICE = (0.60 / 1_000_000)\n",
    "\n",
    "def calc_cost_rough(openai_response):\n",
    "    return openai_response.usage.input_tokens * INPUT_TOKEN_PRICE + openai_response.usage.output_tokens * OUTPUT_TOKEN_PRICE\n",
    "\n",
    "def test_classifier(classifier, test_data, is_router=False):\n",
    "    correct = 0\n",
    "    times = []\n",
    "    costs = []\n",
    "\n",
    "    for data in test_data:\n",
    "        start = time.time()\n",
    "        if is_router:\n",
    "            prediction = classifier(data[\"query\"]).name\n",
    "        else:\n",
    "            openai_response = ask_openai(data[\"query\"])\n",
    "            prediction = openai_response.output_text\n",
    "            costs.append(calc_cost_rough(openai_response))\n",
    "        \n",
    "        if not prediction or prediction.lower() == \"none\":\n",
    "            prediction = \"\"\n",
    "\n",
    "        times.append(time.time() - start)\n",
    "        print(f\"Expected | Observed: {data['query_match']} | {prediction.lower()}\")\n",
    "        if prediction.lower() == data[\"query_match\"]:\n",
    "            correct += 1\n",
    "\n",
    "    accuracy = correct / len(test_data)\n",
    "    avg_time = np.mean(times)\n",
    "    cost = np.sum(costs) if costs else 0\n",
    "    return accuracy, avg_time, round(cost, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "5c6024e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expected | Observed: faq | faq\n",
      "Expected | Observed: faq | general\n",
      "Expected | Observed: faq | faq\n",
      "Expected | Observed: faq | general\n",
      "Expected | Observed: faq | general\n",
      "Expected | Observed: faq | general\n",
      "Expected | Observed: general | general\n",
      "Expected | Observed: general | general\n",
      "Expected | Observed: general | general\n",
      "Expected | Observed: general | general\n",
      "Expected | Observed: general | general\n",
      "Expected | Observed: general | general\n",
      "Expected | Observed: blocked | \n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | general\n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | general\n",
      "Expected | Observed: blocked | blocked\n"
     ]
    }
   ],
   "source": [
    "llm_accuracy, llm_avg_time, llm_cost = test_classifier(ask_openai, test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "c3362a1b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.65, 0.46876639127731323, 0.0004)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm_accuracy, llm_avg_time, llm_cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "40ddc05d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expected | Observed: faq | faq\n",
      "Expected | Observed: faq | \n",
      "Expected | Observed: faq | faq\n",
      "Expected | Observed: faq | \n",
      "Expected | Observed: faq | \n",
      "Expected | Observed: faq | faq\n",
      "Expected | Observed: general | \n",
      "Expected | Observed: general | \n",
      "Expected | Observed: general | \n",
      "Expected | Observed: general | \n",
      "Expected | Observed: general | general\n",
      "Expected | Observed: general | \n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | \n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | blocked\n"
     ]
    }
   ],
   "source": [
    "router_accuracy, router_avg_time, router_cost = test_classifier(ecom_router, test_data, is_router=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "bec49e6f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.55, 0.06067726612091064, 0)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "router_accuracy, router_avg_time, router_cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "8c8e1219",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.725568754914068"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "speed_improvement = (llm_avg_time / router_avg_time)\n",
    "speed_improvement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "cd4b83bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from redisvl.extensions.router.schema import DistanceAggregationMethod\n",
    "from redisvl.extensions.router import RoutingConfig\n",
    "\n",
    "# toggle aggregation method\n",
    "ecom_router.update_routing_config(\n",
    "    RoutingConfig(aggregation_method=DistanceAggregationMethod.min, max_k=3)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "73d2547f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expected | Observed: faq | faq\n",
      "Expected | Observed: faq | \n",
      "Expected | Observed: faq | faq\n",
      "Expected | Observed: faq | \n",
      "Expected | Observed: faq | \n",
      "Expected | Observed: faq | faq\n",
      "Expected | Observed: general | \n",
      "Expected | Observed: general | \n",
      "Expected | Observed: general | \n",
      "Expected | Observed: general | \n",
      "Expected | Observed: general | general\n",
      "Expected | Observed: general | \n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | \n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | blocked\n",
      "Expected | Observed: blocked | blocked\n"
     ]
    }
   ],
   "source": [
    "router_accuracy_min, router_avg_time_min, _  = test_classifier(ecom_router, test_data, is_router=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "e1c3b52b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.55, 0.022012662887573243)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "router_accuracy_min, router_avg_time_min"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acbec96f",
   "metadata": {},
   "source": [
    "## Cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "9baf7a82",
   "metadata": {},
   "outputs": [],
   "source": [
    "ecom_router.delete()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}