docs/docs/examples/llm/portkey.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/docs/examples/llm/portkey.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Portkey\n",
    "\n",
    "**Portkey** is a full-stack LLMOps platform that productionizes your Gen AI app reliably and securely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Key Features of Portkey's Integration with Llamaindex:\n",
    "\n",
    "<img src=\"https://3798672042-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeWEp2XRBGxs7C1jgAdk7%2Fuploads%2FjDGBQvw5aFOCqctr0xwp%2FColab%20Version%202.png?alt=media&token=16057c99-b86c-416c-932e-c2b71549c506\" alt=\"header\" width=600 />\n",
    "\n",
    "\n",
    "1. **[🚪 AI Gateway](#🔁-implementing-fallbacks-and-retries-with-portkey)**:\n",
    "    - **[Automated Fallbacks & Retries](#🔁-implementing-fallbacks-and-retries-with-portkey)**: Ensure your application remains functional even if a primary service fails.\n",
    "    - **[Load Balancing](#⚖️-implementing-load-balancing-with-portkey)**: Efficiently distribute incoming requests among multiple models.\n",
    "    - **[Semantic Caching](#🧠-implementing-semantic-caching-with-portkey)**: Reduce costs and latency by intelligently caching results.\n",
    "2. **[🔬 Observability](#🔬-observability-with-portkey)**:\n",
    "    - **Logging**: Keep track of all requests for monitoring and debugging.\n",
    "    - **Requests Tracing**: Understand the journey of each request for optimization.\n",
    "    - **Custom Tags**: Segment and categorize requests for better insights.\n",
    "3. **[📝 Continuous Improvement with User Feedback](#📝-feedback-with-portkey)**:\n",
    "    - **Feedback Collection**: Seamlessly gather feedback on any served request, be it on a generation or conversation level.\n",
    "    - **Weighted Feedback**: Obtain nuanced information by attaching weights to user feedback values.\n",
    "    - **Feedback Metadata**: Incorporate custom metadata with the feedback to provide context, allowing for richer insights and analyses.\n",
    "4. **[🔑 Secure Key Management](#feedback-with-portkey)**:\n",
    "    - **Virtual Keys**: Portkey transforms original provider keys into virtual keys, ensuring your primary credentials remain untouched.\n",
    "    - **Multiple Identifiers**: Ability to add multiple keys for the same provider or the same key under different names for easy identification without compromising security.\n",
    "\n",
    "To harness these features, let's start with the setup:\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/jerryjliu/llama_index/blob/main/docs/docs/examples/llm/portkey.ipynb\" target=\"_blank\">\n",
    "    <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\\\"Open In Colab\\\" width=150 />\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index-llms-portkey"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install llama-index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Installing Llamaindex & Portkey SDK\n",
    "!pip install -U llama_index\n",
    "!pip install -U portkey-ai\n",
    "\n",
    "# Importing necessary libraries and modules\n",
    "from llama_index.llms.portkey import Portkey\n",
    "from llama_index.core.llms import ChatMessage\n",
    "import portkey as pk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You do not need to install **any** other SDKs or import them in your Llamaindex app."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Step 1️⃣: Get your Portkey API Key and your Virtual Keys for OpenAI, Anthropic, and more**\n",
    "\n",
    "**[Portkey API Key](https://app.portkey.ai/)**: Log into [Portkey here](https://app.portkey.ai/), then click on the profile icon on top left and \"Copy API Key\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"PORTKEY_API_KEY\"] = \"PORTKEY_API_KEY\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Virtual Keys](https://docs.portkey.ai/key-features/ai-provider-keys)**\n",
    "1. Navigate to the \"Virtual Keys\" page on [Portkey dashboard](https://app.portkey.ai/) and hit the \"Add Key\" button located at the top right corner.\n",
    "2. Choose your AI provider (OpenAI, Anthropic, Cohere, HuggingFace, etc.), assign a unique name to your key, and, if needed, jot down any relevant usage notes. Your virtual key is ready!\n",
    "\n",
    "<img src=\"https://3798672042-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeWEp2XRBGxs7C1jgAdk7%2Fuploads%2F66S1ik16Gle8jS1u6smr%2Fvirtual_keys.png?alt=media&token=2fec1c39-df4e-4c93-9549-7445a833321c\" alt=\"header\" width=600 />\n",
    "3. Now copy and paste the keys below - you can use them anywhere within the Portkey ecosystem and keep your original key secure and untouched."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "openai_virtual_key_a = \"\"\n",
    "openai_virtual_key_b = \"\"\n",
    "\n",
    "anthropic_virtual_key_a = \"\"\n",
    "anthropic_virtual_key_b = \"\"\n",
    "\n",
    "cohere_virtual_key_a = \"\"\n",
    "cohere_virtual_key_b = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you don't want to use Portkey's Virtual keys, you can also use your AI provider keys directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"OPENAI_API_KEY\"] = \"\"\n",
    "os.environ[\"ANTHROPIC_API_KEY\"] = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Step 2️⃣: Configure Portkey Features**\n",
    "\n",
    "To harness the full potential of Portkey's integration with Llamaindex, you can configure various features as illustrated above. Here's a guide to all Portkey features and the expected values:\n",
    "\n",
    "| Feature             | Config Key              | Value(Type)                                      | Required    |\n",
    "|---------------------|-------------------------|--------------------------------------------------|-------------|\n",
    "| API Key             | `api_key`               | `string`                                         | ✅ Required (can be set externally) |\n",
    "| Mode                | `mode`                  | `fallback`, `loadbalance`, `single`              | ✅ Required |\n",
    "| Cache Type          | `cache_status`          | `simple`, `semantic`                             | ❔ Optional |\n",
    "| Force Cache Refresh | `cache_force_refresh`   | `True`, `False`                                  | ❔ Optional |\n",
    "| Cache Age           | `cache_age`             | `integer` (in seconds)                           | ❔ Optional |\n",
    "| Trace ID            | `trace_id`              | `string`                                         | ❔ Optional |\n",
    "| Retries         | `retry`           | `integer` [0,5]                                  | ❔ Optional |\n",
    "| Metadata            | `metadata`              | `json object` [More info](https://docs.portkey.ai/key-features/custom-metadata)          | ❔ Optional |\n",
    "| Base URL | `base_url` | `url` | ❔ Optional |\n",
    "\n",
    "* `api_key` and `mode` are required values.\n",
    "* You can set your Portkey API key using the Portkey constructor or you can also set it as an environment variable.\n",
    "* There are **3** modes - Single, Fallback, Loadbalance.\n",
    "  * **Single** - This is the standard mode. Use it if you do not want Fallback OR Loadbalance features.\n",
    "  * **Fallback** - Set this mode if you want to enable the Fallback feature. [Check out the guide here](#implementing-fallbacks-and-retries-with-portkey).\n",
    "  * **Loadbalance** - Set this mode if you want to enable the Loadbalance feature. [Check out the guide here](#implementing-load-balancing-with-portkey).\n",
    "\n",
    "Here's an example of how to set up some of these features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "portkey_client = Portkey(\n",
    "    mode=\"single\",\n",
    ")\n",
    "\n",
    "# Since we have defined the Portkey API Key with os.environ, we do not need to set api_key again here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Step 3️⃣: Constructing the LLM**\n",
    "\n",
    "With the Portkey integration, constructing an LLM is simplified. Use the `LLMOptions` function for all providers, with the exact same keys you're accustomed to in your OpenAI or Anthropic constructors. The only new key is `weight`, essential for the load balancing feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "openai_llm = pk.LLMOptions(\n",
    "    provider=\"openai\",\n",
    "    model=\"gpt-4\",\n",
    "    virtual_key=openai_virtual_key_a,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above code illustrates how to utilize the `LLMOptions` function to set up an LLM with the OpenAI provider and the GPT-4 model. This same function can be used for other providers as well, making the integration process streamlined and consistent across various providers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Step 4️⃣: Activate the Portkey Client**\n",
    "\n",
    "Once you've constructed the LLM using the `LLMOptions` function, the next step is to activate it with Portkey. This step is essential to ensure that all the Portkey features are available for your LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "portkey_client.add_llms(openai_llm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, that's it! In just 4 steps, you have infused your Llamaindex app with sophisticated production capabilities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **🔧 Testing the Integration**\n",
    "\n",
    "Let's ensure that everything is set up correctly. Below, we create a simple chat scenario and pass it through our Portkey client to see the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "messages = [\n",
    "    ChatMessage(role=\"system\", content=\"You are a helpful assistant\"),\n",
    "    ChatMessage(role=\"user\", content=\"What can you do?\"),\n",
    "]\n",
    "print(\"Testing Portkey Llamaindex integration:\")\n",
    "response = portkey_client.chat(messages)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how your logs will appear on your [Portkey dashboard](https://app.portkey.ai/):\n",
    "\n",
    "<img src=\"https://portkey.ai/blog/content/images/2023/09/Log-1.png\" alt=\"Logs\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **⏩ Streaming Responses**\n",
    "\n",
    "With Portkey, streaming responses has never been more straightforward. Portkey has 4 response functions:\n",
    "\n",
    "1. `.complete(prompt)`\n",
    "2. `.stream_complete(prompt)`\n",
    "3. `.chat(messages)`\n",
    "4. `.stream_chat(messages)`\n",
    "\n",
    "While the `complete` function expects a string input(`str`), the `chat` function works with an array of `ChatMessage` objects.\n",
    "\n",
    "**Example usage:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's set up a prompt and then use the stream_complete function to obtain a streamed response.\n",
    "\n",
    "prompt = \"Why is the sky blue?\"\n",
    "\n",
    "print(\"\\nTesting Stream Complete:\\n\")\n",
    "response = portkey_client.stream_complete(prompt)\n",
    "for i in response:\n",
    "    print(i.delta, end=\"\", flush=True)\n",
    "\n",
    "# Let's prepare a set of chat messages and then utilize the stream_chat function to achieve a streamed chat response.\n",
    "\n",
    "messages = [\n",
    "    ChatMessage(role=\"system\", content=\"You are a helpful assistant\"),\n",
    "    ChatMessage(role=\"user\", content=\"What can you do?\"),\n",
    "]\n",
    "\n",
    "print(\"\\nTesting Stream Chat:\\n\")\n",
    "response = portkey_client.stream_chat(messages)\n",
    "for i in response:\n",
    "    print(i.delta, end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **🔍 Recap and References**\n",
    "\n",
    "Congratulations! 🎉 You've successfully set up and tested the Portkey integration with Llamaindex. To recap the steps:\n",
    "\n",
    "1. pip install portkey-ai\n",
    "2. from llama_index.llms import Portkey\n",
    "3. Grab your Portkey API Key and create your virtual provider keys from [here](https://app.portkey.ai/).\n",
    "4. Construct your Portkey client and set mode: `portkey_client=Portkey(mode=\"fallback\")`\n",
    "5. Construct your provider LLM with LLMOptions: `openai_llm = pk.LLMOptions(provider=\"openai\", model=\"gpt-4\", virtual_key=openai_key_a)`\n",
    "6. Add the LLM to Portkey with `portkey_client.add_llms(openai_llm)`\n",
    "7. Call the Portkey methods regularly like you would any other LLM, with `portkey_client.chat(messages)`\n",
    "\n",
    "Here's the guide to all the functions and their params:\n",
    "- [Portkey LLM Constructor](#step-2-add-all-the-portkey-features-you-want-as-illustrated-below-by-calling-the-portkey-class)\n",
    "- [LLMOptions Constructor](https://github.com/Portkey-AI/rubeus-python-sdk/blob/4cf3e17b847225123e92f8e8467b41d082186d60/rubeus/api_resources/utils.py#L179)\n",
    "- [List of Portkey + Llamaindex Features](#portkeys-integration-with-llamaindex-adds-the-following-production-capabilities-to-your-apps-out-of-the-box)\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/jerryjliu/llama_index/blob/main/docs/docs/examples/llm/portkey.ipynb\" target=\"_blank\">\n",
    "    <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\\\"Open In Colab\\\" width=150 />\n",
    "</a>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **🔁 Implementing Fallbacks and Retries with Portkey**\n",
    "\n",
    "Fallbacks and retries are essential for building resilient AI applications. With Portkey, implementing these features is straightforward:\n",
    "\n",
    "- **Fallbacks**: If a primary service or model fails, Portkey will automatically switch to a backup model.\n",
    "- **Retries**: If a request fails, Portkey can be configured to retry the request multiple times.\n",
    "\n",
    "Below, we demonstrate how to set up fallbacks and retries using Portkey:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "portkey_client = Portkey(mode=\"fallback\")\n",
    "messages = [\n",
    "    ChatMessage(role=\"system\", content=\"You are a helpful assistant\"),\n",
    "    ChatMessage(role=\"user\", content=\"What can you do?\"),\n",
    "]\n",
    "\n",
    "llm1 = pk.LLMOptions(\n",
    "    provider=\"openai\",\n",
    "    model=\"gpt-4\",\n",
    "    retry_settings={\"on_status_codes\": [429, 500], \"attempts\": 2},\n",
    "    virtual_key=openai_virtual_key_a,\n",
    ")\n",
    "\n",
    "llm2 = pk.LLMOptions(\n",
    "    provider=\"openai\",\n",
    "    model=\"gpt-3.5-turbo\",\n",
    "    virtual_key=openai_virtual_key_b,\n",
    ")\n",
    "\n",
    "portkey_client.add_llms(llm_params=[llm1, llm2])\n",
    "\n",
    "print(\"Testing Fallback & Retry functionality:\")\n",
    "response = portkey_client.chat(messages)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **⚖️ Implementing Load Balancing with Portkey**\n",
    "\n",
    "Load balancing ensures that incoming requests are efficiently distributed among multiple models. This not only enhances the performance but also provides redundancy in case one model fails.\n",
    "\n",
    "With Portkey, implementing load balancing is simple. You need to:\n",
    "\n",
    "- Define the `weight` parameter for each LLM. This weight determines how requests are distributed among the LLMs.\n",
    "- Ensure that the sum of weights for all LLMs equals 1.\n",
    "\n",
    "Here's an example of setting up load balancing with Portkey:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "portkey_client = Portkey(mode=\"ab_test\")\n",
    "\n",
    "messages = [\n",
    "    ChatMessage(role=\"system\", content=\"You are a helpful assistant\"),\n",
    "    ChatMessage(role=\"user\", content=\"What can you do?\"),\n",
    "]\n",
    "\n",
    "llm1 = pk.LLMOptions(\n",
    "    provider=\"openai\",\n",
    "    model=\"gpt-4\",\n",
    "    virtual_key=openai_virtual_key_a,\n",
    "    weight=0.2,\n",
    ")\n",
    "\n",
    "llm2 = pk.LLMOptions(\n",
    "    provider=\"openai\",\n",
    "    model=\"gpt-3.5-turbo\",\n",
    "    virtual_key=openai_virtual_key_a,\n",
    "    weight=0.8,\n",
    ")\n",
    "\n",
    "portkey_client.add_llms(llm_params=[llm1, llm2])\n",
    "\n",
    "print(\"Testing Loadbalance functionality:\")\n",
    "response = portkey_client.chat(messages)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **🧠 Implementing Semantic Caching with Portkey**\n",
    "\n",
    "Semantic caching is a smart caching mechanism that understands the context of a request. Instead of caching based solely on exact input matches, semantic caching identifies similar requests and serves cached results, reducing redundant requests and improving response times as well as saving money.\n",
    "\n",
    "Let's see how to implement semantic caching with Portkey:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "portkey_client = Portkey(mode=\"single\")\n",
    "\n",
    "openai_llm = pk.LLMOptions(\n",
    "    provider=\"openai\",\n",
    "    model=\"gpt-3.5-turbo\",\n",
    "    virtual_key=openai_virtual_key_a,\n",
    "    cache_status=\"semantic\",\n",
    ")\n",
    "\n",
    "portkey_client.add_llms(openai_llm)\n",
    "\n",
    "current_messages = [\n",
    "    ChatMessage(role=\"system\", content=\"You are a helpful assistant\"),\n",
    "    ChatMessage(role=\"user\", content=\"What are the ingredients of a pizza?\"),\n",
    "]\n",
    "\n",
    "print(\"Testing Portkey Semantic Cache:\")\n",
    "\n",
    "start = time.time()\n",
    "response = portkey_client.chat(current_messages)\n",
    "end = time.time() - start\n",
    "\n",
    "print(response)\n",
    "print(f\"{'-'*50}\\nServed in {end} seconds.\\n{'-'*50}\")\n",
    "\n",
    "new_messages = [\n",
    "    ChatMessage(role=\"system\", content=\"You are a helpful assistant\"),\n",
    "    ChatMessage(role=\"user\", content=\"Ingredients of pizza\"),\n",
    "]\n",
    "\n",
    "print(\"Testing Portkey Semantic Cache:\")\n",
    "\n",
    "start = time.time()\n",
    "response = portkey_client.chat(new_messages)\n",
    "end = time.time() - start\n",
    "\n",
    "print(response)\n",
    "print(f\"{'-'*50}\\nServed in {end} seconds.\\n{'-'*50}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Portkey's cache supports two more cache-critical functions - Force Refresh and Age.\n",
    "\n",
    "`cache_force_refresh`: Force-send a request to your provider instead of serving it from a cache.\n",
    "`cache_age`: Decide the interval at which the cache store for this particular string should get automatically refreshed. The cache age is set in seconds.\n",
    "\n",
    "Here's how you can use it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting the cache status as `semantic` and cache_age as 60s.\n",
    "openai_llm = pk.LLMOptions(\n",
    "    provider=\"openai\",\n",
    "    model=\"gpt-3.5-turbo\",\n",
    "    virtual_key=openai_virtual_key_a,\n",
    "    cache_force_refresh=True,\n",
    "    cache_age=60,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **🔬 Observability with Portkey**\n",
    "\n",
    "Having insight into your application's behavior is paramount. Portkey's observability features allow you to monitor, debug, and optimize your AI applications with ease. You can track each request, understand its journey, and segment them based on custom tags. This level of detail can help in identifying bottlenecks, optimizing costs, and enhancing the overall user experience.\n",
    "\n",
    "Here's how to set up observability with Portkey:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata = {\n",
    "    \"_environment\": \"production\",\n",
    "    \"_prompt\": \"test\",\n",
    "    \"_user\": \"user\",\n",
    "    \"_organisation\": \"acme\",\n",
    "}\n",
    "\n",
    "trace_id = \"llamaindex_portkey\"\n",
    "\n",
    "portkey_client = Portkey(mode=\"single\")\n",
    "\n",
    "openai_llm = pk.LLMOptions(\n",
    "    provider=\"openai\",\n",
    "    model=\"gpt-3.5-turbo\",\n",
    "    virtual_key=openai_virtual_key_a,\n",
    "    metadata=metadata,\n",
    "    trace_id=trace_id,\n",
    ")\n",
    "\n",
    "portkey_client.add_llms(openai_llm)\n",
    "\n",
    "print(\"Testing Observability functionality:\")\n",
    "response = portkey_client.chat(messages)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **🌉 Open Source AI Gateway**\n",
    "\n",
    "Portkey's AI Gateway uses the [open source project Rubeus](https://github.com/portkey-ai/rubeus) internally. Rubeus powers features like interoperability of LLMs, load balancing, fallbacks, and acts as an intermediary, ensuring that your requests are processed optimally.\n",
    "\n",
    "One of the advantages of using Portkey is its flexibility. You can easily customize its behavior, redirect requests to different providers, or even bypass logging to Portkey altogether.\n",
    "\n",
    "Here's an example of customizing the behavior with Portkey:\n",
    "\n",
    "```py\n",
    "portkey_client.base_url=None\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **📝 Feedback with Portkey**\n",
    "\n",
    "Continuous improvement is a cornerstone of AI. To ensure your models and applications evolve and serve users better, feedback is vital. Portkey's Feedback API offers a straightforward way to gather weighted feedback from users, allowing you to refine and improve over time.\n",
    "\n",
    "Here's how to utilize the Feedback API with Portkey:\n",
    "\n",
    "Read more about [Feedback here](https://docs.portkey.ai/key-features/feedback-api)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "# Endpoint URL\n",
    "url = \"https://api.portkey.ai/v1/feedback\"\n",
    "\n",
    "# Headers\n",
    "headers = {\n",
    "    \"x-portkey-api-key\": os.environ.get(\"PORTKEY_API_KEY\"),\n",
    "    \"Content-Type\": \"application/json\",\n",
    "}\n",
    "\n",
    "# Data\n",
    "data = {\"trace_id\": \"llamaindex_portkey\", \"value\": 1}\n",
    "\n",
    "# Making the request\n",
    "response = requests.post(url, headers=headers, data=json.dumps(data))\n",
    "\n",
    "# Print the response\n",
    "print(response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the feedback with `weight` and `value` for each trace id is available on the Portkey dashboard:\n",
    "\n",
    "<img src=\"https://portkey.ai/blog/content/images/2023/09/feedback.png\" alt=\"Feedback\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **✅ Conclusion**\n",
    "\n",
    "Integrating Portkey with Llamaindex simplifies the process of building robust and resilient AI applications. With features like semantic caching, observability, load balancing, feedback, and fallbacks, you can ensure optimal performance and continuous improvement.\n",
    "\n",
    "By following this guide, you've set up and tested the Portkey integration with Llamaindex. As you continue to build and deploy AI applications, remember to leverage the full potential of this integration!\n",
    "\n",
    "For further assistance or questions, reach out to the developers ➡️ <br />\n",
    "<a href=\"https://twitter.com/intent/follow?screen_name=portkeyai\" target=\"_blank\">\n",
    "  <img src=\"https://img.shields.io/twitter/follow/portkeyai?style=social&logo=twitter\" alt=\"Twitter\">\n",
    "</a>\n",
    "\n",
    "Join our community of practitioners putting LLMs into production ➡️ <br />\n",
    "<a href=\"https://discord.gg/sDk9JaNfK8\" target=\"_blank\">\n",
    "  <img src=\"https://img.shields.io/discord/1143393887742861333?logo=discord\" alt=\"Discord\">\n",
    "</a>"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}