From 78cb02aa7b2414dfe076a2a088baa4d4787656d8 Mon Sep 17 00:00:00 2001
From: "max.cembalest" <max.cembalest@arthur.ai>
Date: Tue, 4 Jul 2023 11:26:44 -0400
Subject: [PATCH] cleaned arthur tracking demo notebook

---
 .../integrations/arthur_tracking.ipynb        | 460 ++++--------------
 1 file changed, 97 insertions(+), 363 deletions(-)

diff --git a/docs/extras/ecosystem/integrations/arthur_tracking.ipynb b/docs/extras/ecosystem/integrations/arthur_tracking.ipynb
index 188a8ec4547726..5633dd194cf602 100644
--- a/docs/extras/ecosystem/integrations/arthur_tracking.ipynb
+++ b/docs/extras/ecosystem/integrations/arthur_tracking.ipynb
@@ -2,203 +2,121 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "944e4194",
    "metadata": {},
    "source": [
-    "# Arthur LangChain integration"
+    "# Arthur"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b1ccdfe8",
    "metadata": {},
    "source": [
-    "[Arthur](https://www.arthur.ai/) is a model monitoring and observability platform.\n",
+    "[Arthur](https://arthur.ai) is a model monitoring and observability platform.\n",
     "\n",
-    "This notebook shows how to register LLMs (chat and non-chat) as models with the Arthur platform. Then we show how to set up langchain LLMs with an Arthur callback that will automatically log model inferences to Arthur.\n",
+    "The following guide shows how to run a registered chat LLM with the Arthur callback handler to automatically log model inferences to Arthur.\n",
     "\n",
-    "For more information about how to use the Arthur SDK, visit our [docs](http://docs.arthur.ai), in particular our [model onboarding guide](https://docs.arthur.ai/user-guide/walkthroughs/model-onboarding/index.html)"
+    "If you do not have a model currently onboarded to Arthur, visit our [onboarding guide for generative text models](https://docs.arthur.ai/user-guide/walkthroughs/model-onboarding/generative_text_onboarding.html). For more information about how to use the Arthur SDK, visit our [docs](https://docs.arthur.ai/)."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
-   "id": "961c6691",
-   "metadata": {},
+   "execution_count": 2,
+   "metadata": {
+    "id": "y8ku6X96sebl"
+   },
    "outputs": [],
    "source": [
     "from langchain.callbacks import ArthurCallbackHandler\n",
     "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
-    "from langchain.chat_models import ChatOpenAI, ChatAnthropic\n",
-    "from langchain.schema import HumanMessage\n",
-    "from langchain.llms import OpenAI, Cohere, HuggingFacePipeline"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "a23d1963",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from arthurai import ArthurAI\n",
-    "from arthurai.common.constants import InputType, OutputType, Stage, ValueType\n",
-    "from arthurai.core.attributes import ArthurAttribute, AttributeCategory"
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.schema import HumanMessage"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "4d1b90c0",
    "metadata": {},
    "source": [
-    "# ArthurModel for chatbot with only input text and output text attributes"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "1a4a4a8a",
-   "metadata": {},
-   "source": [
-    "Connect to Arthur client"
+    "Place Arthur credentials here"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
-   "id": "f49e9b79",
-   "metadata": {},
+   "execution_count": 3,
+   "metadata": {
+    "id": "Me3prhqjsoqz"
+   },
    "outputs": [],
    "source": [
     "arthur_url = \"https://app.arthur.ai\"\n",
-    "arthur_login = \"your-username-here\"\n",
-    "arthur = ArthurAI(url=arthur_url, login=arthur_login)"
+    "arthur_login = \"your-arthur-login-username-here\"\n",
+    "arthur_model_id = \"your-arthur-model-id-here\""
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c6e063bf",
    "metadata": {},
    "source": [
-    "Before you can register model inferences to Arthur, you must have a registered model with an ID in the Arthur platform. We will provide this ID to the ArthurCallbackHandler.\n",
-    "\n",
-    "You can register a model with Arthur here in the notebook using this `register_chat_llm()` function. This function returns the ID of the model saved to the platform. To use the function, uncomment `arthur_model_chatbot_id = register_chat_llm()` in the cell below."
+    "Create Langchain LLM with Arthur callback handler"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
-   "id": "31b17b5e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def register_chat_llm():\n",
-    "\n",
-    "    arthur_model = arthur.model(\n",
-    "        display_name=\"LangChainChat\",\n",
-    "        input_type=InputType.NLP,\n",
-    "        output_type=OutputType.TokenSequence\n",
-    "    )\n",
-    "\n",
-    "    arthur_model._add_attribute_to_model(ArthurAttribute(\n",
-    "        name=\"my_input_text\",\n",
-    "        stage=Stage.ModelPipelineInput,\n",
-    "        value_type=ValueType.Unstructured_Text,\n",
-    "        categorical=True,\n",
-    "        is_unique=True\n",
-    "    ))\n",
-    "    arthur_model._add_attribute_to_model(ArthurAttribute(\n",
-    "        name=\"my_output_text\",\n",
-    "        stage=Stage.PredictedValue,\n",
-    "        value_type=ValueType.Unstructured_Text,\n",
-    "        categorical=True,\n",
-    "        is_unique=False,\n",
-    "    ))\n",
-    "    \n",
-    "    return arthur_model.save()\n",
-    "# arthur_model_chatbot_id = register_chat_llm()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0d1d1e60",
-   "metadata": {},
-   "source": [
-    "Alternatively, you can set the `arthur_model_chatbot_id` variable to be the ID of your model on your [model dashboard](https://app.arthur.ai/)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "cdfa02c8",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "arthur_model_chatbot_id = \"your-model-id-here\""
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "58be5234",
-   "metadata": {},
-   "source": [
-    "This function creates a Langchain chat LLM with the ArthurCallbackHandler to log inferences to Arthur. We provide our `arthur_model_chatbot_id`, as well as the Arthur url and login we are using."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "448a8fee",
-   "metadata": {},
+   "execution_count": 4,
+   "metadata": {
+    "id": "9Hq9snQasynA"
+   },
    "outputs": [],
    "source": [
-    "def make_langchain_chat_llm(chat_model=ChatOpenAI):\n",
-    "    if chat_model not in [ChatOpenAI, ChatAnthropic]:\n",
-    "        raise ValueError(\"For this notebook, use one of the chat models imported from langchain.chat_models\")\n",
-    "    return chat_model(\n",
-    "        streaming=True, \n",
+    "def make_langchain_chat_llm(chat_model=):\n",
+    "    return ChatOpenAI(\n",
+    "        streaming=True,\n",
     "        temperature=0.1,\n",
     "        callbacks=[\n",
-    "            StreamingStdOutCallbackHandler(), \n",
-    "            ArthurCallbackHandler.from_credentials(arthur_model_chatbot_id, arthur_url=arthur_url, arthur_login=arthur_login)\n",
-    "        ])\n"
+    "            StreamingStdOutCallbackHandler(),\n",
+    "            ArthurCallbackHandler.from_credentials(\n",
+    "                arthur_model_id, \n",
+    "                arthur_url=arthur_url, \n",
+    "                arthur_login=arthur_login)\n",
+    "        ])"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "id": "17c182da",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "2dfc00ed",
+   "execution_count": 10,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Please enter password for admin: ········\n"
+     ]
+    }
+   ],
    "source": [
-    "chat_llm = make_langchain_chat_llm()"
+    "chatgpt = make_langchain_chat_llm()"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "139291f2",
-   "metadata": {},
+   "metadata": {
+    "id": "aXRyj50Ls8eP"
+   },
    "source": [
-    "Run the chatbot (it will save the chat history in the `history` list so that the conversation can reference earlier messages)\n",
+    "Running the chat LLM with this `run` function will save the chat history in an ongoing list so that the conversation can reference earlier messages and log each response to the Arthur platform. You can view the history of this model's inferences on your [model dashboard page](https://app.arthur.ai/).\n",
     "\n",
-    "Type `q` to quit"
+    "Enter `q` to quit the run loop"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
-   "id": "7480a443",
-   "metadata": {},
+   "execution_count": 13,
+   "metadata": {
+    "id": "4taWSbN-s31Y"
+   },
    "outputs": [],
    "source": [
-    "def run_langchain_chat_llm(llm):\n",
+    "def run(llm):\n",
     "    history = []\n",
     "    while True:\n",
     "        user_input = input(\"\\n>>> input >>>\\n>>>: \")\n",
@@ -207,240 +125,56 @@
     "        history.append(llm(history))"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "6868ce71",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "run_langchain_chat_llm(chat_llm)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a0be7d01",
-   "metadata": {},
-   "source": [
-    "# ArthurModel with input text, output text, token likelihoods, finish reason, and amount of token usage attributes"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "1ee4b741",
-   "metadata": {},
-   "source": [
-    "This function registers an LLM with additional metadata attributes to log to Arthur with each inference\n",
-    "\n",
-    "As above, you can register your callback handler for an LLM using this function here in the notebook or by pasting the ID of an already-registered model from your [model dashboard](https://app.arthur.ai/)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "e671836c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def register_llm():\n",
-    "\n",
-    "    arthur_model = arthur.model(\n",
-    "        display_name=\"LangChainLLM\",\n",
-    "        input_type=InputType.NLP,\n",
-    "        output_type=OutputType.TokenSequence\n",
-    "    )\n",
-    "    arthur_model._add_attribute_to_model(ArthurAttribute(\n",
-    "        name=\"my_input_text\",\n",
-    "        stage=Stage.ModelPipelineInput,\n",
-    "        value_type=ValueType.Unstructured_Text,\n",
-    "        categorical=True,\n",
-    "        is_unique=True\n",
-    "    ))\n",
-    "    arthur_model._add_attribute_to_model(ArthurAttribute(\n",
-    "        name=\"my_output_text\",\n",
-    "        stage=Stage.PredictedValue,\n",
-    "        value_type=ValueType.Unstructured_Text,\n",
-    "        categorical=True,\n",
-    "        is_unique=False,\n",
-    "        token_attribute_link=\"my_output_likelihoods\"\n",
-    "    ))\n",
-    "    arthur_model._add_attribute_to_model(ArthurAttribute(\n",
-    "        name=\"my_output_likelihoods\",\n",
-    "        stage=Stage.PredictedValue,\n",
-    "        value_type=ValueType.TokenLikelihoods,\n",
-    "        token_attribute_link=\"my_output_text\"\n",
-    "    ))\n",
-    "    arthur_model._add_attribute_to_model(ArthurAttribute(\n",
-    "        name=\"finish_reason\",\n",
-    "        stage=Stage.NonInputData,\n",
-    "        value_type=ValueType.String,\n",
-    "        categorical=True,\n",
-    "        categories=[\n",
-    "            AttributeCategory(value='stop'),\n",
-    "            AttributeCategory(value='length'),\n",
-    "            AttributeCategory(value='content_filter'),\n",
-    "            AttributeCategory(value='null')\n",
-    "        ]\n",
-    "    ))\n",
-    "    arthur_model._add_attribute_to_model(ArthurAttribute(\n",
-    "        name=\"prompt_tokens\",\n",
-    "        stage=Stage.NonInputData,\n",
-    "        value_type=ValueType.Integer\n",
-    "    ))\n",
-    "    arthur_model._add_attribute_to_model(ArthurAttribute(\n",
-    "        name=\"completion_tokens\",\n",
-    "        stage=Stage.NonInputData,\n",
-    "        value_type=ValueType.Integer\n",
-    "    ))\n",
-    "    arthur_model._add_attribute_to_model(ArthurAttribute(\n",
-    "        name=\"duration\",\n",
-    "        stage=Stage.NonInputData,\n",
-    "        value_type=ValueType.Float\n",
-    "    ))\n",
-    "    \n",
-    "    return arthur_model.save()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "2a6686f7",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "arthur_model_llm_id = \"your-model-id-here\""
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "2dcacb96",
-   "metadata": {},
-   "source": [
-    "These functions create Langchain LLMs with the ArthurCallbackHandler to log inferences to Arthur.\n",
-    "\n",
-    "There are small differences in the underlying Langchain integrations with these libraries and the available metadata for model inputs & outputs"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 23,
-   "id": "34cf0072",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def make_langchain_openai_llm():\n",
-    "    return OpenAI(\n",
-    "        temperature=0.1,\n",
-    "        model_kwargs = {'logprobs': 3},\n",
-    "        callbacks=[\n",
-    "            ArthurCallbackHandler.from_credentials(arthur_model_llm_id, arthur_url=arthur_url, arthur_login=arthur_login)\n",
-    "        ])\n",
-    "\n",
-    "def make_langchain_cohere_llm():\n",
-    "    return Cohere(\n",
-    "        temperature=0.1,\n",
-    "        callbacks=[\n",
-    "            ArthurCallbackHandler.from_credentials(arthur_model_chatbot_id, arthur_url=arthur_url, arthur_login=arthur_login)\n",
-    "        ])\n",
-    "\n",
-    "def make_langchain_huggingface_llm():\n",
-    "    llm = HuggingFacePipeline.from_model_id(\n",
-    "        model_id=\"bert-base-uncased\", \n",
-    "        task=\"text-generation\", \n",
-    "        model_kwargs={\"temperature\":2.5, \"max_length\":64})\n",
-    "    llm.callbacks = [\n",
-    "        ArthurCallbackHandler.from_credentials(arthur_model_chatbot_id, arthur_url=arthur_url, arthur_login=arthur_login)\n",
-    "    ]\n",
-    "    return llm"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 24,
-   "id": "f40c3ce0",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "openai_llm = make_langchain_openai_llm()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 25,
-   "id": "8476d531",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "cohere_llm = make_langchain_cohere_llm()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "7483b9d3",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "huggingface_llm = make_langchain_huggingface_llm()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c17d8e86",
-   "metadata": {},
-   "source": [
-    "Run the LLM (each completion is independent, no chat history is saved as we were doing above with the chat llms)\n",
-    "\n",
-    "Type `q` to quit"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": 17,
-   "id": "72ee0790",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def run_langchain_llm(llm):\n",
-    "    while True:\n",
-    "        print(\"Type your text for completion:\\n\")\n",
-    "        user_input = input(\"\\n>>> input >>>\\n>>>: \")\n",
-    "        if user_input == 'q': break\n",
-    "        print(llm(user_input), \"\\n================\\n\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "id": "fb864057",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "run_langchain_llm(openai_llm)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "id": "e6673769",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "run_langchain_llm(cohere_llm)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "85541f1c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "run_langchain_llm(huggingface_llm)"
+   "metadata": {
+    "id": "MEx8nWJps-EG"
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      ">>> input >>>\n",
+      ">>>: What is a callback handler?\n",
+      "A callback handler, also known as a callback function or callback method, is a piece of code that is executed in response to a specific event or condition. It is commonly used in programming languages that support event-driven or asynchronous programming paradigms.\n",
+      "\n",
+      "The purpose of a callback handler is to provide a way for developers to define custom behavior that should be executed when a certain event occurs. Instead of waiting for a result or blocking the execution, the program registers a callback function and continues with other tasks. When the event is triggered, the callback function is invoked, allowing the program to respond accordingly.\n",
+      "\n",
+      "Callback handlers are commonly used in various scenarios, such as handling user input, responding to network requests, processing asynchronous operations, and implementing event-driven architectures. They provide a flexible and modular way to handle events and decouple different components of a system.\n",
+      ">>> input >>>\n",
+      ">>>: What do I need to do to get the full benefits of this\n",
+      "To get the full benefits of using a callback handler, you should consider the following:\n",
+      "\n",
+      "1. Understand the event or condition: Identify the specific event or condition that you want to respond to with a callback handler. This could be user input, network requests, or any other asynchronous operation.\n",
+      "\n",
+      "2. Define the callback function: Create a function that will be executed when the event or condition occurs. This function should contain the desired behavior or actions you want to take in response to the event.\n",
+      "\n",
+      "3. Register the callback function: Depending on the programming language or framework you are using, you may need to register or attach the callback function to the appropriate event or condition. This ensures that the callback function is invoked when the event occurs.\n",
+      "\n",
+      "4. Handle the callback: Implement the necessary logic within the callback function to handle the event or condition. This could involve updating the user interface, processing data, making further requests, or triggering other actions.\n",
+      "\n",
+      "5. Consider error handling: It's important to handle any potential errors or exceptions that may occur within the callback function. This ensures that your program can gracefully handle unexpected situations and prevent crashes or undesired behavior.\n",
+      "\n",
+      "6. Maintain code readability and modularity: As your codebase grows, it's crucial to keep your callback handlers organized and maintainable. Consider using design patterns or architectural principles to structure your code in a modular and scalable way.\n",
+      "\n",
+      "By following these steps, you can leverage the benefits of callback handlers, such as asynchronous and event-driven programming, improved responsiveness, and modular code design.\n",
+      ">>> input >>>\n",
+      ">>>: q\n"
+     ]
+    }
+   ],
+   "source": [
+    "run(chatgpt)"
    ]
   }
  ],
  "metadata": {
+  "colab": {
+   "provenance": []
+  },
   "kernelspec": {
    "display_name": "Python 3 (ipykernel)",
    "language": "python",
@@ -456,9 +190,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.8"
+   "version": "3.10.11"
   }
  },
  "nbformat": 4,
- "nbformat_minor": 5
+ "nbformat_minor": 1
 }