langkit/examples/Batch_to_Whylabs.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">### 🚩 *Create a free WhyLabs account to get more value out of whylogs!*<br> \n",
    ">*Did you know you can store, visualize, and monitor language model profiles with the [WhyLabs Observability Platform](https://whylabs.ai/whylogs-free-signup?utm_source=github&utm_medium=referral&utm_campaign=langkit)? Sign up for a [free WhyLabs account](https://whylabs.ai/whylogs-free-signup?utm_source=github&utm_medium=referral&utm_campaign=langkit) to leverage the power of LangKit and WhyLabs together!*"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logging and Monitoring Text Metrics for LLMs with LangKit and WhyLabs"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/whylabs/LanguageToolkit/blob/main/langkit/examples/Batch_to_Whylabs.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install 'langkit[all]' -q"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we'll show how you can generate out-of-the-box text metrics using LangKit and whylogs, and then log and monitor them in the WhyLabs Observability Platform.\n",
    "\n",
    "With LangKit, you'll be able to extract relevant signals from unstructured text data, such as:\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Dataset - Chatbot prompts\n",
    "\n",
    "Let's first download a huggingface dataset containing prompts and responses from a chatbot. We'll generate text metrics for the prompts and responses, and then log them to WhyLabs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "initialize hugging face archived chat prompt/response dataset...\n"
     ]
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "print(\"initialize hugging face archived chat prompt/response dataset...\")\n",
    "archived_chats = load_dataset('alespalla/chatbot_instruction_prompts', split=\"test\", streaming=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to send our profile to WhyLabs, let's first set up an account. You can skip this if you already have an account and a model set up.\n",
    "\n",
    "We will need three pieces of information:\n",
    "\n",
    "- API token\n",
    "- Organization ID\n",
    "- Dataset ID (or model-id)\n",
    "\n",
    "Go to https://whylabs.ai/free and grab a free account. You can follow along with the examples if you wish, but if you’re interested in only following this demonstration, you can go ahead and skip the quick start instructions.\n",
    "\n",
    "After that, you’ll be prompted to create an API token. Once you create it, copy and store it locally. The second important information here is your org ID. Take note of it as well. After you get your API Token and Org ID, you can go to https://hub.whylabsapp.com/models to see your projects dashboard. You can create a new project and take note of it's ID (if it's a model project it will look like `model-xxxx`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langkit.config import check_or_prompt_for_api_keys\n",
    "\n",
    "check_or_prompt_for_api_keys()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initializing Metrics from LangKit\n",
    "\n",
    "In order to calculate the text metrics, we simply need to import the relevant modules from `LangKit`. In this case, we will calculate metrics using the following modules:\n",
    "\n",
    "- textstat: text statistics such as scores for readability, complexity, and grade\n",
    "- sentiment: sentiment scores\n",
    "- regexes: label text according to user-defined regex pattern groups\n",
    "- themes: compute sentence similarity scores with respect to groups of: a) known jailbreak and b) LLM refusal of service responses\n",
    "\n",
    "After importing the modules, we can generate a schema that will inform whylogs of the metrics we want to calculate. We can then use this schema to log our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langkit import llm_metrics\n",
    "\n",
    "print(\"downloading models and initialized metrics...\")\n",
    "text_metrics_schema = llm_metrics.init()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Profiling and Writing to WhyLabs - Single Example\n",
    "\n",
    "The following code block will log a single prompt/response pair. The resulting profile will then be sent over to your dashboard at WhyLabs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log this first prompt with whylogs and grab the profile\n",
      "Writing initial profile to WhyLabs:\n",
      "Done writing initial profile to WhyLabs, with success: (True, 'log-zj4EscAKHbaokR7c')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import whylogs as why\n",
    "from whylogs.api.writer.whylabs import WhyLabsWriter\n",
    "\n",
    "# Let's define a WhyLabs writer\n",
    "writer = WhyLabsWriter()\n",
    "\n",
    "# define an iterator over the hugging face dataset of archived prompts\n",
    "chats = iter(archived_chats)\n",
    "# grab the first archived prompt/response and log it\n",
    "archived_prompt_response = next(chats)\n",
    "print(\"Log this first prompt with whylogs and grab the profile\")\n",
    "profile = why.log(archived_prompt_response, schema=text_metrics_schema).profile()\n",
    "\n",
    "# This is a single prompt profile for today, lets write it to WhyLabs\n",
    "print(\"Writing initial profile to WhyLabs:\")\n",
    "status = writer.write(profile)\n",
    "print(f\"Done writing initial profile to WhyLabs, with success: {status}\")\n",
    "print()\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Profiling and Writing to WhyLabs - Multiple Batches\n",
    "\n",
    "Let's get us closer to a real scenario. If you have an LLM-powered system, you'll be interested in monitoring your text inputs/outputs in a streaming fashion. In this case, we'll simulate a streaming scenario by iterating through the examples and logging them into daily batches. Let's say we have 7 days worth of data, with 10 examples per day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Now lets write some data to simulate daily logging for the past 7 days.\n",
      "Downloading 10 records from Hugging Face for 2023-10-11 16:25:50.201344+00:00 and profiling\n",
      "..........\n",
      "Downloading 10 records from Hugging Face for 2023-10-10 16:25:50.201344+00:00 and profiling\n",
      "..........\n",
      "Downloading 10 records from Hugging Face for 2023-10-09 16:25:50.201344+00:00 and profiling\n",
      "..........\n",
      "Downloading 10 records from Hugging Face for 2023-10-08 16:25:50.201344+00:00 and profiling\n",
      "..........\n",
      "Downloading 10 records from Hugging Face for 2023-10-07 16:25:50.201344+00:00 and profiling\n",
      "..........\n",
      "Downloading 10 records from Hugging Face for 2023-10-06 16:25:50.201344+00:00 and profiling\n",
      "..........\n",
      "Done. Go see your metrics on the WhyLabs dashboard!\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime, timedelta, timezone\n",
    "\n",
    "current_date = datetime.now(timezone.utc)\n",
    "\n",
    "print(f\"Now lets write some data to simulate daily logging for the past 7 days.\")\n",
    "batch_size = 10\n",
    "for day in range(1, 7):\n",
    "  # create a separate profile for each day\n",
    "  archived_prompt_response = next(chats)\n",
    "  profile = why.log(archived_prompt_response, schema=text_metrics_schema).profile()\n",
    "  # now log some additional archived prompt/response pairs for this profile to aggregat statistics\n",
    "  # in this profile. The number of prompt/response pairs logged per profile can be very large\n",
    "  # or calculated on different machines, but because the statistics are all mergeable\n",
    "  # we get the rollup of these statistics across the instances processing your data for this day.\n",
    "  archived_prompt_responses = []\n",
    "  dataset_date = current_date - timedelta(days=day)\n",
    "  print(f\"Downloading {batch_size} records from Hugging Face for {dataset_date} and profiling\")\n",
    "\n",
    "  for _ in range(10):\n",
    "    record = next(chats)\n",
    "    archived_prompt_responses.append(record)\n",
    "\n",
    "  for record in archived_prompt_responses:\n",
    "    profile.track(record)\n",
    "    print(\".\", end=\"\", flush=True)\n",
    "  # Now lets take the aggregate profile, set the timestamp and write it to WhyLabs\n",
    "  profile.set_dataset_timestamp(dataset_date)\n",
    "  writer.write(profile)\n",
    "  print()\n",
    "print(\"Done. Go see your metrics on the WhyLabs dashboard!\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And that's it! You can now go to your WhyLabs dashboard and explore the profiles for the past 7 days.\n",
    "\n",
    "Feel free to play around with the code and the metrics. You can inject anomalies manually to see how the metrics change, or you can set monitors and alert over at the WhyLabs dashboard."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "whylabs-textmetricstoolkit-EeFODeF5-py3.8",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}