 archived/open-assistant-chatbot.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "676d388a-e04f-4681-8bd9-e8ed5eff1db4",
   "metadata": {},
   "source": [
    "# Hugging Face LLM Chatbot application using Gradio  \n",
    "\n",
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.\n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n",
    "\n",
    "---\n",
    "\n",
    "[AWS Deep Learning Containers (DLC)](https://docs.aws.amazon.com/sagemaker/latest/dg/large-model-inference-dlc.html) quickly deploy deep learning environments with optimized, prepackaged container images. This sample notebook is a quick start to deploy open source LLMs to Amazon SageMaker for inference using the [Hugging Face LLM Inference Container](https://huggingface.co/docs/sagemaker/index) which is powered by Text Generation Inference (TGI). [TGI](https://github.com/huggingface/text-generation-inference) is a toolkit for deploying and serving Large Language Models (LLMs). \n",
    "\n",
    "This sample notebook uses open source chat LLM trained using the [Open assistant](https://open-assistant.io/bye) initiative with [Gradio](https://www.gradio.app/) as demo Chat app to validate inference. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12f14136-f7ed-44d1-8f7a-73273f8fa4df",
   "metadata": {},
   "source": [
    "# Enviroment \n",
    "\n",
    "\n",
    "- `Amazon SageMaker Studio JupyterLab` `(JupyterLab 3.0, 5GB storage and ml.t3.medium instance)` with `public internet access`\n",
    "- `Amazon Sagemaker` python SDK version `2.163.0 +`\n",
    "- `Gradio` python version `4.16.0 +`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf8fad8e-50dc-4e90-b72b-b41a0a725934",
   "metadata": {},
   "source": [
    "# Step-1: Uninstall packages\n",
    "\n",
    "This is an optional step included to minimize chances of package dependency conflicts while running the [Gradio](https://www.gradio.app/) chat apps later in this notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45ac511c-d7a1-4d65-8c41-58b2604a83a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cell 01 : Optional step to reduce dependency conflicts with packages\n",
    "\n",
    "!pip uninstall sagemaker -y\n",
    "!pip uninstall gradio -y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e24be05-0f35-42fb-803c-43ab7490882b",
   "metadata": {},
   "source": [
    "# Step-2: Install packages and environment  \n",
    "\n",
    "Install `Amazon SageMaker Python SDK` and `Gradio` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97cbe74a-24e9-4043-a6b4-da62a7760973",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cell 02 : Install packages required for this notebook\n",
    "\n",
    "!pip install sagemaker --quiet\n",
    "!pip install gradio  --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76e85a82-c47b-4329-ad04-5cb7991316f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cell 03 : Get SageMaker session details using AWS SDK for Python (boto3)\n",
    "\n",
    "import sagemaker\n",
    "from sagemaker.huggingface import HuggingFaceModel, get_huggingface_llm_image_uri\n",
    "import time\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "region = sagemaker_session.boto_region_name\n",
    "role = sagemaker.get_execution_role()\n",
    "\n",
    "print(f\"Amazon SageMaker role arn: {role}\")\n",
    "print(f\"Amazon SageMaker session region: {region}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17b435a4-847f-4d82-a3a0-28d18b7acaef",
   "metadata": {},
   "source": [
    "# Step-3: Retrieve LLM image URI\n",
    "\n",
    "Using Amazon SageMaker SDK `get_huggingface_llm_image_uri()` helper function to retrieve appropriate image URI from `Amazon ECR` for the Hugging Face Large Language Model (LLM). This method allows to retrieve the URI for the desired Hugging Face LLM DLC based on the specified `backend`, `session`, `region`, and `version`. Refer [Amazon SageMaker SDK](https://github.com/aws/sagemaker-python-sdk) for details. \n",
    "\n",
    "- `backend` : Valid values include \"huggingface\" and \"lmi\". The \"lmi\" stands for SageMaker LMI inference backend, and \"huggingface\" refers to using Hugging Face TGI inference backend\n",
    "- `session` : The SageMaker Session to use. (Default: None)\n",
    "- `region`  : The AWS region to use for image URI. (default: None)\n",
    "- `version` : The framework version for which to retrieve an image URI\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db6788fc-620e-40d3-bef4-984de9fad64b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cell 04 : Get the image URI for the Hugging Face Large Language Model (LLM) inference\n",
    "\n",
    "# from sagemaker.huggingface import get_huggingface_llm_image_uri\n",
    "\n",
    "llm_image = get_huggingface_llm_image_uri(backend=\"huggingface\", region=region, version=\"0.8.2\")\n",
    "\n",
    "# print Amazon ECR image uri\n",
    "\n",
    "print(f\"llm image uri: {llm_image}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3321460c-265d-4319-aade-54e27012280d",
   "metadata": {},
   "source": [
    "# Step-4: Configure and create SageMaker Endpoint\n",
    "\n",
    "\n",
    "\n",
    "In this section, \n",
    "\n",
    "- Configure the instance type for the SageMaker Deployment that will be used in the `deploy()` function in `Step-5`\n",
    "- Configure the `env` variables for the `HuggingFaceModel` class. Environment variables include\n",
    "    - `HF_MODEL_ID` which corresponds to the model from the HuggingFace Hub that will be deployed\n",
    "    - `SM_NUM_GPUS` to the number of available GPUs on the selected instance type\n",
    "    - `HF_MODEL_QUANTIZE`  environment variable to reduce the memory footprint of the model\n",
    "- Configure the Hugging Face `model` object `image_uri`, Amazon SageMaker execution `role` and `env` variables specified \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dff31700-9b29-4aeb-a848-43c9a3a44120",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cell 05: Configure the model with the ideal sagemaker inference instance & create SageMaker endpoint\n",
    "\n",
    "import json\n",
    "from sagemaker.huggingface import HuggingFaceModel\n",
    "\n",
    "# sagemaker config\n",
    "instance_type = \"ml.g4dn.12xlarge\"\n",
    "number_of_gpu = 4\n",
    "health_check_timeout = 300\n",
    "\n",
    "# Define Model and Endpoint configuration parameter\n",
    "\n",
    "model_name = \"pythia-12b-sft-\" + time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
    "\n",
    "hub = {\n",
    "    \"HF_MODEL_ID\": \"OpenAssistant/pythia-12b-sft-v8-7k-steps\",  # model_id from hf.co/models\n",
    "    \"SM_NUM_GPUS\": json.dumps(number_of_gpu),  # Number of GPU used per replica\n",
    "    \"MAX_INPUT_LENGTH\": json.dumps(1024),  # Max length of input text\n",
    "    \"MAX_TOTAL_TOKENS\": json.dumps(2048),  # Max length of the generation (including input text)\n",
    "    # 'HF_MODEL_QUANTIZE': \"bitsandbytes\", # comment in to quantize\n",
    "}\n",
    "\n",
    "# create HuggingFaceModel with the image uri\n",
    "\n",
    "llm_model = HuggingFaceModel(name=model_name, role=role, image_uri=llm_image, env=hub)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c871a4d5-6328-4057-9e5c-a1f6a2d2b331",
   "metadata": {},
   "source": [
    "# Step-5: Deploy model for real-time inference\n",
    "\n",
    "After creating the HuggingFaceModel, deploy the model by invoking the `deploy()` function. Here the model will be deployed to `ml.g4dn.12xlarge` instance type. TGI will automatically distribute and shard the model across all GPUs.\n",
    "This may take up to `15 mins -20 mins` to complete.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8235fefa-025e-44b2-b271-29ce22bec7ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cell 06: Deploy model to an endpoint\n",
    "\n",
    "llm = llm_model.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type=instance_type,\n",
    "    endpoint_name=model_name,\n",
    "    # volume_size=400, # If using an instance with local SSD storage, volume_size must be None, e.g. p4 but not p3\n",
    "    container_startup_health_check_timeout=health_check_timeout,  # 15 mins to be able to load the model\n",
    ")\n",
    "\n",
    "print(f\"Amazon SageMaker endpoint name : {model_name}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c7abb27-8210-4868-86ce-2d2fdf14241e",
   "metadata": {},
   "source": [
    "# Step-6: Evaluation \n",
    "\n",
    "Once the endpoint is `InService` status, execute the following cell for validation. \n",
    "\n",
    "We will use the predict method from the predictor to evaluate the inference endpoint. We can evaluate with different parameters to impact the generation. Parameters can be defined as in the parameters attribute of the payload. Please refer [open api specification of the TGI](https://huggingface.github.io/text-generation-inference/) in the swagger documentation for parameters supported by TGI.\n",
    "\n",
    "The `OpenAssistant/pythia-12b-sft-v8-7k-steps` is a conversational chat model meaning we can chat with it using the following prompt:\n",
    "\n",
    "```\n",
    "<|prompter|>[Instruction]<|endoftext|>\n",
    "<|assistant|>\n",
    "```\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0307609-1823-4ad0-9c68-9bcdca642b65",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cell 07: Validate the model using a simple prompt\n",
    "\n",
    "chat = llm.predict(\n",
    "    {\n",
    "        \"inputs\": \"\"\"<|prompter|>what are federal reserve's responsibilities ?<|endoftext|><|assistant|>\"\"\"\n",
    "    }\n",
    ")\n",
    "\n",
    "print(chat[0][\"generated_text\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0133c234-2f9e-4edc-9638-75e595f622de",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cell 08:  Define payload and  run inference with different parameters to impact the generation. Parameters can be defined as in the parameters attribute of the payload\n",
    "\n",
    "\n",
    "prompt = \"\"\"<|prompter|>give me 3 business ideas that will help when federal reserve decreases interest rate<|endoftext|><|assistant|>\"\"\"\n",
    "\n",
    "# hyperparameters for llm\n",
    "payload = {\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\n",
    "        \"do_sample\": True,\n",
    "        \"top_p\": 0.7,\n",
    "        \"temperature\": 0.7,\n",
    "        \"top_k\": 50,\n",
    "        \"max_new_tokens\": 256,\n",
    "        \"repetition_penalty\": 1.03,\n",
    "        \"stop\": [\"<|endoftext|>\"],\n",
    "    },\n",
    "}\n",
    "\n",
    "# send request to endpoint\n",
    "response = llm.predict(payload)\n",
    "\n",
    "# print(response[0][\"generated_text\"][:-len(\"<human>:\")])\n",
    "\n",
    "print(response[0][\"generated_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54d1a5a0-7c3d-4518-b391-b87035edbc65",
   "metadata": {},
   "source": [
    "# Step-7: Create a chatbot dummy app using Gradio\n",
    "\n",
    "[Gradio](https://www.gradio.app/) helps to build and share demo applications quickly. Here we will use its rich functions to develop [chat applications](https://www.gradio.app/guides/creating-a-chatbot-fast) to demonstrate the chatbot. In this section, a dummy chatbot will be created to validate that the Gradio dependencies are imported and working fine. \n",
    "\n",
    "If the result shows a successful dummy chatbot app with `local URL` and `public URL`, proceed to `Step-8`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53ddf7b6-bb67-48cc-9ceb-e153acf93f3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cel 09 : Creating a dummy chat application to validate that the Gradio dependencies are imported and working fine !\n",
    "\n",
    "import gradio as gr\n",
    "import random\n",
    "\n",
    "\n",
    "def random_response(message, history):\n",
    "    return random.choice([\"Yes\", \"No\"])\n",
    "\n",
    "\n",
    "gr.ChatInterface(random_response).launch()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79cb4fb3-2629-4313-b0a1-1d03332b1ec8",
   "metadata": {},
   "source": [
    "# Step-8: Chatbot application \n",
    "\n",
    "Once the dummy chatbot application is working. Let's create a customized Chat App that will invoke the Hugging face chat LLM using the SageMaker inference end point that we deployed and validated using prompts in previous steps. \n",
    "\n",
    "In this chatbot application, Gradio’s [low-level Blocks APIs](https://www.gradio.app/guides/creating-a-custom-chatbot-with-blocks) are used.\n",
    "\n",
    "Successful Result will show and output with a responsive chatbot with `local URL` and `Public URL`\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98043fe5-faf9-4694-a341-5f1f0c9e796d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cell 10: Gradio Chatbot that invoke the chat LLM\n",
    "\n",
    "import gradio as gr\n",
    "\n",
    "# hyperparameters for llm\n",
    "parameters = {\n",
    "    \"do_sample\": True,\n",
    "    \"top_p\": 0.7,\n",
    "    \"temperature\": 0.7,\n",
    "    \"top_k\": 50,\n",
    "    \"max_new_tokens\": 256,\n",
    "    \"repetition_penalty\": 1.03,\n",
    "    \"stop\": [\"<|endoftext|>\"],\n",
    "}\n",
    "\n",
    "with gr.Blocks() as demo:\n",
    "    gr.Markdown(\"## Gradio ChatBot for HuggingFace LLM\")\n",
    "    with gr.Column():\n",
    "        chatbot = gr.Chatbot()\n",
    "        with gr.Row():\n",
    "            with gr.Column():\n",
    "                message = gr.Textbox(\n",
    "                    label=\"Chat Message Box\", placeholder=\"Chat Message Box\", show_label=False\n",
    "                )\n",
    "            with gr.Column():\n",
    "                with gr.Row():\n",
    "                    submit = gr.Button(\"Submit\")\n",
    "                    clear = gr.Button(\"Clear\")\n",
    "\n",
    "    def respond(message, chat_history):\n",
    "        # convert chat history to prompt\n",
    "        converted_chat_history = \"\"\n",
    "        if len(chat_history) > 0:\n",
    "            for c in chat_history:\n",
    "                converted_chat_history += (\n",
    "                    f\"<|prompter|>{c[0]}<|endoftext|><|assistant|>{c[1]}<|endoftext|>\"\n",
    "                )\n",
    "        prompt = f\"{converted_chat_history}<|prompter|>{message}<|endoftext|><|assistant|>\"\n",
    "\n",
    "        # send request to endpoint\n",
    "        llm_response = llm.predict({\"inputs\": prompt, \"parameters\": parameters})\n",
    "\n",
    "        # remove prompt from response\n",
    "        parsed_response = llm_response[0][\"generated_text\"][len(prompt) :]\n",
    "        chat_history.append((message, parsed_response))\n",
    "        return \"\", chat_history\n",
    "\n",
    "    submit.click(respond, [message, chatbot], [message, chatbot], queue=False)\n",
    "    clear.click(lambda: None, None, chatbot, queue=False)\n",
    "\n",
    "demo.launch(share=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cf954cb-78dd-40a4-86b3-8cdb639cf51e",
   "metadata": {},
   "source": [
    "# Step-9: Cleaning Up\n",
    "\n",
    "As a best practice and to avoid incurring costs, delete Amazon Sagemaker endpoints "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b72ab439-f0a0-41e9-8f32-e12ba36b93bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm.delete_model()\n",
    "llm.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd66f327-2269-4a99-a08f-9be90e79f462",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/inference|generativeai|huggingfacetgi|open-assistant|open-assistant-chatbot.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}