notebooks/varco_model_13_IST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy Varco LLM Model 13B IST Package from AWS Marketplace \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "VARCO-LLM is NCSOFT’s large language model, which can be applied to develop various NLP-based AI services such as Q&A, chatbot, summarization, information extraction etc. VACRO-LLM, trained with public pre-training data and internally constructed high-quality Korean data, boasts the highest performance among the Korean LLMs of similar sizes that have been released to date (see https://ncsoft.github.io/ncresearch/ for evaluation results). Our models will continue to be updated and we will also release LLMs that support multiple languages or are fined-tuned to specific tasks. As VARCO-LLM is currently in beta service (15 Aug to 10 Sep 2023), usage fees will not be charged temporally for this period. For inquiries regarding further performance improvement or collaboration for service applications, please contact us via email (varco_llm@ncsoft.com).\n",
    "\n",
    "This sample notebook shows you how to deploy [varco llm](https://aws.amazon.com/marketplace/pp/prodview-aout3755izrye)  using Amazon SageMaker.\n",
    "\n",
    "> **Note**: This is a reference notebook and it cannot run unless you make changes suggested in the notebook.\n",
    "\n",
    "## Pre-requisites:\n",
    "1. **Note**: This notebook contains elements which render correctly in Jupyter interface. Open this notebook from an Amazon SageMaker Notebook Instance or Amazon SageMaker Studio.\n",
    "1. Ensure that IAM role used has **AmazonSageMakerFullAccess**\n",
    "1. To deploy this ML model successfully, ensure that:\n",
    "    1. Either your IAM role has these three permissions and you have authority to make AWS Marketplace subscriptions in the AWS account used: \n",
    "        1. **aws-marketplace:ViewSubscriptions**\n",
    "        1. **aws-marketplace:Unsubscribe**\n",
    "        1. **aws-marketplace:Subscribe**  \n",
    "\n",
    "## Contents:\n",
    "1. [Subscribe to the model package](#1.-Subscribe-to-the-model-package)\n",
    "2. [Create an endpoint and perform real-time inference](#2.-Create-an-endpoint-and-perform-real-time-inference)\n",
    "   1. [Create an endpoint](#A.-Create-an-endpoint)\n",
    "   2. [Create input payload](#B.-Create-input-payload)\n",
    "   3. [Perform real-time inference](#C.-Perform-real-time-inference)\n",
    "   4. [Visualize output](#D.-Visualize-output)\n",
    "   5. [Delete the endpoint](#E.-Delete-the-endpoint)\n",
    "3. [Perform batch inference](#3.-Perform-batch-inference) \n",
    "4. [Clean-up](#4.-Clean-up)\n",
    "    1. [Delete the model](#A.-Delete-the-model)\n",
    "    2. [Unsubscribe to the listing (optional)](#B.-Unsubscribe-to-the-listing-(optional))\n",
    "    \n",
    "\n",
    "## Usage instructions\n",
    "You can run this notebook one cell at a time (By using Shift+Enter for running a cell)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Subscribe to the model package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "To subscribe to the model package:\n",
    "1. Open the model package [listing page](https://aws.amazon.com/marketplace/pp/prodview-m3sqbtxwpacom?sr=0-5&ref_=beagle&applicationId=AWSMPContessa)\n",
    "1. On the AWS Marketplace listing, click on the **Continue to subscribe** button.\n",
    "1. On the **Subscribe to this software** page, review and click on **\"Accept Offer\"** if you and your organization agrees with EULA, pricing, and support terms. \n",
    "1. Once you click on **Continue to configuration button** and then choose a **region**, you will see a **Product Arn** displayed. This is the model package ARN that you need to specify while creating a deployable model using Boto3. Copy the ARN corresponding to your region and specify the same in the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_package_arn = \"arn:aws:sagemaker:us-west-2:594846645681:model-package/ncsoft-ko-varco-13b-ist-final-dbf0cc4de8bf39e8926e3aa10e15a7fd\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import base64\n",
    "import json\n",
    "import uuid\n",
    "from sagemaker import ModelPackage\n",
    "import sagemaker as sage\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker import ModelPackage\n",
    "import boto3\n",
    "from IPython.display import Image\n",
    "from PIL import Image as ImageEdit\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "role = get_execution_role()\n",
    "\n",
    "sagemaker_session = sage.Session()\n",
    "\n",
    "bucket = sagemaker_session.default_bucket()\n",
    "runtime = boto3.client(\"runtime.sagemaker\")\n",
    "bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Create an endpoint and perform real-time inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to understand how real-time inference with Amazon SageMaker works, see [Documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-hosting.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_name = \"varco-llm-13b-ist\"\n",
    "\n",
    "content_type = \"application/json\"\n",
    "\n",
    "real_time_inference_instance_type = (\n",
    "    \"ml.g4dn.12xlarge\"\n",
    ")\n",
    "batch_transform_inference_instance_type = (\n",
    "    \"ml.g4dn.12xlarge\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A. Create an endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# create a deployable model from the model package.\n",
    "model = ModelPackage(\n",
    "    role=role, model_package_arn=model_package_arn, sagemaker_session=sagemaker_session\n",
    ")\n",
    "\n",
    "# Deploy the model\n",
    "predictor = model.deploy(1, real_time_inference_instance_type, endpoint_name=model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once endpoint has been created, you would be able to perform real-time inference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### B. Create input payload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "input = {\"text\":\"우주의 탄생 원리를 알려줘.\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### C. Perform real-time inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "response = runtime.invoke_endpoint(\n",
    "    EndpointName=model_name,\n",
    "    ContentType=content_type,\n",
    "    Accept=\"application/json\",\n",
    "    Body=json.dumps(input)\n",
    ")\n",
    "\n",
    "json.load(response[\"Body\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### D. Delete the endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have successfully performed a real-time inference, you do not need the endpoint any more. You can terminate the endpoint to avoid being charged."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.sagemaker_session.delete_endpoint(model_name)\n",
    "model.sagemaker_session.delete_endpoint_config(model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Perform batch inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, you will perform batch inference using multiple input payloads together. If you are not familiar with batch transform, and want to learn more, see this link:\n",
    "1. [Batch trasfrom job documents](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# upload the batch-transform job input files to S3\n",
    "transform_input_folder = \"data/input/batch\"\n",
    "transform_input = sagemaker_session.upload_data(transform_input_folder, key_prefix=model_name)\n",
    "print(\"Transform input uploaded to \" + transform_input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the batch-transform job\n",
    "transformer = model.transformer(1, batch_transform_inference_instance_type)\n",
    "transformer.transform(transform_input, content_type=content_type)\n",
    "transformer.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# output is available on following path\n",
    "transformer.output_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Clean-up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A. Delete the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.delete_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### B. Unsubscribe to the listing (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to unsubscribe to the model package, follow these steps. Before you cancel the subscription, ensure that you do not have any [deployable model](https://console.aws.amazon.com/sagemaker/home#/models) created from the model package or using the algorithm. Note - You can find this information by looking at the container name associated with the model. \n",
    "\n",
    "**Steps to unsubscribe to product from AWS Marketplace**:\n",
    "1. Navigate to __Machine Learning__ tab on [__Your Software subscriptions page__](https://aws.amazon.com/marketplace/ai/library?productType=ml&ref_=mlmp_gitdemo_indust)\n",
    "2. Locate the listing that you want to cancel the subscription for, and then choose __Cancel Subscription__  to cancel the subscription.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}