usage_komt_with_lora.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d78a7778-ee8d-4796-a82d-282c39133699",
   "metadata": {},
   "source": [
    "# inference komt model with lora"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a314283-484f-4124-9d90-b799fdcd4520",
   "metadata": {},
   "source": [
    "## Install Dependencies "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c2e3e4aa-39cc-4b86-934b-12894a420722",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting git+https://github.com/huggingface/peft.git (from -r requirements.txt (line 3))\n",
      "  Cloning https://github.com/huggingface/peft.git to /tmp/pip-req-build-jrotx6iw\n",
      "  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/peft.git /tmp/pip-req-build-jrotx6iw\n",
      "  Resolved https://github.com/huggingface/peft.git to commit 139624750a6a431b9958be5c9485aec5571d64c1\n",
      "  Installing build dependencies ... \u001b[?25ldone\n",
      "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n",
      "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting git+https://github.com/huggingface/accelerate.git (from -r requirements.txt (line 4))\n",
      "  Cloning https://github.com/huggingface/accelerate.git to /tmp/pip-req-build-nq5k1okx\n",
      "  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/accelerate.git /tmp/pip-req-build-nq5k1okx\n",
      "  Resolved https://github.com/huggingface/accelerate.git to commit 629d02c8446354860c9bdf58b6bc006186cbc818\n",
      "  Installing build dependencies ... \u001b[?25ldone\n",
      "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n",
      "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting git+https://github.com/huggingface/transformers.git (from -r requirements.txt (line 5))\n",
      "  Cloning https://github.com/huggingface/transformers.git to /tmp/pip-req-build-h1gtjup7\n",
      "  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers.git /tmp/pip-req-build-h1gtjup7\n",
      "  Resolved https://github.com/huggingface/transformers.git to commit eb8489971ac1415f67b0abdd1584fde8b659ced9\n",
      "  Installing build dependencies ... \u001b[?25ldone\n",
      "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n",
      "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: uvicorn in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from -r requirements.txt (line 1)) (0.23.2)\n",
      "Requirement already satisfied: bitsandbytes>=0.39.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from -r requirements.txt (line 2)) (0.41.1)\n",
      "Requirement already satisfied: datasets in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from -r requirements.txt (line 6)) (2.14.5)\n",
      "Requirement already satisfied: scipy in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from -r requirements.txt (line 7)) (1.11.2)\n",
      "Requirement already satisfied: numpy in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from -r requirements.txt (line 8)) (1.26.0)\n",
      "Requirement already satisfied: rouge_score in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from -r requirements.txt (line 9)) (0.1.2)\n",
      "Requirement already satisfied: fire in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from -r requirements.txt (line 10)) (0.5.0)\n",
      "Requirement already satisfied: openai in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from -r requirements.txt (line 11)) (0.28.0)\n",
      "Requirement already satisfied: sentencepiece in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from -r requirements.txt (line 12)) (0.1.99)\n",
      "Requirement already satisfied: wandb in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from -r requirements.txt (line 13)) (0.15.10)\n",
      "Requirement already satisfied: gradio==3.9 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from -r requirements.txt (line 14)) (3.9)\n",
      "Requirement already satisfied: deepspeed==0.9.2 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from -r requirements.txt (line 15)) (0.9.2)\n",
      "Requirement already satisfied: tensorboardX in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from -r requirements.txt (line 16)) (2.6.2.2)\n",
      "Requirement already satisfied: jupyter in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from -r requirements.txt (line 17)) (1.0.0)\n",
      "Requirement already satisfied: ipykernel in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from -r requirements.txt (line 18)) (6.25.2)\n",
      "Requirement already satisfied: aiohttp in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (3.8.5)\n",
      "Requirement already satisfied: fastapi in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (0.103.1)\n",
      "Requirement already satisfied: ffmpy in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (0.3.1)\n",
      "Requirement already satisfied: fsspec in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (2023.6.0)\n",
      "Requirement already satisfied: h11<0.13,>=0.11 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (0.12.0)\n",
      "Requirement already satisfied: httpx in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (0.24.1)\n",
      "Requirement already satisfied: jinja2 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (3.1.2)\n",
      "Requirement already satisfied: markdown-it-py[linkify,plugins] in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (3.0.0)\n",
      "Requirement already satisfied: matplotlib in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (3.8.0)\n",
      "Requirement already satisfied: orjson in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (3.9.7)\n",
      "Requirement already satisfied: pandas in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (2.1.0)\n",
      "Requirement already satisfied: paramiko in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (3.3.1)\n",
      "Requirement already satisfied: pillow in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (10.0.1)\n",
      "Requirement already satisfied: pycryptodome in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (3.19.0)\n",
      "Requirement already satisfied: pydantic in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (1.10.12)\n",
      "Requirement already satisfied: pydub in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (0.25.1)\n",
      "Requirement already satisfied: python-multipart in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (0.0.6)\n",
      "Requirement already satisfied: pyyaml in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (6.0.1)\n",
      "Requirement already satisfied: requests in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (2.31.0)\n",
      "Requirement already satisfied: websockets in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gradio==3.9->-r requirements.txt (line 14)) (11.0.3)\n",
      "Requirement already satisfied: hjson in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from deepspeed==0.9.2->-r requirements.txt (line 15)) (3.1.0)\n",
      "Requirement already satisfied: ninja in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from deepspeed==0.9.2->-r requirements.txt (line 15)) (1.11.1)\n",
      "Requirement already satisfied: packaging>=20.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from deepspeed==0.9.2->-r requirements.txt (line 15)) (23.1)\n",
      "Requirement already satisfied: psutil in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from deepspeed==0.9.2->-r requirements.txt (line 15)) (5.9.5)\n",
      "Requirement already satisfied: py-cpuinfo in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from deepspeed==0.9.2->-r requirements.txt (line 15)) (9.0.0)\n",
      "Requirement already satisfied: torch in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from deepspeed==0.9.2->-r requirements.txt (line 15)) (2.0.1)\n",
      "Requirement already satisfied: tqdm in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from deepspeed==0.9.2->-r requirements.txt (line 15)) (4.66.1)\n",
      "Requirement already satisfied: click>=7.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from uvicorn->-r requirements.txt (line 1)) (8.1.7)\n",
      "Requirement already satisfied: typing-extensions>=4.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from uvicorn->-r requirements.txt (line 1)) (4.8.0)\n",
      "Requirement already satisfied: safetensors in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from peft==0.6.0.dev0->-r requirements.txt (line 3)) (0.3.3)\n",
      "Requirement already satisfied: huggingface-hub in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from accelerate==0.24.0.dev0->-r requirements.txt (line 4)) (0.16.4)\n",
      "Requirement already satisfied: filelock in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from transformers==4.34.0.dev0->-r requirements.txt (line 5)) (3.12.4)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from transformers==4.34.0.dev0->-r requirements.txt (line 5)) (2023.8.8)\n",
      "Requirement already satisfied: tokenizers<0.15,>=0.14 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from transformers==4.34.0.dev0->-r requirements.txt (line 5)) (0.14.0)\n",
      "Requirement already satisfied: pyarrow>=8.0.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from datasets->-r requirements.txt (line 6)) (13.0.0)\n",
      "Requirement already satisfied: dill<0.3.8,>=0.3.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from datasets->-r requirements.txt (line 6)) (0.3.7)\n",
      "Requirement already satisfied: xxhash in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from datasets->-r requirements.txt (line 6)) (3.3.0)\n",
      "Requirement already satisfied: multiprocess in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from datasets->-r requirements.txt (line 6)) (0.70.15)\n",
      "Requirement already satisfied: absl-py in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from rouge_score->-r requirements.txt (line 9)) (1.4.0)\n",
      "Requirement already satisfied: nltk in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from rouge_score->-r requirements.txt (line 9)) (3.8.1)\n",
      "Requirement already satisfied: six>=1.14.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from rouge_score->-r requirements.txt (line 9)) (1.16.0)\n",
      "Requirement already satisfied: termcolor in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from fire->-r requirements.txt (line 10)) (2.3.0)\n",
      "Requirement already satisfied: GitPython!=3.1.29,>=1.0.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from wandb->-r requirements.txt (line 13)) (3.1.36)\n",
      "Requirement already satisfied: sentry-sdk>=1.0.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from wandb->-r requirements.txt (line 13)) (1.31.0)\n",
      "Requirement already satisfied: docker-pycreds>=0.4.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from wandb->-r requirements.txt (line 13)) (0.4.0)\n",
      "Requirement already satisfied: pathtools in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from wandb->-r requirements.txt (line 13)) (0.1.2)\n",
      "Requirement already satisfied: setproctitle in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from wandb->-r requirements.txt (line 13)) (1.3.2)\n",
      "Requirement already satisfied: setuptools in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from wandb->-r requirements.txt (line 13)) (68.0.0)\n",
      "Requirement already satisfied: appdirs>=1.4.3 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from wandb->-r requirements.txt (line 13)) (1.4.4)\n",
      "Requirement already satisfied: protobuf!=4.21.0,<5,>=3.19.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from wandb->-r requirements.txt (line 13)) (4.24.3)\n",
      "Requirement already satisfied: notebook in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter->-r requirements.txt (line 17)) (7.0.3)\n",
      "Requirement already satisfied: qtconsole in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter->-r requirements.txt (line 17)) (5.4.4)\n",
      "Requirement already satisfied: jupyter-console in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter->-r requirements.txt (line 17)) (6.6.3)\n",
      "Requirement already satisfied: nbconvert in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter->-r requirements.txt (line 17)) (7.8.0)\n",
      "Requirement already satisfied: ipywidgets in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter->-r requirements.txt (line 17)) (8.1.1)\n",
      "Requirement already satisfied: comm>=0.1.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipykernel->-r requirements.txt (line 18)) (0.1.4)\n",
      "Requirement already satisfied: debugpy>=1.6.5 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipykernel->-r requirements.txt (line 18)) (1.8.0)\n",
      "Requirement already satisfied: ipython>=7.23.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipykernel->-r requirements.txt (line 18)) (8.15.0)\n",
      "Requirement already satisfied: jupyter-client>=6.1.12 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipykernel->-r requirements.txt (line 18)) (8.3.1)\n",
      "Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipykernel->-r requirements.txt (line 18)) (5.3.1)\n",
      "Requirement already satisfied: matplotlib-inline>=0.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipykernel->-r requirements.txt (line 18)) (0.1.6)\n",
      "Requirement already satisfied: nest-asyncio in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipykernel->-r requirements.txt (line 18)) (1.5.8)\n",
      "Requirement already satisfied: pyzmq>=20 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipykernel->-r requirements.txt (line 18)) (25.1.1)\n",
      "Requirement already satisfied: tornado>=6.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipykernel->-r requirements.txt (line 18)) (6.3.3)\n",
      "Requirement already satisfied: traitlets>=5.4.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipykernel->-r requirements.txt (line 18)) (5.10.0)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from aiohttp->gradio==3.9->-r requirements.txt (line 14)) (23.1.0)\n",
      "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from aiohttp->gradio==3.9->-r requirements.txt (line 14)) (3.2.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from aiohttp->gradio==3.9->-r requirements.txt (line 14)) (6.0.4)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from aiohttp->gradio==3.9->-r requirements.txt (line 14)) (4.0.3)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from aiohttp->gradio==3.9->-r requirements.txt (line 14)) (1.9.2)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from aiohttp->gradio==3.9->-r requirements.txt (line 14)) (1.4.0)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from aiohttp->gradio==3.9->-r requirements.txt (line 14)) (1.3.1)\n",
      "Requirement already satisfied: gitdb<5,>=4.0.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from GitPython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 13)) (4.0.10)\n",
      "Requirement already satisfied: backcall in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipython>=7.23.1->ipykernel->-r requirements.txt (line 18)) (0.2.0)\n",
      "Requirement already satisfied: decorator in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipython>=7.23.1->ipykernel->-r requirements.txt (line 18)) (5.1.1)\n",
      "Requirement already satisfied: jedi>=0.16 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipython>=7.23.1->ipykernel->-r requirements.txt (line 18)) (0.19.0)\n",
      "Requirement already satisfied: pickleshare in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipython>=7.23.1->ipykernel->-r requirements.txt (line 18)) (0.7.5)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipython>=7.23.1->ipykernel->-r requirements.txt (line 18)) (3.0.39)\n",
      "Requirement already satisfied: pygments>=2.4.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipython>=7.23.1->ipykernel->-r requirements.txt (line 18)) (2.16.1)\n",
      "Requirement already satisfied: stack-data in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipython>=7.23.1->ipykernel->-r requirements.txt (line 18)) (0.6.2)\n",
      "Requirement already satisfied: exceptiongroup in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipython>=7.23.1->ipykernel->-r requirements.txt (line 18)) (1.1.3)\n",
      "Requirement already satisfied: pexpect>4.3 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipython>=7.23.1->ipykernel->-r requirements.txt (line 18)) (4.8.0)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter-client>=6.1.12->ipykernel->-r requirements.txt (line 18)) (2.8.2)\n",
      "Requirement already satisfied: platformdirs>=2.5 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter-core!=5.0.*,>=4.12->ipykernel->-r requirements.txt (line 18)) (3.10.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from requests->gradio==3.9->-r requirements.txt (line 14)) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from requests->gradio==3.9->-r requirements.txt (line 14)) (2.0.4)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from requests->gradio==3.9->-r requirements.txt (line 14)) (2023.7.22)\n",
      "Requirement already satisfied: sympy in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (1.12)\n",
      "Requirement already satisfied: networkx in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (3.1)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (11.7.101)\n",
      "Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (8.5.0.96)\n",
      "Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (11.10.3.66)\n",
      "Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (10.9.0.58)\n",
      "Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (10.2.10.91)\n",
      "Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (11.4.0.1)\n",
      "Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (11.7.4.91)\n",
      "Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (2.14.3)\n",
      "Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (11.7.91)\n",
      "Requirement already satisfied: triton==2.0.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (2.0.0)\n",
      "Requirement already satisfied: wheel in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (0.38.4)\n",
      "Requirement already satisfied: cmake in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from triton==2.0.0->torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (3.27.5)\n",
      "Requirement already satisfied: lit in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from triton==2.0.0->torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (16.0.6)\n",
      "Requirement already satisfied: anyio<4.0.0,>=3.7.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from fastapi->gradio==3.9->-r requirements.txt (line 14)) (3.7.1)\n",
      "Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from fastapi->gradio==3.9->-r requirements.txt (line 14)) (0.27.0)\n",
      "Requirement already satisfied: httpcore<0.18.0,>=0.15.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from httpx->gradio==3.9->-r requirements.txt (line 14)) (0.15.0)\n",
      "Requirement already satisfied: sniffio in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from httpx->gradio==3.9->-r requirements.txt (line 14)) (1.3.0)\n",
      "Requirement already satisfied: widgetsnbextension~=4.0.9 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipywidgets->jupyter->-r requirements.txt (line 17)) (4.0.9)\n",
      "Requirement already satisfied: jupyterlab-widgets~=3.0.9 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from ipywidgets->jupyter->-r requirements.txt (line 17)) (3.0.9)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jinja2->gradio==3.9->-r requirements.txt (line 14)) (2.1.3)\n",
      "Requirement already satisfied: mdurl~=0.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from markdown-it-py[linkify,plugins]->gradio==3.9->-r requirements.txt (line 14)) (0.1.2)\n",
      "Requirement already satisfied: mdit-py-plugins in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from markdown-it-py[linkify,plugins]->gradio==3.9->-r requirements.txt (line 14)) (0.4.0)\n",
      "Requirement already satisfied: linkify-it-py<3,>=1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from markdown-it-py[linkify,plugins]->gradio==3.9->-r requirements.txt (line 14)) (2.0.2)\n",
      "Requirement already satisfied: contourpy>=1.0.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from matplotlib->gradio==3.9->-r requirements.txt (line 14)) (1.1.1)\n",
      "Requirement already satisfied: cycler>=0.10 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from matplotlib->gradio==3.9->-r requirements.txt (line 14)) (0.11.0)\n",
      "Requirement already satisfied: fonttools>=4.22.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from matplotlib->gradio==3.9->-r requirements.txt (line 14)) (4.42.1)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from matplotlib->gradio==3.9->-r requirements.txt (line 14)) (1.4.5)\n",
      "Requirement already satisfied: pyparsing>=2.3.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from matplotlib->gradio==3.9->-r requirements.txt (line 14)) (3.1.1)\n",
      "Requirement already satisfied: beautifulsoup4 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from nbconvert->jupyter->-r requirements.txt (line 17)) (4.12.2)\n",
      "Requirement already satisfied: bleach!=5.0.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from nbconvert->jupyter->-r requirements.txt (line 17)) (6.0.0)\n",
      "Requirement already satisfied: defusedxml in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from nbconvert->jupyter->-r requirements.txt (line 17)) (0.7.1)\n",
      "Requirement already satisfied: jupyterlab-pygments in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from nbconvert->jupyter->-r requirements.txt (line 17)) (0.2.2)\n",
      "Requirement already satisfied: mistune<4,>=2.0.3 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from nbconvert->jupyter->-r requirements.txt (line 17)) (3.0.1)\n",
      "Requirement already satisfied: nbclient>=0.5.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from nbconvert->jupyter->-r requirements.txt (line 17)) (0.8.0)\n",
      "Requirement already satisfied: nbformat>=5.7 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from nbconvert->jupyter->-r requirements.txt (line 17)) (5.9.2)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from nbconvert->jupyter->-r requirements.txt (line 17)) (1.5.0)\n",
      "Requirement already satisfied: tinycss2 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from nbconvert->jupyter->-r requirements.txt (line 17)) (1.2.1)\n",
      "Requirement already satisfied: joblib in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from nltk->rouge_score->-r requirements.txt (line 9)) (1.3.2)\n",
      "Requirement already satisfied: jupyter-server<3,>=2.4.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from notebook->jupyter->-r requirements.txt (line 17)) (2.7.3)\n",
      "Requirement already satisfied: jupyterlab-server<3,>=2.22.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from notebook->jupyter->-r requirements.txt (line 17)) (2.25.0)\n",
      "Requirement already satisfied: jupyterlab<5,>=4.0.2 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from notebook->jupyter->-r requirements.txt (line 17)) (4.0.6)\n",
      "Requirement already satisfied: notebook-shim<0.3,>=0.2 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from notebook->jupyter->-r requirements.txt (line 17)) (0.2.3)\n",
      "Requirement already satisfied: pytz>=2020.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from pandas->gradio==3.9->-r requirements.txt (line 14)) (2023.3.post1)\n",
      "Requirement already satisfied: tzdata>=2022.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from pandas->gradio==3.9->-r requirements.txt (line 14)) (2023.3)\n",
      "Requirement already satisfied: bcrypt>=3.2 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from paramiko->gradio==3.9->-r requirements.txt (line 14)) (4.0.1)\n",
      "Requirement already satisfied: cryptography>=3.3 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from paramiko->gradio==3.9->-r requirements.txt (line 14)) (41.0.3)\n",
      "Requirement already satisfied: pynacl>=1.5 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from paramiko->gradio==3.9->-r requirements.txt (line 14)) (1.5.0)\n",
      "Requirement already satisfied: ipython-genutils in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from qtconsole->jupyter->-r requirements.txt (line 17)) (0.2.0)\n",
      "Requirement already satisfied: qtpy>=2.4.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from qtconsole->jupyter->-r requirements.txt (line 17)) (2.4.0)\n",
      "Requirement already satisfied: webencodings in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from bleach!=5.0.0->nbconvert->jupyter->-r requirements.txt (line 17)) (0.5.1)\n",
      "Requirement already satisfied: cffi>=1.12 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from cryptography>=3.3->paramiko->gradio==3.9->-r requirements.txt (line 14)) (1.15.1)\n",
      "Requirement already satisfied: smmap<6,>=3.0.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from gitdb<5,>=4.0.1->GitPython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 13)) (5.0.1)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jedi>=0.16->ipython>=7.23.1->ipykernel->-r requirements.txt (line 18)) (0.8.3)\n",
      "Requirement already satisfied: argon2-cffi in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->notebook->jupyter->-r requirements.txt (line 17)) (23.1.0)\n",
      "Requirement already satisfied: jupyter-events>=0.6.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->notebook->jupyter->-r requirements.txt (line 17)) (0.7.0)\n",
      "Requirement already satisfied: jupyter-server-terminals in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->notebook->jupyter->-r requirements.txt (line 17)) (0.4.4)\n",
      "Requirement already satisfied: overrides in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->notebook->jupyter->-r requirements.txt (line 17)) (7.4.0)\n",
      "Requirement already satisfied: prometheus-client in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->notebook->jupyter->-r requirements.txt (line 17)) (0.17.1)\n",
      "Requirement already satisfied: send2trash>=1.8.2 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->notebook->jupyter->-r requirements.txt (line 17)) (1.8.2)\n",
      "Requirement already satisfied: terminado>=0.8.3 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->notebook->jupyter->-r requirements.txt (line 17)) (0.17.1)\n",
      "Requirement already satisfied: websocket-client in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->notebook->jupyter->-r requirements.txt (line 17)) (1.6.3)\n",
      "Requirement already satisfied: async-lru>=1.0.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyterlab<5,>=4.0.2->notebook->jupyter->-r requirements.txt (line 17)) (2.0.4)\n",
      "Requirement already satisfied: jupyter-lsp>=2.0.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyterlab<5,>=4.0.2->notebook->jupyter->-r requirements.txt (line 17)) (2.2.0)\n",
      "Requirement already satisfied: tomli in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyterlab<5,>=4.0.2->notebook->jupyter->-r requirements.txt (line 17)) (2.0.1)\n",
      "Requirement already satisfied: babel>=2.10 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyterlab-server<3,>=2.22.1->notebook->jupyter->-r requirements.txt (line 17)) (2.12.1)\n",
      "Requirement already satisfied: json5>=0.9.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyterlab-server<3,>=2.22.1->notebook->jupyter->-r requirements.txt (line 17)) (0.9.14)\n",
      "Requirement already satisfied: jsonschema>=4.18.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyterlab-server<3,>=2.22.1->notebook->jupyter->-r requirements.txt (line 17)) (4.19.0)\n",
      "Requirement already satisfied: uc-micro-py in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from linkify-it-py<3,>=1->markdown-it-py[linkify,plugins]->gradio==3.9->-r requirements.txt (line 14)) (1.0.2)\n",
      "Requirement already satisfied: fastjsonschema in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from nbformat>=5.7->nbconvert->jupyter->-r requirements.txt (line 17)) (2.18.0)\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from pexpect>4.3->ipython>=7.23.1->ipykernel->-r requirements.txt (line 18)) (0.7.0)\n",
      "Requirement already satisfied: wcwidth in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30->ipython>=7.23.1->ipykernel->-r requirements.txt (line 18)) (0.2.6)\n",
      "Requirement already satisfied: soupsieve>1.2 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from beautifulsoup4->nbconvert->jupyter->-r requirements.txt (line 17)) (2.5)\n",
      "Requirement already satisfied: executing>=1.2.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from stack-data->ipython>=7.23.1->ipykernel->-r requirements.txt (line 18)) (1.2.0)\n",
      "Requirement already satisfied: asttokens>=2.1.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from stack-data->ipython>=7.23.1->ipykernel->-r requirements.txt (line 18)) (2.4.0)\n",
      "Requirement already satisfied: pure-eval in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from stack-data->ipython>=7.23.1->ipykernel->-r requirements.txt (line 18)) (0.2.2)\n",
      "Requirement already satisfied: mpmath>=0.19 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from sympy->torch->deepspeed==0.9.2->-r requirements.txt (line 15)) (1.3.0)\n",
      "Requirement already satisfied: pycparser in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=3.3->paramiko->gradio==3.9->-r requirements.txt (line 14)) (2.21)\n",
      "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.22.1->notebook->jupyter->-r requirements.txt (line 17)) (2023.7.1)\n",
      "Requirement already satisfied: referencing>=0.28.4 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.22.1->notebook->jupyter->-r requirements.txt (line 17)) (0.30.2)\n",
      "Requirement already satisfied: rpds-py>=0.7.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.22.1->notebook->jupyter->-r requirements.txt (line 17)) (0.10.3)\n",
      "Requirement already satisfied: python-json-logger>=2.0.4 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter-events>=0.6.0->jupyter-server<3,>=2.4.0->notebook->jupyter->-r requirements.txt (line 17)) (2.0.7)\n",
      "Requirement already satisfied: rfc3339-validator in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter-events>=0.6.0->jupyter-server<3,>=2.4.0->notebook->jupyter->-r requirements.txt (line 17)) (0.1.4)\n",
      "Requirement already satisfied: rfc3986-validator>=0.1.1 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jupyter-events>=0.6.0->jupyter-server<3,>=2.4.0->notebook->jupyter->-r requirements.txt (line 17)) (0.1.1)\n",
      "Requirement already satisfied: argon2-cffi-bindings in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from argon2-cffi->jupyter-server<3,>=2.4.0->notebook->jupyter->-r requirements.txt (line 17)) (21.2.0)\n",
      "Requirement already satisfied: fqdn in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.22.1->notebook->jupyter->-r requirements.txt (line 17)) (1.5.1)\n",
      "Requirement already satisfied: isoduration in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.22.1->notebook->jupyter->-r requirements.txt (line 17)) (20.11.0)\n",
      "Requirement already satisfied: jsonpointer>1.13 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.22.1->notebook->jupyter->-r requirements.txt (line 17)) (2.4)\n",
      "Requirement already satisfied: uri-template in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.22.1->notebook->jupyter->-r requirements.txt (line 17)) (1.3.0)\n",
      "Requirement already satisfied: webcolors>=1.11 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.22.1->notebook->jupyter->-r requirements.txt (line 17)) (1.13)\n",
      "Requirement already satisfied: arrow>=0.15.0 in /home/david/anaconda3/envs/komt/lib/python3.10/site-packages (from isoduration->jsonschema>=4.18.0->jupyterlab-server<3,>=2.22.1->notebook->jupyter->-r requirements.txt (line 17)) (1.2.3)\n"
     ]
    }
   ],
   "source": [
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be04d8d3-5163-4a8e-8f37-f8b21a4d004a",
   "metadata": {},
   "source": [
    "## load model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d08649fe-cf49-4e61-9af1-ac95fe7f25d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n",
    "from peft import PeftModel, PeftConfig\n",
    "from transformers import TextStreamer, GenerationConfig\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b0c92da3-1b8d-4151-a5f7-638d0c6c6263",
   "metadata": {},
   "outputs": [],
   "source": [
    "model='davidkim205/komt-llama2-7b-v1'\n",
    "peft_model_name = 'davidkim205/komt-llama2-7b-v1-lora'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a6fb322c-880a-4c20-8008-ada43b58dec1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading the tokenizer from the `special_tokens_map.json` and the `added_tokens.json` will be removed in `transformers 5`,  it is kept for forward compatibility, but it is recommended to update your `tokenizer_config.json` by uploading it again. You will see the new `added_tokens_decoder` attribute that will store the relevant information.\n",
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n",
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    }
   ],
   "source": [
    "config = PeftConfig.from_pretrained(peft_model_name)\n",
    "bnb_config = BitsAndBytesConfig(\n",
    "    load_in_4bit=True,\n",
    "    bnb_4bit_use_double_quant=True,\n",
    "    bnb_4bit_quant_type=\"nf4\",\n",
    "    bnb_4bit_compute_dtype=torch.bfloat16\n",
    ")\n",
    "config.base_model_name_or_path =model\n",
    "model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, quantization_config=bnb_config, device_map=\"auto\")\n",
    "model = PeftModel.from_pretrained(model, peft_model_name)\n",
    "tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)\n",
    "streamer = TextStreamer(tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9a1847ff-509b-4ad5-b127-ff880704ab4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def gen(x):\n",
    "    generation_config = GenerationConfig(\n",
    "        temperature=0.8,\n",
    "        top_p=0.8,\n",
    "        top_k=100,\n",
    "        max_new_tokens=512,\n",
    "        early_stopping=True,\n",
    "        do_sample=True,\n",
    "    )\n",
    "    q = f\"### instruction: {x}\\n\\n### Response: \"\n",
    "    gened = model.generate(\n",
    "        **tokenizer(\n",
    "            q,\n",
    "            return_tensors='pt',\n",
    "            return_token_type_ids=False\n",
    "        ).to('cuda'),\n",
    "        generation_config=generation_config,\n",
    "        pad_token_id=tokenizer.eos_token_id,\n",
    "        eos_token_id=tokenizer.eos_token_id,\n",
    "        streamer=streamer,\n",
    "    )\n",
    "    result_str = tokenizer.decode(gened[0])\n",
    "\n",
    "    start_tag = f\"\\n\\n### Response: \"\n",
    "    start_index = result_str.find(start_tag)\n",
    "\n",
    "    if start_index != -1:\n",
    "        result_str = result_str[start_index + len(start_tag):].strip()\n",
    "    return result_str\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c77eba08-9e02-441f-9a63-9c59041c9b1a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "</s> ### instruction: 넷플릭스는 어떤 시청 문화를 확산시켰나요?\n",
      "\n",
      "### Response: "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/david/anaconda3/envs/komt/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:399: UserWarning: `num_beams` is set to 1. However, `early_stopping` is set to `True` -- this flag is only used in beam-based generation modes. You should set `num_beams>1` or unset `early_stopping`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.\n",
      "  warnings.warn(\n",
      "/home/david/anaconda3/envs/komt/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:399: UserWarning: `num_beams` is set to 1. However, `early_stopping` is set to `True` -- this flag is only used in beam-based generation modes. You should set `num_beams>1` or unset `early_stopping`.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "넷플릭스는 매주 1~2편을 공개했던 TV 드라마에 맞서 한번에 전편을 몰아볼 수 있는 '몰아보기' 시청 문화를 확산시켰습니다.</s>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"넷플릭스는 매주 1~2편을 공개했던 TV 드라마에 맞서 한번에 전편을 몰아볼 수 있는 '몰아보기' 시청 문화를 확산시켰습니다.</s>\""
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gen('넷플릭스는 어떤 시청 문화를 확산시켰나요?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f01cafa0-b082-4017-aace-0ec49d625194",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "</s> ### instruction: 잠을 잘자는 방법 알려줘\n",
      "\n",
      "### Response: 잠을 잘 자는 방법을 알려드리겠습니다.\n",
      "\n",
      "1. 매일 일정한 시간에 규칙적으로 일어나세요.\n",
      "2. 카페인, 니코틴, 알코올과 같이 중추신경계에 작용하는 약물을 섭취하지 않으세요.\n",
      "3. 매일 같은 시간에 잠자리에 들기\n",
      "4. 잠자리에 잠든 후 잠을 잘 때는 밝은 환경을 추천합니다.\n",
      "5. 잠자리는 편안하고 쾌적한 환경이어야 합니다.\n",
      "6. 잠자리에서 편안하게 느낄 수 있는 방법으로 침대에 누워 자거나 편안한 자세를 취하세요.\n",
      "7. 잠자리에 들기 1시간 전에 전자기기 사용을 자제하세요.\n",
      "8. 잠자리에 들기 1시간 전에 밝은 음악, 밝은 빛 등 자극을 제거하세요.\n",
      "9. 잠자리에 들기 30분 전에 따뜻한 온도의 운동을 하세요.\n",
      "10. \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'잠을 잘 자는 방법을 알려드리겠습니다.\\n\\n1. 매일 일정한 시간에 규칙적으로 일어나세요.\\n2. 카페인, 니코틴, 알코올과 같이 중추신경계에 작용하는 약물을 섭취하지 않으세요.\\n3. 매일 같은 시간에 잠자리에 들기\\n4. 잠자리에 잠든 후 잠을 잘 때는 밝은 환경을 추천합니다.\\n5. 잠자리는 편안하고 쾌적한 환경이어야 합니다.\\n6. 잠자리에서 편안하게 느낄 수 있는 방법으로 침대에 누워 자거나 편안한 자세를 취하세요.\\n7. 잠자리에 들기 1시간 전에 전자기기 사용을 자제하세요.\\n8. 잠자리에 들기 1시간 전에 밝은 음악, 밝은 빛 등 자극을 제거하세요.\\n9. 잠자리에 들기 30분 전에 따뜻한 온도의 운동을 하세요.\\n10.'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gen('잠을 잘자는 방법 알려줘')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "komt",
   "language": "python",
   "name": "komt"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}