Llama_2_13B_chat.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Troyanovsky/Local-LLM-comparison/blob/main/Llama_2_13B_chat.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XNkpBtLvuTOp"
      },
      "source": [
        "## Llama-2-13B-chat WebUI\n",
        "1. Run the following cell, takes ~5 min\n",
        "2. Click the gradio link at the bottom\n",
        "3. In Chat settings - Instruction Template: Llama-2-Chat\n",
        "```\n",
        "SYSTEM: You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\n",
        "USER: {prompt}\n",
        "ASSISTANT:\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "D-MiHp_bveP6",
        "outputId": "ce290ef8-9899-4055-d867-00709479d8b5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content\n",
            "Selecting previously unselected package libc-ares2:amd64.\n",
            "(Reading database ... 120509 files and directories currently installed.)\n",
            "Preparing to unpack .../libc-ares2_1.18.1-1ubuntu0.22.04.2_amd64.deb ...\n",
            "Unpacking libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.2) ...\n",
            "Selecting previously unselected package libaria2-0:amd64.\n",
            "Preparing to unpack .../libaria2-0_1.36.0-1_amd64.deb ...\n",
            "Unpacking libaria2-0:amd64 (1.36.0-1) ...\n",
            "Selecting previously unselected package aria2.\n",
            "Preparing to unpack .../aria2_1.36.0-1_amd64.deb ...\n",
            "Unpacking aria2 (1.36.0-1) ...\n",
            "Setting up libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.2) ...\n",
            "Setting up libaria2-0:amd64 (1.36.0-1) ...\n",
            "Setting up aria2 (1.36.0-1) ...\n",
            "Processing triggers for man-db (2.10.2-1) ...\n",
            "Processing triggers for libc-bin (2.35-0ubuntu3.1) ...\n",
            "Cloning into 'text-generation-webui'...\n",
            "remote: Enumerating objects: 9462, done.\u001b[K\n",
            "remote: Counting objects: 100% (242/242), done.\u001b[K\n",
            "remote: Compressing objects: 100% (128/128), done.\u001b[K\n",
            "remote: Total 9462 (delta 130), reused 194 (delta 114), pack-reused 9220\u001b[K\n",
            "Receiving objects: 100% (9462/9462), 3.28 MiB | 15.56 MiB/s, done.\n",
            "Resolving deltas: 100% (6307/6307), done.\n",
            "Note: switching to '87926d033d9164a8fc77874bd6eaec72f7bb5b8a'.\n",
            "\n",
            "You are in 'detached HEAD' state. You can look around, make experimental\n",
            "changes and commit them, and you can discard any commits you make in this\n",
            "state without impacting any branches by switching back to a branch.\n",
            "\n",
            "If you want to create a new branch to retain commits you create, you may\n",
            "do so (now or later) by using -c with the switch command. Example:\n",
            "\n",
            "  git switch -c <new-branch-name>\n",
            "\n",
            "Or undo this operation with:\n",
            "\n",
            "  git switch -\n",
            "\n",
            "Turn off this advice by setting config variable advice.detachedHead to false\n",
            "\n",
            "/content/text-generation-webui\n",
            "Collecting git+https://github.com/huggingface/peft@03eb378eb914fbee709ff7c86ba5b1d033b89524 (from -r requirements.txt (line 22))\n",
            "  Cloning https://github.com/huggingface/peft (to revision 03eb378eb914fbee709ff7c86ba5b1d033b89524) to /tmp/pip-req-build-__fb9suq\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/peft /tmp/pip-req-build-__fb9suq\n",
            "  Running command git rev-parse -q --verify 'sha^03eb378eb914fbee709ff7c86ba5b1d033b89524'\n",
            "  Running command git fetch -q https://github.com/huggingface/peft 03eb378eb914fbee709ff7c86ba5b1d033b89524\n",
            "  Running command git checkout -q 03eb378eb914fbee709ff7c86ba5b1d033b89524\n",
            "  Resolved https://github.com/huggingface/peft to commit 03eb378eb914fbee709ff7c86ba5b1d033b89524\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Ignoring bitsandbytes: markers 'platform_system == \"Windows\"' don't match your environment\n",
            "Ignoring llama-cpp-python: markers 'platform_system == \"Windows\"' don't match your environment\n",
            "Ignoring auto-gptq: markers 'platform_system == \"Windows\"' don't match your environment\n",
            "Collecting auto-gptq==0.3.0+cu117 (from -r requirements.txt (line 28))\n",
            "  Downloading https://github.com/PanQiWei/AutoGPTQ/releases/download/v0.3.0/auto_gptq-0.3.0+cu117-cp310-cp310-linux_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m11.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring exllama: markers 'platform_system == \"Windows\"' don't match your environment\n",
            "Collecting exllama==0.0.7+cu117 (from -r requirements.txt (line 30))\n",
            "  Downloading https://github.com/jllllll/exllama/releases/download/0.0.7/exllama-0.0.7+cu117-cp310-cp310-linux_x86_64.whl (353 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m353.7/353.7 kB\u001b[0m \u001b[31m17.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting accelerate==0.21.0 (from -r requirements.txt (line 1))\n",
            "  Downloading accelerate-0.21.0-py3-none-any.whl (244 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m244.2/244.2 kB\u001b[0m \u001b[31m6.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting colorama (from -r requirements.txt (line 2))\n",
            "  Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n",
            "Collecting datasets (from -r requirements.txt (line 3))\n",
            "  Downloading datasets-2.13.1-py3-none-any.whl (486 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m486.2/486.2 kB\u001b[0m \u001b[31m44.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting einops (from -r requirements.txt (line 4))\n",
            "  Downloading einops-0.6.1-py3-none-any.whl (42 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.2/42.2 kB\u001b[0m \u001b[31m5.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting fastapi==0.95.2 (from -r requirements.txt (line 5))\n",
            "  Downloading fastapi-0.95.2-py3-none-any.whl (56 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.0/57.0 kB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting flexgen==0.1.7 (from -r requirements.txt (line 6))\n",
            "  Downloading flexgen-0.1.7-py3-none-any.whl (50 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.8/50.8 kB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting gradio_client==0.2.5 (from -r requirements.txt (line 7))\n",
            "  Downloading gradio_client-0.2.5-py3-none-any.whl (288 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m288.1/288.1 kB\u001b[0m \u001b[31m39.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting gradio==3.33.1 (from -r requirements.txt (line 8))\n",
            "  Downloading gradio-3.33.1-py3-none-any.whl (20.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m20.0/20.0 MB\u001b[0m \u001b[31m72.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: markdown in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 9)) (3.4.3)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 10)) (1.22.4)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 11)) (1.5.3)\n",
            "Collecting Pillow>=9.5.0 (from -r requirements.txt (line 12))\n",
            "  Downloading Pillow-10.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (3.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.4/3.4 MB\u001b[0m \u001b[31m89.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 13)) (6.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 14)) (2.27.1)\n",
            "Collecting safetensors==0.3.1 (from -r requirements.txt (line 15))\n",
            "  Downloading safetensors-0.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m82.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sentencepiece (from -r requirements.txt (line 16))\n",
            "  Downloading sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m66.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 17)) (4.65.0)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 18)) (1.10.1)\n",
            "Requirement already satisfied: tensorboard in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 19)) (2.12.3)\n",
            "Collecting wandb (from -r requirements.txt (line 20))\n",
            "  Downloading wandb-0.15.5-py3-none-any.whl (2.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m84.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting transformers==4.31.* (from -r requirements.txt (line 21))\n",
            "  Downloading transformers-4.31.0-py3-none-any.whl (7.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.4/7.4 MB\u001b[0m \u001b[31m86.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting bitsandbytes==0.40.2 (from -r requirements.txt (line 23))\n",
            "  Downloading bitsandbytes-0.40.2-py3-none-any.whl (92.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m92.5/92.5 MB\u001b[0m \u001b[31m9.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting llama-cpp-python==0.1.73 (from -r requirements.txt (line 25))\n",
            "  Downloading llama_cpp_python-0.1.73.tar.gz (1.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m62.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.21.0->-r requirements.txt (line 1)) (23.1)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate==0.21.0->-r requirements.txt (line 1)) (5.9.5)\n",
            "Requirement already satisfied: torch>=1.10.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.21.0->-r requirements.txt (line 1)) (2.0.1+cu118)\n",
            "Requirement already satisfied: pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2 in /usr/local/lib/python3.10/dist-packages (from fastapi==0.95.2->-r requirements.txt (line 5)) (1.10.11)\n",
            "Collecting starlette<0.28.0,>=0.27.0 (from fastapi==0.95.2->-r requirements.txt (line 5))\n",
            "  Downloading starlette-0.27.0-py3-none-any.whl (66 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.0/67.0 kB\u001b[0m \u001b[31m10.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pulp (from flexgen==0.1.7->-r requirements.txt (line 6))\n",
            "  Downloading PuLP-2.7.0-py3-none-any.whl (14.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.3/14.3 MB\u001b[0m \u001b[31m32.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: attrs in /usr/local/lib/python3.10/dist-packages (from flexgen==0.1.7->-r requirements.txt (line 6)) (23.1.0)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio_client==0.2.5->-r requirements.txt (line 7)) (2023.6.0)\n",
            "Collecting httpx (from gradio_client==0.2.5->-r requirements.txt (line 7))\n",
            "  Downloading httpx-0.24.1-py3-none-any.whl (75 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.4/75.4 kB\u001b[0m \u001b[31m11.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting huggingface-hub>=0.13.0 (from gradio_client==0.2.5->-r requirements.txt (line 7))\n",
            "  Downloading huggingface_hub-0.16.4-py3-none-any.whl (268 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m268.8/268.8 kB\u001b[0m \u001b[31m35.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from gradio_client==0.2.5->-r requirements.txt (line 7)) (4.7.1)\n",
            "Collecting websockets (from gradio_client==0.2.5->-r requirements.txt (line 7))\n",
            "  Downloading websockets-11.0.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (129 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.9/129.9 kB\u001b[0m \u001b[31m13.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting aiofiles (from gradio==3.33.1->-r requirements.txt (line 8))\n",
            "  Downloading aiofiles-23.1.0-py3-none-any.whl (14 kB)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from gradio==3.33.1->-r requirements.txt (line 8)) (3.8.4)\n",
            "Requirement already satisfied: altair>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.33.1->-r requirements.txt (line 8)) (4.2.2)\n",
            "Collecting ffmpy (from gradio==3.33.1->-r requirements.txt (line 8))\n",
            "  Downloading ffmpy-0.3.1.tar.gz (5.5 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from gradio==3.33.1->-r requirements.txt (line 8)) (3.1.2)\n",
            "Requirement already satisfied: markdown-it-py[linkify]>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.33.1->-r requirements.txt (line 8)) (3.0.0)\n",
            "Requirement already satisfied: markupsafe in /usr/local/lib/python3.10/dist-packages (from gradio==3.33.1->-r requirements.txt (line 8)) (2.1.3)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from gradio==3.33.1->-r requirements.txt (line 8)) (3.7.1)\n",
            "Collecting mdit-py-plugins<=0.3.3 (from gradio==3.33.1->-r requirements.txt (line 8))\n",
            "  Downloading mdit_py_plugins-0.3.3-py3-none-any.whl (50 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.5/50.5 kB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting orjson (from gradio==3.33.1->-r requirements.txt (line 8))\n",
            "  Downloading orjson-3.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (138 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m138.7/138.7 kB\u001b[0m \u001b[31m19.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pydub (from gradio==3.33.1->-r requirements.txt (line 8))\n",
            "  Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)\n",
            "Requirement already satisfied: pygments>=2.12.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.33.1->-r requirements.txt (line 8)) (2.14.0)\n",
            "Collecting python-multipart (from gradio==3.33.1->-r requirements.txt (line 8))\n",
            "  Downloading python_multipart-0.0.6-py3-none-any.whl (45 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.7/45.7 kB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting semantic-version (from gradio==3.33.1->-r requirements.txt (line 8))\n",
            "  Downloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)\n",
            "Collecting uvicorn>=0.14.0 (from gradio==3.33.1->-r requirements.txt (line 8))\n",
            "  Downloading uvicorn-0.23.1-py3-none-any.whl (59 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m59.5/59.5 kB\u001b[0m \u001b[31m9.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.*->-r requirements.txt (line 21)) (3.12.2)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.*->-r requirements.txt (line 21)) (2022.10.31)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.31.*->-r requirements.txt (line 21))\n",
            "  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m86.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting diskcache>=5.6.1 (from llama-cpp-python==0.1.73->-r requirements.txt (line 25))\n",
            "  Downloading diskcache-5.6.1-py3-none-any.whl (45 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.6/45.6 kB\u001b[0m \u001b[31m5.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 3)) (9.0.0)\n",
            "Collecting dill<0.3.7,>=0.3.0 (from datasets->-r requirements.txt (line 3))\n",
            "  Downloading dill-0.3.6-py3-none-any.whl (110 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m110.5/110.5 kB\u001b[0m \u001b[31m15.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting xxhash (from datasets->-r requirements.txt (line 3))\n",
            "  Downloading xxhash-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m212.5/212.5 kB\u001b[0m \u001b[31m22.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting multiprocess (from datasets->-r requirements.txt (line 3))\n",
            "  Downloading multiprocess-0.70.14-py310-none-any.whl (134 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.3/134.3 kB\u001b[0m \u001b[31m15.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 11)) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 11)) (2022.7.1)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 14)) (1.26.16)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 14)) (2023.5.7)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 14)) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 14)) (3.4)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 19)) (1.4.0)\n",
            "Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 19)) (1.56.0)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 19)) (2.17.3)\n",
            "Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 19)) (1.0.0)\n",
            "Requirement already satisfied: protobuf>=3.19.6 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 19)) (3.20.3)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 19)) (67.7.2)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 19)) (0.7.1)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 19)) (2.3.6)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 19)) (0.40.0)\n",
            "Requirement already satisfied: Click!=8.0.0,>=7.1 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 20)) (8.1.4)\n",
            "Collecting GitPython!=3.1.29,>=1.0.0 (from wandb->-r requirements.txt (line 20))\n",
            "  Downloading GitPython-3.1.32-py3-none-any.whl (188 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m188.5/188.5 kB\u001b[0m \u001b[31m20.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sentry-sdk>=1.0.0 (from wandb->-r requirements.txt (line 20))\n",
            "  Downloading sentry_sdk-1.28.1-py2.py3-none-any.whl (214 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m214.7/214.7 kB\u001b[0m \u001b[31m24.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting docker-pycreds>=0.4.0 (from wandb->-r requirements.txt (line 20))\n",
            "  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)\n",
            "Collecting pathtools (from wandb->-r requirements.txt (line 20))\n",
            "  Downloading pathtools-0.1.2.tar.gz (11 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting setproctitle (from wandb->-r requirements.txt (line 20))\n",
            "  Downloading setproctitle-1.3.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30 kB)\n",
            "Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 20)) (1.4.4)\n",
            "Collecting rouge (from auto-gptq==0.3.0+cu117->-r requirements.txt (line 28))\n",
            "  Downloading rouge-1.0.1-py3-none-any.whl (13 kB)\n",
            "Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair>=4.2.0->gradio==3.33.1->-r requirements.txt (line 8)) (0.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair>=4.2.0->gradio==3.33.1->-r requirements.txt (line 8)) (4.3.3)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair>=4.2.0->gradio==3.33.1->-r requirements.txt (line 8)) (0.12.0)\n",
            "Requirement already satisfied: six>=1.4.0 in /usr/local/lib/python3.10/dist-packages (from docker-pycreds>=0.4.0->wandb->-r requirements.txt (line 20)) (1.16.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.33.1->-r requirements.txt (line 8)) (6.0.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.33.1->-r requirements.txt (line 8)) (4.0.2)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.33.1->-r requirements.txt (line 8)) (1.9.2)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.33.1->-r requirements.txt (line 8)) (1.4.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.33.1->-r requirements.txt (line 8)) (1.3.1)\n",
            "Collecting gitdb<5,>=4.0.1 (from GitPython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 20))\n",
            "  Downloading gitdb-4.0.10-py3-none-any.whl (62 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m9.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 19)) (5.3.1)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 19)) (0.3.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 19)) (4.9)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard->-r requirements.txt (line 19)) (1.3.1)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py[linkify]>=2.0.0->gradio==3.33.1->-r requirements.txt (line 8)) (0.1.2)\n",
            "Collecting linkify-it-py<3,>=1 (from markdown-it-py[linkify]>=2.0.0->gradio==3.33.1->-r requirements.txt (line 8))\n",
            "  Downloading linkify_it_py-2.0.2-py3-none-any.whl (19 kB)\n",
            "INFO: pip is looking at multiple versions of mdit-py-plugins to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting mdit-py-plugins<=0.3.3 (from gradio==3.33.1->-r requirements.txt (line 8))\n",
            "  Downloading mdit_py_plugins-0.3.2-py3-none-any.whl (50 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.4/50.4 kB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading mdit_py_plugins-0.3.1-py3-none-any.whl (46 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.5/46.5 kB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading mdit_py_plugins-0.3.0-py3-none-any.whl (43 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.7/43.7 kB\u001b[0m \u001b[31m5.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading mdit_py_plugins-0.2.8-py3-none-any.whl (41 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m41.0/41.0 kB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading mdit_py_plugins-0.2.7-py3-none-any.whl (41 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m41.0/41.0 kB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading mdit_py_plugins-0.2.6-py3-none-any.whl (39 kB)\n",
            "  Downloading mdit_py_plugins-0.2.5-py3-none-any.whl (39 kB)\n",
            "INFO: pip is looking at multiple versions of mdit-py-plugins to determine which version is compatible with other requirements. This could take a while.\n",
            "  Downloading mdit_py_plugins-0.2.4-py3-none-any.whl (39 kB)\n",
            "  Downloading mdit_py_plugins-0.2.3-py3-none-any.whl (39 kB)\n",
            "  Downloading mdit_py_plugins-0.2.2-py3-none-any.whl (39 kB)\n",
            "  Downloading mdit_py_plugins-0.2.1-py3-none-any.whl (38 kB)\n",
            "  Downloading mdit_py_plugins-0.2.0-py3-none-any.whl (38 kB)\n",
            "INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.\n",
            "  Downloading mdit_py_plugins-0.1.0-py3-none-any.whl (37 kB)\n",
            "Collecting markdown-it-py[linkify]>=2.0.0 (from gradio==3.33.1->-r requirements.txt (line 8))\n",
            "  Downloading markdown_it_py-3.0.0-py3-none-any.whl (87 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m87.5/87.5 kB\u001b[0m \u001b[31m13.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading markdown_it_py-2.2.0-py3-none-any.whl (84 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m84.5/84.5 kB\u001b[0m \u001b[31m12.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: anyio<5,>=3.4.0 in /usr/local/lib/python3.10/dist-packages (from starlette<0.28.0,>=0.27.0->fastapi==0.95.2->-r requirements.txt (line 5)) (3.7.1)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.21.0->-r requirements.txt (line 1)) (1.11.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.21.0->-r requirements.txt (line 1)) (3.1)\n",
            "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.21.0->-r requirements.txt (line 1)) (2.0.0)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.10.0->accelerate==0.21.0->-r requirements.txt (line 1)) (3.25.2)\n",
            "Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.10.0->accelerate==0.21.0->-r requirements.txt (line 1)) (16.0.6)\n",
            "Collecting h11>=0.8 (from uvicorn>=0.14.0->gradio==3.33.1->-r requirements.txt (line 8))\n",
            "  Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m9.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting httpcore<0.18.0,>=0.15.0 (from httpx->gradio_client==0.2.5->-r requirements.txt (line 7))\n",
            "  Downloading httpcore-0.17.3-py3-none-any.whl (74 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m74.5/74.5 kB\u001b[0m \u001b[31m11.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx->gradio_client==0.2.5->-r requirements.txt (line 7)) (1.3.0)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.33.1->-r requirements.txt (line 8)) (1.1.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.33.1->-r requirements.txt (line 8)) (0.11.0)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.33.1->-r requirements.txt (line 8)) (4.41.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.33.1->-r requirements.txt (line 8)) (1.4.4)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.33.1->-r requirements.txt (line 8)) (3.1.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi==0.95.2->-r requirements.txt (line 5)) (1.1.2)\n",
            "Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 20))\n",
            "  Downloading smmap-5.0.0-py3-none-any.whl (24 kB)\n",
            "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair>=4.2.0->gradio==3.33.1->-r requirements.txt (line 8)) (0.19.3)\n",
            "Collecting uc-micro-py (from linkify-it-py<3,>=1->markdown-it-py[linkify]>=2.0.0->gradio==3.33.1->-r requirements.txt (line 8))\n",
            "  Downloading uc_micro_py-1.0.2-py3-none-any.whl (6.2 kB)\n",
            "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 19)) (0.5.0)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard->-r requirements.txt (line 19)) (3.2.2)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.10.0->accelerate==0.21.0->-r requirements.txt (line 1)) (1.3.0)\n",
            "Building wheels for collected packages: llama-cpp-python, peft, ffmpy, pathtools\n",
            "  Building wheel for llama-cpp-python (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for llama-cpp-python: filename=llama_cpp_python-0.1.73-cp310-cp310-linux_x86_64.whl size=264154 sha256=b4e191da1606237b89e1ef4ac34960f08e7ad9fba6c8067be241de96d3a85f39\n",
            "  Stored in directory: /root/.cache/pip/wheels/69/6a/cb/41fd7c0f7f77613d996b4981a3e9b345d2b416b016786956bc\n",
            "  Building wheel for peft (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for peft: filename=peft-0.4.0.dev0-py3-none-any.whl size=61146 sha256=06505c626e19ebc730cc635c2941c79c441516a276c570165899855ad07bb14d\n",
            "  Stored in directory: /root/.cache/pip/wheels/73/14/95/62d30d224b10f5601d4d015b8ff71bb9b8e23876f07086198c\n",
            "  Building wheel for ffmpy (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ffmpy: filename=ffmpy-0.3.1-py3-none-any.whl size=5579 sha256=42544a0146a9e7a65240b52798b647c46ab8edf04f6b77b7faa07394e8fbca87\n",
            "  Stored in directory: /root/.cache/pip/wheels/01/a6/d1/1c0828c304a4283b2c1639a09ad86f83d7c487ef34c6b4a1bf\n",
            "  Building wheel for pathtools (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pathtools: filename=pathtools-0.1.2-py3-none-any.whl size=8791 sha256=477891f3ae3999acd2355c09a5e7bc0b279aad7a3c86092751fd7a2c2dc2b9e2\n",
            "  Stored in directory: /root/.cache/pip/wheels/e7/f3/22/152153d6eb222ee7a56ff8617d80ee5207207a8c00a7aab794\n",
            "Successfully built llama-cpp-python peft ffmpy pathtools\n",
            "Installing collected packages: tokenizers, sentencepiece, safetensors, pydub, pulp, pathtools, ffmpy, bitsandbytes, xxhash, websockets, uc-micro-py, smmap, setproctitle, sentry-sdk, semantic-version, rouge, python-multipart, Pillow, orjson, markdown-it-py, h11, einops, docker-pycreds, diskcache, dill, colorama, aiofiles, uvicorn, starlette, multiprocess, mdit-py-plugins, llama-cpp-python, linkify-it-py, huggingface-hub, httpcore, gitdb, transformers, httpx, GitPython, fastapi, wandb, gradio_client, datasets, gradio, accelerate, peft, flexgen, exllama, auto-gptq\n",
            "  Attempting uninstall: Pillow\n",
            "    Found existing installation: Pillow 8.4.0\n",
            "    Uninstalling Pillow-8.4.0:\n",
            "      Successfully uninstalled Pillow-8.4.0\n",
            "  Attempting uninstall: markdown-it-py\n",
            "    Found existing installation: markdown-it-py 3.0.0\n",
            "    Uninstalling markdown-it-py-3.0.0:\n",
            "      Successfully uninstalled markdown-it-py-3.0.0\n",
            "Successfully installed GitPython-3.1.32 Pillow-10.0.0 accelerate-0.21.0 aiofiles-23.1.0 auto-gptq-0.3.0+cu117 bitsandbytes-0.40.2 colorama-0.4.6 datasets-2.13.1 dill-0.3.6 diskcache-5.6.1 docker-pycreds-0.4.0 einops-0.6.1 exllama-0.0.7+cu117 fastapi-0.95.2 ffmpy-0.3.1 flexgen-0.1.7 gitdb-4.0.10 gradio-3.33.1 gradio_client-0.2.5 h11-0.14.0 httpcore-0.17.3 httpx-0.24.1 huggingface-hub-0.16.4 linkify-it-py-2.0.2 llama-cpp-python-0.1.73 markdown-it-py-2.2.0 mdit-py-plugins-0.3.3 multiprocess-0.70.14 orjson-3.9.2 pathtools-0.1.2 peft-0.4.0.dev0 pulp-2.7.0 pydub-0.25.1 python-multipart-0.0.6 rouge-1.0.1 safetensors-0.3.1 semantic-version-2.10.0 sentencepiece-0.1.99 sentry-sdk-1.28.1 setproctitle-1.3.2 smmap-5.0.0 starlette-0.27.0 tokenizers-0.13.3 transformers-4.31.0 uc-micro-py-1.0.2 uvicorn-0.23.1 wandb-0.15.5 websockets-11.0.3 xxhash-3.2.0\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "PIL"
                ]
              }
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting gradio==3.32.0\n",
            "  Downloading gradio-3.32.0-py3-none-any.whl (19.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m19.9/19.9 MB\u001b[0m \u001b[31m60.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: aiofiles in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (23.1.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (3.8.4)\n",
            "Requirement already satisfied: altair>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (4.2.2)\n",
            "Requirement already satisfied: fastapi in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.95.2)\n",
            "Requirement already satisfied: ffmpy in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.3.1)\n",
            "Requirement already satisfied: gradio-client>=0.2.4 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.2.5)\n",
            "Requirement already satisfied: httpx in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.24.1)\n",
            "Requirement already satisfied: huggingface-hub>=0.13.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.16.4)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (3.1.2)\n",
            "Requirement already satisfied: markdown-it-py[linkify]>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (2.2.0)\n",
            "Requirement already satisfied: markupsafe in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (2.1.3)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (3.7.1)\n",
            "Requirement already satisfied: mdit-py-plugins<=0.3.3 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.3.3)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (1.22.4)\n",
            "Requirement already satisfied: orjson in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (3.9.2)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (1.5.3)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (10.0.0)\n",
            "Requirement already satisfied: pydantic in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (1.10.11)\n",
            "Requirement already satisfied: pydub in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.25.1)\n",
            "Requirement already satisfied: pygments>=2.12.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (2.14.0)\n",
            "Requirement already satisfied: python-multipart in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.0.6)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (6.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (2.27.1)\n",
            "Requirement already satisfied: semantic-version in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (2.10.0)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (4.7.1)\n",
            "Requirement already satisfied: uvicorn>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.23.1)\n",
            "Requirement already satisfied: websockets>=10.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (11.0.3)\n",
            "Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair>=4.2.0->gradio==3.32.0) (0.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair>=4.2.0->gradio==3.32.0) (4.3.3)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair>=4.2.0->gradio==3.32.0) (0.12.0)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio-client>=0.2.4->gradio==3.32.0) (2023.6.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from gradio-client>=0.2.4->gradio==3.32.0) (23.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.13.0->gradio==3.32.0) (3.12.2)\n",
            "Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.13.0->gradio==3.32.0) (4.65.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py[linkify]>=2.0.0->gradio==3.32.0) (0.1.2)\n",
            "Requirement already satisfied: linkify-it-py<3,>=1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py[linkify]>=2.0.0->gradio==3.32.0) (2.0.2)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->gradio==3.32.0) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->gradio==3.32.0) (2022.7.1)\n",
            "Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/dist-packages (from uvicorn>=0.14.0->gradio==3.32.0) (8.1.4)\n",
            "Requirement already satisfied: h11>=0.8 in /usr/local/lib/python3.10/dist-packages (from uvicorn>=0.14.0->gradio==3.32.0) (0.14.0)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.32.0) (23.1.0)\n",
            "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.32.0) (2.0.12)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.32.0) (6.0.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.32.0) (4.0.2)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.32.0) (1.9.2)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.32.0) (1.4.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.32.0) (1.3.1)\n",
            "Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /usr/local/lib/python3.10/dist-packages (from fastapi->gradio==3.32.0) (0.27.0)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx->gradio==3.32.0) (2023.5.7)\n",
            "Requirement already satisfied: httpcore<0.18.0,>=0.15.0 in /usr/local/lib/python3.10/dist-packages (from httpx->gradio==3.32.0) (0.17.3)\n",
            "Requirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx->gradio==3.32.0) (3.4)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx->gradio==3.32.0) (1.3.0)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.32.0) (1.1.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.32.0) (0.11.0)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.32.0) (4.41.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.32.0) (1.4.4)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.32.0) (3.1.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->gradio==3.32.0) (1.26.16)\n",
            "Requirement already satisfied: anyio<5.0,>=3.0 in /usr/local/lib/python3.10/dist-packages (from httpcore<0.18.0,>=0.15.0->httpx->gradio==3.32.0) (3.7.1)\n",
            "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair>=4.2.0->gradio==3.32.0) (0.19.3)\n",
            "Requirement already satisfied: uc-micro-py in /usr/local/lib/python3.10/dist-packages (from linkify-it-py<3,>=1->markdown-it-py[linkify]>=2.0.0->gradio==3.32.0) (1.0.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas->gradio==3.32.0) (1.16.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5.0,>=3.0->httpcore<0.18.0,>=0.15.0->httpx->gradio==3.32.0) (1.1.2)\n",
            "Installing collected packages: gradio\n",
            "  Attempting uninstall: gradio\n",
            "    Found existing installation: gradio 3.33.1\n",
            "    Uninstalling gradio-3.33.1:\n",
            "      Successfully uninstalled gradio-3.33.1\n",
            "Successfully installed gradio-3.32.0\n",
            "Found existing installation: llama-cpp-python 0.1.73\n",
            "Uninstalling llama-cpp-python-0.1.73:\n",
            "  Successfully uninstalled llama-cpp-python-0.1.73\n",
            "Collecting llama-cpp-python\n",
            "  Downloading llama_cpp_python-0.1.73.tar.gz (1.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m14.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: typing-extensions>=4.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (4.7.1)\n",
            "Requirement already satisfied: numpy>=1.20.0 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (1.22.4)\n",
            "Requirement already satisfied: diskcache>=5.6.1 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (5.6.1)\n",
            "Building wheels for collected packages: llama-cpp-python\n",
            "  Building wheel for llama-cpp-python (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for llama-cpp-python: filename=llama_cpp_python-0.1.73-cp310-cp310-linux_x86_64.whl size=487290 sha256=48447d41e6e2f61fd1643611ec9e8699f2c5081c9161aa521212e875cee415da\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-ihofkbuy/wheels/69/6a/cb/41fd7c0f7f77613d996b4981a3e9b345d2b416b016786956bc\n",
            "Successfully built llama-cpp-python\n",
            "Installing collected packages: llama-cpp-python\n",
            "Successfully installed llama-cpp-python-0.1.73\n",
            "\u001b[0m\n",
            "Download Results:\n",
            "gid   |stat|avg speed  |path/URI\n",
            "======+====+===========+=======================================================\n",
            "9a3ebe|\u001b[1;32mOK\u001b[0m  |   238MiB/s|/content/text-generation-webui/models//llama-2-13b-chat.ggmlv3.q5_K_M.bin\n",
            "\n",
            "Status Legend:\n",
            "(OK):download completed.\n",
            "/content/text-generation-webui\n",
            "2023-07-20 06:11:07 WARNING:\u001b[33mThe gradio \"share link\" feature uses a proprietary executable to create a reverse tunnel. Use it with care.\u001b[0m\n",
            "2023-07-20 06:11:13.858681: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "2023-07-20 06:11:15 INFO:\u001b[32mLoading llama-2-13b-chat.ggmlv3.q5_K_M.bin...\u001b[0m\n",
            "ggml_init_cublas: found 1 CUDA devices:\n",
            "  Device 0: Tesla T4, compute capability 7.5\n",
            "2023-07-20 06:11:25 INFO:\u001b[32mllama.cpp weights detected: models/llama-2-13b-chat.ggmlv3.q5_K_M.bin\n",
            "\u001b[0m\n",
            "2023-07-20 06:11:25 INFO:\u001b[32mCache capacity is 0 bytes\u001b[0m\n",
            "llama.cpp: loading model from models/llama-2-13b-chat.ggmlv3.q5_K_M.bin\n",
            "llama_model_load_internal: format     = ggjt v3 (latest)\n",
            "llama_model_load_internal: n_vocab    = 32000\n",
            "llama_model_load_internal: n_ctx      = 2048\n",
            "llama_model_load_internal: n_embd     = 5120\n",
            "llama_model_load_internal: n_mult     = 256\n",
            "llama_model_load_internal: n_head     = 40\n",
            "llama_model_load_internal: n_layer    = 40\n",
            "llama_model_load_internal: n_rot      = 128\n",
            "llama_model_load_internal: freq_base  = 10000.0\n",
            "llama_model_load_internal: freq_scale = 1\n",
            "llama_model_load_internal: ftype      = 17 (mostly Q5_K - Medium)\n",
            "llama_model_load_internal: n_ff       = 13824\n",
            "llama_model_load_internal: model size = 13B\n",
            "llama_model_load_internal: ggml ctx size =    0.09 MB\n",
            "llama_model_load_internal: using CUDA for GPU acceleration\n",
            "llama_model_load_internal: mem required  = 2077.52 MB (+ 1608.00 MB per state)\n",
            "llama_model_load_internal: allocating batch_size x (640 kB + n_ctx x 160 B) = 480 MB VRAM for the scratch buffer\n",
            "llama_model_load_internal: offloading 40 repeating layers to GPU\n",
            "llama_model_load_internal: offloading non-repeating layers to GPU\n",
            "llama_model_load_internal: offloading v cache to GPU\n",
            "llama_model_load_internal: offloading k cache to GPU\n",
            "llama_model_load_internal: offloaded 43/43 layers to GPU\n",
            "llama_model_load_internal: total VRAM used: 10783 MB\n",
            "llama_new_context_with_model: kv self size  = 1600.00 MB\n",
            "AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | \n",
            "2023-07-20 06:12:39 INFO:\u001b[32mLoaded the model in 84.34 seconds.\n",
            "\u001b[0m\n",
            "2023-07-20 06:12:39 INFO:\u001b[32mLoading the extension \"gallery\"...\u001b[0m\n",
            "Running on local URL:  http://127.0.0.1:7860\n",
            "Running on public URL: https://4de099bbf0339a9ec7.gradio.live\n",
            "\n",
            "This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =    68.20 ms /    36 runs   (    1.89 ms per token,   527.84 tokens per second)\n",
            "llama_print_timings: prompt eval time =  1128.48 ms /    48 tokens (   23.51 ms per token,    42.54 tokens per second)\n",
            "llama_print_timings:        eval time =  2254.77 ms /    35 runs   (   64.42 ms per token,    15.52 tokens per second)\n",
            "llama_print_timings:       total time =  3825.55 ms\n",
            "Output generated in 4.88 seconds (7.18 tokens/s, 35 tokens, context 48, seed 1332260347)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =    94.13 ms /    73 runs   (    1.29 ms per token,   775.49 tokens per second)\n",
            "llama_print_timings: prompt eval time =  1342.28 ms /    78 tokens (   17.21 ms per token,    58.11 tokens per second)\n",
            "llama_print_timings:        eval time =  4768.51 ms /    72 runs   (   66.23 ms per token,    15.10 tokens per second)\n",
            "llama_print_timings:       total time =  6708.75 ms\n",
            "Output generated in 7.47 seconds (9.64 tokens/s, 72 tokens, context 98, seed 1028588962)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =   993.76 ms /   800 runs   (    1.24 ms per token,   805.02 tokens per second)\n",
            "llama_print_timings: prompt eval time =  1006.73 ms /    57 tokens (   17.66 ms per token,    56.62 tokens per second)\n",
            "llama_print_timings:        eval time = 57416.40 ms /   799 runs   (   71.86 ms per token,    13.92 tokens per second)\n",
            "llama_print_timings:       total time = 64542.97 ms\n",
            "Output generated in 65.32 seconds (12.25 tokens/s, 800 tokens, context 77, seed 778763828)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =   371.90 ms /   310 runs   (    1.20 ms per token,   833.56 tokens per second)\n",
            "llama_print_timings: prompt eval time =  6627.78 ms /   800 tokens (    8.28 ms per token,   120.70 tokens per second)\n",
            "llama_print_timings:        eval time = 24265.96 ms /   309 runs   (   78.53 ms per token,    12.73 tokens per second)\n",
            "llama_print_timings:       total time = 33096.58 ms\n",
            "Output generated in 33.85 seconds (9.13 tokens/s, 309 tokens, context 876, seed 1064984253)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =   184.44 ms /   165 runs   (    1.12 ms per token,   894.59 tokens per second)\n",
            "llama_print_timings: prompt eval time =   971.71 ms /    58 tokens (   16.75 ms per token,    59.69 tokens per second)\n",
            "llama_print_timings:        eval time = 11435.44 ms /   164 runs   (   69.73 ms per token,    14.34 tokens per second)\n",
            "llama_print_timings:       total time = 13286.62 ms\n",
            "Output generated in 14.14 seconds (11.60 tokens/s, 164 tokens, context 78, seed 2131392435)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =    35.82 ms /    37 runs   (    0.97 ms per token,  1032.94 tokens per second)\n",
            "llama_print_timings: prompt eval time =   960.10 ms /    52 tokens (   18.46 ms per token,    54.16 tokens per second)\n",
            "llama_print_timings:        eval time =  2481.79 ms /    36 runs   (   68.94 ms per token,    14.51 tokens per second)\n",
            "llama_print_timings:       total time =  3576.52 ms\n",
            "Output generated in 4.32 seconds (8.32 tokens/s, 36 tokens, context 72, seed 16628462)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =    18.71 ms /    19 runs   (    0.98 ms per token,  1015.61 tokens per second)\n",
            "llama_print_timings: prompt eval time =   946.92 ms /    39 tokens (   24.28 ms per token,    41.19 tokens per second)\n",
            "llama_print_timings:        eval time =  1237.18 ms /    18 runs   (   68.73 ms per token,    14.55 tokens per second)\n",
            "llama_print_timings:       total time =  2258.21 ms\n",
            "Output generated in 3.01 seconds (5.98 tokens/s, 18 tokens, context 59, seed 440082101)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =    19.06 ms /    20 runs   (    0.95 ms per token,  1049.21 tokens per second)\n",
            "llama_print_timings: prompt eval time =   965.89 ms /    46 tokens (   21.00 ms per token,    47.62 tokens per second)\n",
            "llama_print_timings:        eval time =  1300.01 ms /    19 runs   (   68.42 ms per token,    14.62 tokens per second)\n",
            "llama_print_timings:       total time =  2338.89 ms\n",
            "Output generated in 3.09 seconds (6.15 tokens/s, 19 tokens, context 66, seed 1242610206)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =    41.10 ms /    21 runs   (    1.96 ms per token,   510.90 tokens per second)\n",
            "llama_print_timings: prompt eval time =   712.78 ms /    26 tokens (   27.41 ms per token,    36.48 tokens per second)\n",
            "llama_print_timings:        eval time =  1286.52 ms /    20 runs   (   64.33 ms per token,    15.55 tokens per second)\n",
            "llama_print_timings:       total time =  2248.05 ms\n",
            "Output generated in 3.27 seconds (6.12 tokens/s, 20 tokens, context 46, seed 618105233)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =    47.94 ms /    49 runs   (    0.98 ms per token,  1022.03 tokens per second)\n",
            "llama_print_timings: prompt eval time =   700.56 ms /    15 tokens (   46.70 ms per token,    21.41 tokens per second)\n",
            "llama_print_timings:        eval time =  3238.37 ms /    48 runs   (   67.47 ms per token,    14.82 tokens per second)\n",
            "llama_print_timings:       total time =  4117.39 ms\n",
            "Output generated in 4.89 seconds (9.82 tokens/s, 48 tokens, context 35, seed 639104230)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =    68.71 ms /    56 runs   (    1.23 ms per token,   815.02 tokens per second)\n",
            "llama_print_timings: prompt eval time =   961.24 ms /    53 tokens (   18.14 ms per token,    55.14 tokens per second)\n",
            "llama_print_timings:        eval time =  3689.08 ms /    55 runs   (   67.07 ms per token,    14.91 tokens per second)\n",
            "llama_print_timings:       total time =  4982.41 ms\n",
            "Output generated in 5.94 seconds (9.26 tokens/s, 55 tokens, context 73, seed 34574907)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =    34.86 ms /    27 runs   (    1.29 ms per token,   774.57 tokens per second)\n",
            "llama_print_timings: prompt eval time =   691.87 ms /    23 tokens (   30.08 ms per token,    33.24 tokens per second)\n",
            "llama_print_timings:        eval time =  1721.19 ms /    26 runs   (   66.20 ms per token,    15.11 tokens per second)\n",
            "llama_print_timings:       total time =  2599.64 ms\n",
            "Output generated in 3.37 seconds (7.72 tokens/s, 26 tokens, context 43, seed 1924613948)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =   138.10 ms /    76 runs   (    1.82 ms per token,   550.31 tokens per second)\n",
            "llama_print_timings: prompt eval time =  1756.13 ms /   182 tokens (    9.65 ms per token,   103.64 tokens per second)\n",
            "llama_print_timings:        eval time =  5031.65 ms /    75 runs   (   67.09 ms per token,    14.91 tokens per second)\n",
            "llama_print_timings:       total time =  7632.72 ms\n",
            "Output generated in 8.58 seconds (8.74 tokens/s, 75 tokens, context 202, seed 1780884341)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =    99.36 ms /    72 runs   (    1.38 ms per token,   724.62 tokens per second)\n",
            "llama_print_timings: prompt eval time =  2710.88 ms /   261 tokens (   10.39 ms per token,    96.28 tokens per second)\n",
            "llama_print_timings:        eval time =  4955.16 ms /    71 runs   (   69.79 ms per token,    14.33 tokens per second)\n",
            "llama_print_timings:       total time =  8172.42 ms\n",
            "Output generated in 8.95 seconds (7.93 tokens/s, 71 tokens, context 330, seed 1965288558)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =    47.96 ms /    49 runs   (    0.98 ms per token,  1021.68 tokens per second)\n",
            "llama_print_timings: prompt eval time =   696.39 ms /    14 tokens (   49.74 ms per token,    20.10 tokens per second)\n",
            "llama_print_timings:        eval time =  3230.15 ms /    48 runs   (   67.29 ms per token,    14.86 tokens per second)\n",
            "llama_print_timings:       total time =  4103.70 ms\n",
            "Output generated in 4.86 seconds (9.87 tokens/s, 48 tokens, context 34, seed 1230583247)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =    72.27 ms /    73 runs   (    0.99 ms per token,  1010.13 tokens per second)\n",
            "llama_print_timings: prompt eval time =  1326.30 ms /    91 tokens (   14.57 ms per token,    68.61 tokens per second)\n",
            "llama_print_timings:        eval time =  4896.58 ms /    72 runs   (   68.01 ms per token,    14.70 tokens per second)\n",
            "llama_print_timings:       total time =  6494.05 ms\n",
            "Output generated in 7.26 seconds (9.91 tokens/s, 72 tokens, context 111, seed 2066306018)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =  1128.62 ms\n",
            "llama_print_timings:      sample time =   306.49 ms /   255 runs   (    1.20 ms per token,   832.00 tokens per second)\n",
            "llama_print_timings: prompt eval time =  1745.34 ms /   173 tokens (   10.09 ms per token,    99.12 tokens per second)\n",
            "llama_print_timings:        eval time = 17700.43 ms /   254 runs   (   69.69 ms per token,    14.35 tokens per second)\n",
            "llama_print_timings:       total time = 20997.59 ms\n",
            "Output generated in 22.07 seconds (11.51 tokens/s, 254 tokens, context 193, seed 735929894)\n"
          ]
        }
      ],
      "source": [
        "%cd /content\n",
        "!apt-get -y install -qq aria2\n",
        "\n",
        "!git clone -b V20230720 https://github.com/Troyanovsky/text-generation-webui\n",
        "%cd /content/text-generation-webui\n",
        "!pip install -r requirements.txt\n",
        "!pip install -U gradio==3.32.0\n",
        "\n",
        "!pip uninstall -y llama-cpp-python\n",
        "!CMAKE_ARGS=\"-DLLAMA_CUBLAS=on\" FORCE_CMAKE=1 pip install llama-cpp-python==0.1.78 --no-cache-dir\n",
        "\n",
        "!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML/resolve/main/llama-2-13b-chat.ggmlv3.q5_K_M.bin -d /content/text-generation-webui/models/ -o llama-2-13b-chat.ggmlv3.q5_K_M.bin\n",
        "\n",
        "%cd /content/text-generation-webui\n",
        "!python server.py --share --chat --n-gpu-layers 200000 --model llama-2-13b-chat.ggmlv3.q5_K_M.bin"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyN+lYVJMWxt0u3lEYaBXv4P",
      "include_colab_link": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}