Evaluator_Mistral_7B_text_to_sql.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "machine_shape": "hm",
      "gpuType": "L4",
      "collapsed_sections": [
        "cCGJhXUKlZTp",
        "bPcvPNghliYH",
        "7UPkiC-Yl38G",
        "U9F0m1TlmGWH",
        "daZJRXxuMsQp",
        "wEsmtqGJ1pxE"
      ],
      "authorship_tag": "ABX9TyMMhTNxurebSP+/AKxQOI0r",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/frank-morales2020/MLxDL/blob/main/Evaluator_Mistral_7B_text_to_sql.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "https://medium.com/@frankmorales_91352/fine-tuning-the-llm-mistral-7b-instruct-v0-3-249c1814ceaf"
      ],
      "metadata": {
        "id": "a0h4qeKt2PsS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Dependencies"
      ],
      "metadata": {
        "id": "cCGJhXUKlZTp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!nvidia-smi"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oJi-Qn1p-po2",
        "outputId": "50f3a5ea-674a-4e60-a09b-a10bdc1f28ea"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tue Jul  2 08:19:23 2024       \n",
            "+---------------------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |\n",
            "|-----------------------------------------+----------------------+----------------------+\n",
            "| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                                         |                      |               MIG M. |\n",
            "|=========================================+======================+======================|\n",
            "|   0  NVIDIA L4                      Off | 00000000:00:03.0 Off |                    0 |\n",
            "| N/A   49C    P8              12W /  72W |      1MiB / 23034MiB |      0%      Default |\n",
            "|                                         |                      |                  N/A |\n",
            "+-----------------------------------------+----------------------+----------------------+\n",
            "                                                                                         \n",
            "+---------------------------------------------------------------------------------------+\n",
            "| Processes:                                                                            |\n",
            "|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |\n",
            "|        ID   ID                                                             Usage      |\n",
            "|=======================================================================================|\n",
            "|  No running processes found                                                           |\n",
            "+---------------------------------------------------------------------------------------+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-FdHyfrhXutZ"
      },
      "outputs": [],
      "source": [
        "# Install Pytorch & other libraries\n",
        "!pip install torch tensorboard --quiet\n",
        "\n",
        "# Install Hugging Face libraries\n",
        "!pip install  --upgrade transformers datasets accelerate evaluate bitsandbytes --quiet\n",
        "\n",
        "#FlashAttention only supports Ampere GPUs or newer. #NEED A100 , L4  IN GOOGLE COLAB\n",
        "!pip install -U flash-attn --no-build-isolation --quiet\n",
        "\n",
        "!pip install colab-env --quiet\n",
        "\n",
        "!pip install mistral_inference -q\n",
        "\n",
        "!pip install peft -q"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Hugging Face Setup"
      ],
      "metadata": {
        "id": "bPcvPNghliYH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import colab_env\n",
        "import os\n",
        "from huggingface_hub import login\n",
        "\n",
        "\n",
        "access_token_write = os.getenv(\"HUGGINGFACE_ACCESS_TOKEN_WRITE\")\n",
        "\n",
        "login(\n",
        "  token=access_token_write,\n",
        "  add_to_git_credential=True\n",
        ")\n",
        "\n",
        "\n",
        "#from huggingface_hub import notebook_login\n",
        "#notebook_login(write_permission=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ivq2gu9-YD3I",
        "outputId": "e5a851da-29c0-47ec-f0c1-c0b916b9a5a3"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/gdrive\n",
            "Token is valid (permission: write).\n",
            "Your token has been saved in your configured git credential helpers (store).\n",
            "Your token has been saved to /root/.cache/huggingface/token\n",
            "Login successful\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Load the Fine Tuned Model"
      ],
      "metadata": {
        "id": "7UPkiC-Yl38G"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import os\n",
        "import sys\n",
        "import json\n",
        "import IPython\n",
        "from datetime import datetime\n",
        "from datasets import load_dataset\n",
        "from peft import LoraConfig, PeftModel, prepare_model_for_kbit_training, get_peft_model\n",
        "from transformers import (\n",
        "    AutoModelForCausalLM,\n",
        "    AutoTokenizer,\n",
        "    BitsAndBytesConfig,\n",
        "    AutoTokenizer,\n",
        "    TrainingArguments,\n",
        ")"
      ],
      "metadata": {
        "id": "vfO7FZ2GYn3C"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "from peft import AutoPeftModelForCausalLM\n",
        "from transformers import AutoTokenizer, pipeline\n",
        "\n",
        "peft_model_id = \"frankmorales2020/Mistral-7B-text-to-sql-flash-attention-2-dataeval\"\n",
        "\n",
        "# BitsAndBytesConfig int-4 config\n",
        "bnb_config = BitsAndBytesConfig(\n",
        "    load_in_4bit=True,\n",
        "    bnb_4bit_use_double_quant=True,\n",
        "    bnb_4bit_quant_type=\"nf4\",\n",
        "    bnb_4bit_compute_dtype=torch.bfloat16\n",
        ")\n",
        "\n",
        "# Load Model with PEFT adapter\n",
        "model = AutoPeftModelForCausalLM.from_pretrained(\n",
        "    peft_model_id,\n",
        "    device_map=\"auto\",\n",
        "    attn_implementation=\"flash_attention_2\",\n",
        "    torch_dtype=torch.bfloat16,\n",
        "    quantization_config=bnb_config\n",
        ")\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(peft_model_id)\n",
        "\n",
        "# load into pipeline\n",
        "pipe = pipeline(\"text-generation\", model=model, tokenizer=tokenizer)"
      ],
      "metadata": {
        "id": "KuqnjucvY31a"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Tensorboad Setup"
      ],
      "metadata": {
        "id": "U9F0m1TlmGWH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import colab_env"
      ],
      "metadata": {
        "id": "vqjY6z8IZc6I"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "/content/gdrive/MyDrive/model/POC-Mistral-7B-text-to-sql-flash-attention-2-dataeval/logs"
      ],
      "metadata": {
        "id": "EF79dpClhkdY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%load_ext tensorboard\n",
        "\n",
        "##only in my personal dev-environment\n",
        "%tensorboard --logdir /content/gdrive/MyDrive/model/Mistral-7B-text-to-sql-flash-attention-2-dataeval/logs"
      ],
      "metadata": {
        "id": "S-sML0q_lJGu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Dataset Settings"
      ],
      "metadata": {
        "id": "s-obRbA0mSxQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from datasets import load_dataset\n",
        "\n",
        "# Convert dataset to OAI messages\n",
        "system_message = \"\"\"You are an text to SQL query translator. Users will ask you questions in English and you will generate a SQL query based on the provided SCHEMA.\n",
        "SCHEMA:\n",
        "{schema}\"\"\"\n",
        "\n",
        "def create_conversation(sample):\n",
        "  return {\n",
        "    \"messages\": [\n",
        "      {\"role\": \"system\", \"content\": system_message.format(schema=sample[\"context\"])},\n",
        "      {\"role\": \"user\", \"content\": sample[\"question\"]},\n",
        "      {\"role\": \"assistant\", \"content\": sample[\"answer\"]}\n",
        "    ]\n",
        "  }\n",
        "\n",
        "# Load dataset from the hub\n",
        "dataset = load_dataset(\"b-mc2/sql-create-context\", split=\"train\")\n",
        "#dataset = dataset.shuffle().select(range(12500))\n",
        "dataset = dataset.select(range(12500))\n",
        "\n",
        "# Convert dataset to OAI messages\n",
        "dataset = dataset.map(create_conversation, remove_columns=dataset.features,batched=False)\n",
        "\n",
        "# split dataset into 10,000 training samples and 2,500 test samples\n",
        "dataset = dataset.train_test_split(test_size=2500/12500)\n",
        "\n",
        "%cd /content/\n",
        "# save datasets to disk\n",
        "dataset[\"train\"].to_json(\"train_dataset.json\", orient=\"records\")\n",
        "dataset[\"test\"].to_json(\"test_dataset.json\", orient=\"records\")"
      ],
      "metadata": {
        "id": "mdb2mwS7c-ep"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Load our test dataset\n",
        "eval_dataset = load_dataset(\"json\", data_files=\"/content/test_dataset.json\", split=\"train\")"
      ],
      "metadata": {
        "id": "UMVmaasrfvul"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "eval_dataset"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Rq9mpc--Y19Y",
        "outputId": "3e60c9b9-b786-4ca8-ca97-a88f7bd53ea9"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Dataset({\n",
              "    features: ['messages'],\n",
              "    num_rows: 2500\n",
              "})"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Model Evaluation - Inference"
      ],
      "metadata": {
        "id": "qR7q5ST5muU1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Postgresql Settings"
      ],
      "metadata": {
        "id": "daZJRXxuMsQp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#ADDED By FM 01/06/2024\n",
        "!apt-get update -y\n",
        "!apt-get install postgresql-14 -y\n",
        "\n",
        "!service postgresql restart\n",
        "!sudo apt install postgresql-server-dev-all"
      ],
      "metadata": {
        "id": "k-DA-VwDpQLJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0686e2c9-8f26-4233-bac3-9245087b4d76"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Hit:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease\n",
            "Hit:2 https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/ InRelease\n",
            "Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease\n",
            "Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease\n",
            "Hit:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease\n",
            "Hit:6 https://ppa.launchpadcontent.net/c2d4u.team/c2d4u4.0+/ubuntu jammy InRelease\n",
            "Hit:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease\n",
            "Hit:8 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy InRelease\n",
            "Hit:9 https://ppa.launchpadcontent.net/graphics-drivers/ppa/ubuntu jammy InRelease\n",
            "Hit:10 https://ppa.launchpadcontent.net/ubuntugis/ppa/ubuntu jammy InRelease\n",
            "Reading package lists... Done\n",
            "Reading package lists... Done\n",
            "Building dependency tree... Done\n",
            "Reading state information... Done\n",
            "postgresql-14 is already the newest version (14.12-0ubuntu0.22.04.1).\n",
            "0 upgraded, 0 newly installed, 0 to remove and 48 not upgraded.\n",
            " * Restarting PostgreSQL 14 database server\n",
            "   ...done.\n",
            "Reading package lists... Done\n",
            "Building dependency tree... Done\n",
            "Reading state information... Done\n",
            "postgresql-server-dev-all is already the newest version (238).\n",
            "0 upgraded, 0 newly installed, 0 to remove and 48 not upgraded.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# (proceed_to_quarter_final VARCHAR, eliminated_from_competition VARCHAR)\n",
        "QUERY_create='CREATE TABLE table_name_24 (score VARCHAR, date VARCHAR)'"
      ],
      "metadata": {
        "id": "h9rbH7SB6Sae"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# PostGRES SQL Settings\n",
        "!sudo -u postgres psql -c \"CREATE USER postgres WITH SUPERUSER\"\n",
        "!sudo -u postgres psql -c \"ALTER USER postgres PASSWORD 'postgres'\""
      ],
      "metadata": {
        "id": "0sjsskj-61vu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dd4d359c-306f-412d-c8cf-9f75068dc12c"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ERROR:  role \"postgres\" already exists\n",
            "ALTER ROLE\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "QUERY_select='SELECT 2009 FROM table_name_50 WHERE 2011 = \"a\"'"
      ],
      "metadata": {
        "id": "E0ekOeVc8AZS"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import psycopg2 as ps\n",
        "import pandas as pd\n",
        "\n",
        "DB_NAME = \"postgres\"\n",
        "DB_USER = \"postgres\"\n",
        "DB_PASS = \"postgres\"\n",
        "DB_HOST = \"localhost\"\n",
        "DB_PORT = \"5432\""
      ],
      "metadata": {
        "id": "aaoBQ65q6kWH"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def table_creator(query):\n",
        "    import os\n",
        "    import psycopg2 as ps\n",
        "    import pandas as pd\n",
        "\n",
        "    DB_NAME = \"postgres\"\n",
        "    DB_USER = \"postgres\"\n",
        "    DB_PASS = \"postgres\"\n",
        "    DB_HOST = \"localhost\"\n",
        "    DB_PORT = \"5432\"\n",
        "\n",
        "    conn = ps.connect(database=DB_NAME,\n",
        "                  user=DB_USER,\n",
        "                  password=DB_PASS,\n",
        "                  host=DB_HOST,\n",
        "                  port=DB_PORT)\n",
        "\n",
        "    cur = conn.cursor() # creating a cursor\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "    # Wrap the execute command in a try-except block to handle potential errors\n",
        "    try:\n",
        "        cur.execute(\"\"\"\n",
        "                            %s\n",
        "                            \"\"\"%query)\n",
        "        conn.commit()\n",
        "        print(\"Table Created successfully\")\n",
        "    except Exception as e:\n",
        "        conn.rollback() # Rollback the transaction in case of an error\n",
        "        print(\"Error creating table:\", e)\n",
        "\n",
        "    conn.close()"
      ],
      "metadata": {
        "id": "UEU9tm1W6WOd"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "table_creator(QUERY_create)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hL-Kb3okSPyn",
        "outputId": "94978269-3900-4dbd-e05f-24448e99c454"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Error creating table: relation \"table_name_24\" already exists\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import psycopg2 as ps\n",
        "import pandas as pd\n",
        "\n",
        "def table_select(query):\n",
        "    conn = ps.connect(database=DB_NAME,\n",
        "                      user=DB_USER,\n",
        "                      password=DB_PASS,\n",
        "                      host=DB_HOST,\n",
        "                      port=DB_PORT)\n",
        "    #print(\"Database connected successfully\")\n",
        "\n",
        "    query = query.replace('\"', \"'\") # Replace double quotes with single quotes for potential date values\n",
        "\n",
        "    try:\n",
        "        #df = pd.read_sql_query(\"%s\"%query, con=conn)\n",
        "        print('rec: %'%df) # Print the resulting DataFrame\n",
        "        cursor = conn.cursor()\n",
        "        cursor.execute(query)\n",
        "        results = cursor.fetchall()\n",
        "\n",
        "        for row in results:\n",
        "            print(row)\n",
        "\n",
        "            print()\n",
        "\n",
        "            # Commit the transaction to save the changes\n",
        "            conn.commit()\n",
        "            #print(\"QUERY successfully\")\n",
        "            print()\n",
        "\n",
        "            # Close the cursor and connection\n",
        "            cursor.close()\n",
        "            conn.close()\n",
        "    except Exception as e:\n",
        "        #conn.rollback() # Rollback the transaction in case of an error\n",
        "        #print(\"Error executing query:\", e)\n",
        "        print('TABLE IS EMPTY')\n",
        "\n",
        "\n",
        "        conn.close()\n",
        "    #return bad"
      ],
      "metadata": {
        "id": "fLBlG3lmTvgb"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print()\n",
        "# PostGRES SQL Settings\n",
        "%cd /content/\n",
        "!sudo -u postgres psql -c \"ALTER USER postgres PASSWORD 'postgres'\"\n",
        "\n",
        "print('START: PG embedding COMPILATION')\n",
        "%cd /content/\n",
        "!git clone https://github.com/neondatabase/pg_embedding.git\n",
        "%cd /content/pg_embedding\n",
        "!make\n",
        "!make install # may need sudo\n",
        "print('END: PG embedding COMPILATION')\n",
        "print()\n",
        "\n",
        "#!sudo -u postgres psql -c \"DROP EXTENSION embedding\"\n",
        "!sudo -u postgres psql -c \"CREATE EXTENSION embedding\"\n",
        "#!sudo -u postgres psql -c \"DROP TABLE documents\"\n",
        "!sudo -u postgres psql -c \"CREATE TABLE documents(id integer PRIMARY KEY, embedding real[])\""
      ],
      "metadata": {
        "id": "qwkhZYHopR8u",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "007e85f8-e701-4bb2-8703-beecc4b71a30"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "/content\n",
            "ALTER ROLE\n",
            "START: PG embedding COMPILATION\n",
            "/content\n",
            "fatal: destination path 'pg_embedding' already exists and is not an empty directory.\n",
            "/content/pg_embedding\n",
            "make: Nothing to be done for 'all'.\n",
            "/bin/mkdir -p '/usr/lib/postgresql/14/lib'\n",
            "/bin/mkdir -p '/usr/share/postgresql/14/extension'\n",
            "/bin/mkdir -p '/usr/share/postgresql/14/extension'\n",
            "/usr/bin/install -c -m 755  embedding.so '/usr/lib/postgresql/14/lib/embedding.so'\n",
            "/usr/bin/install -c -m 644 .//embedding.control '/usr/share/postgresql/14/extension/'\n",
            "/usr/bin/install -c -m 644 .//embedding--0.3.5--0.3.6.sql .//embedding--0.3.5.sql .//embedding--0.3.6.sql  '/usr/share/postgresql/14/extension/'\n",
            "/bin/mkdir -p '/usr/lib/postgresql/14/lib/bitcode/embedding'\n",
            "/bin/mkdir -p '/usr/lib/postgresql/14/lib/bitcode'/embedding/\n",
            "/usr/bin/install -c -m 644 embedding.bc '/usr/lib/postgresql/14/lib/bitcode'/embedding/./\n",
            "/usr/bin/install -c -m 644 hnswalg.bc '/usr/lib/postgresql/14/lib/bitcode'/embedding/./\n",
            "/usr/bin/install -c -m 644 distfunc.bc '/usr/lib/postgresql/14/lib/bitcode'/embedding/./\n",
            "cd '/usr/lib/postgresql/14/lib/bitcode' && /usr/lib/llvm-14/bin/llvm-lto -thinlto -thinlto-action=thinlink -o embedding.index.bc embedding/embedding.bc embedding/hnswalg.bc embedding/distfunc.bc\n",
            "END: PG embedding COMPILATION\n",
            "\n",
            "ERROR:  extension \"embedding\" already exists\n",
            "ERROR:  relation \"documents\" already exists\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Dataset Settings"
      ],
      "metadata": {
        "id": "lCZstEQINH-m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "eval_dataset[0][\"messages\"][1][\"content\"]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "p6cWWnjunYw-",
        "outputId": "17272a26-683a-4399-e2af-d6ce4bf18847"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Name the interview for peru delegate'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "eval_dataset[0][\"messages\"][2][\"content\"]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "T3Sh1Er2nbJK",
        "outputId": "f0f9d3aa-c3da-4055-b34c-abe42af44712"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'SELECT interview FROM table_11674683_2 WHERE delegate = \"Peru\"'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "eval_dataset[0][\"messages\"][0]['content'][153:len(eval_dataset[0][\"messages\"][0]['content'])]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "uEtG6BvKnha-",
        "outputId": "43d590de-cc3d-4a32-d3d7-c2de4369fe10"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'CREATE TABLE table_11674683_2 (interview VARCHAR, delegate VARCHAR)'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Model Inference"
      ],
      "metadata": {
        "id": "9vQZl05tNYqz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from difflib import SequenceMatcher\n",
        "\n",
        "def similar(a, b):\n",
        "    return SequenceMatcher(None, a, b).ratio()\n",
        "\n",
        "similar(\"Apple\",\"Appel\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3UJHFwwqgf9r",
        "outputId": "a1cb92b7-69e2-4fc5-c0c1-fc14d1fabf3a"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from tqdm import tqdm\n",
        "from random import randint\n",
        "from datasets import load_dataset\n",
        "import psycopg2\n",
        "from psycopg2 import sql\n",
        "\n",
        "def evaluate(sample):\n",
        "    prompt = pipe.tokenizer.apply_chat_template(sample[\"messages\"][:2], tokenize=False, add_generation_prompt=True)\n",
        "    outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95, eos_token_id=pipe.tokenizer.eos_token_id, pad_token_id=pipe.tokenizer.pad_token_id)\n",
        "    predicted_answer = outputs[0]['generated_text'][len(prompt):].strip()\n",
        "\n",
        "    print()\n",
        "    print()\n",
        "    schema=sample[\"messages\"][0]['content']\n",
        "    schema_query=schema[153:len(schema)]\n",
        "    question = sample[\"messages\"][1][\"content\"]\n",
        "    original_answer = sample[\"messages\"][2][\"content\"]\n",
        "\n",
        "    if predicted_answer ==  original_answer:\n",
        "        print()\n",
        "        print()\n",
        "        print('SUCCESS!')\n",
        "        print()\n",
        "        print(f'QUESTION: {question}')\n",
        "        print()\n",
        "        print(f'SCHEMA QUERY: {schema_query}')\n",
        "        table_creator(schema_query)\n",
        "        print()\n",
        "        print(f'Generated Answer: {predicted_answer}')\n",
        "        table_select(predicted_answer)\n",
        "        print()\n",
        "        print(f'Original Answer: {original_answer}')\n",
        "        print()\n",
        "        return 1\n",
        "    else:\n",
        "        print()\n",
        "        print()\n",
        "        print('NO - SUCCESS!')\n",
        "        print()\n",
        "\n",
        "        ps=similar(predicted_answer,original_answer)\n",
        "        print(f'Generated Answer: {predicted_answer}')\n",
        "        print(f' Original Answer: {original_answer}')\n",
        "        print(f'        SIMILARY: {ps}')\n",
        "        print()\n",
        "        return 0\n",
        "\n",
        "success_rate = []\n",
        "number_of_eval_samples = 10\n",
        "\n",
        "# iterate over eval dataset and predict\n",
        "for n in tqdm(range(number_of_eval_samples)):\n",
        "    s=eval_dataset[n]\n",
        "    success_rate.append(evaluate(s))\n",
        "\n",
        "# compute accuracy\n",
        "accuracy = sum(success_rate)/len(success_rate)"
      ],
      "metadata": {
        "id": "g9Y0qGKoY-9p",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2dd6974f-b038-4266-edf3-352eb7d23e99"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            " 10%|█         | 1/10 [00:04<00:39,  4.44s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "SUCCESS!\n",
            "\n",
            "QUESTION: Name the interview for peru delegate\n",
            "\n",
            "SCHEMA QUERY: CREATE TABLE table_11674683_2 (interview VARCHAR, delegate VARCHAR)\n",
            "Table Created successfully\n",
            "\n",
            "Generated Answer: SELECT interview FROM table_11674683_2 WHERE delegate = \"Peru\"\n",
            "TABLE IS EMPTY\n",
            "\n",
            "Original Answer: SELECT interview FROM table_11674683_2 WHERE delegate = \"Peru\"\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r 20%|██        | 2/10 [00:09<00:38,  4.81s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "NO - SUCCESS!\n",
            "\n",
            "Generated Answer: SELECT game FROM table_23286158_8 WHERE high_rebounds = \"Andre Miller, Rudy Fernandez (7)\"\n",
            " Original Answer: SELECT game FROM table_23286158_8 WHERE high_rebounds = \"Andre Miller , Rudy Fernandez (7)\"\n",
            "        SIMILARY: 0.994475138121547\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r 30%|███       | 3/10 [00:14<00:35,  5.03s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "SUCCESS!\n",
            "\n",
            "QUESTION: What is the highest week when the game site was Verizon Wireless Arena, and the result was w 59–42?\n",
            "\n",
            "SCHEMA QUERY: CREATE TABLE table_name_69 (week INTEGER, game_site VARCHAR, result VARCHAR)\n",
            "Table Created successfully\n",
            "\n",
            "Generated Answer: SELECT MAX(week) FROM table_name_69 WHERE game_site = \"verizon wireless arena\" AND result = \"w 59–42\"\n",
            "TABLE IS EMPTY\n",
            "\n",
            "Original Answer: SELECT MAX(week) FROM table_name_69 WHERE game_site = \"verizon wireless arena\" AND result = \"w 59–42\"\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r 40%|████      | 4/10 [00:18<00:26,  4.47s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "SUCCESS!\n",
            "\n",
            "QUESTION: Who is the senator for district 24?\n",
            "\n",
            "SCHEMA QUERY: CREATE TABLE table_11948857_1 (member_senator VARCHAR, district VARCHAR)\n",
            "Table Created successfully\n",
            "\n",
            "Generated Answer: SELECT member_senator FROM table_11948857_1 WHERE district = 24\n",
            "TABLE IS EMPTY\n",
            "\n",
            "Original Answer: SELECT member_senator FROM table_11948857_1 WHERE district = 24\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r 50%|█████     | 5/10 [00:21<00:20,  4.12s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "SUCCESS!\n",
            "\n",
            "QUESTION: What race even has NASCAR Featherlite Modified Series as the championship?\n",
            "\n",
            "SCHEMA QUERY: CREATE TABLE table_name_26 (event VARCHAR, championship VARCHAR)\n",
            "Table Created successfully\n",
            "\n",
            "Generated Answer: SELECT event FROM table_name_26 WHERE championship = \"nascar featherlite modified series\"\n",
            "TABLE IS EMPTY\n",
            "\n",
            "Original Answer: SELECT event FROM table_name_26 WHERE championship = \"nascar featherlite modified series\"\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r 60%|██████    | 6/10 [00:24<00:15,  3.76s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "SUCCESS!\n",
            "\n",
            "QUESTION: What is the Region with a Catalog of alca-487?\n",
            "\n",
            "SCHEMA QUERY: CREATE TABLE table_name_89 (region VARCHAR, catalog VARCHAR)\n",
            "Table Created successfully\n",
            "\n",
            "Generated Answer: SELECT region FROM table_name_89 WHERE catalog = \"alca-487\"\n",
            "TABLE IS EMPTY\n",
            "\n",
            "Original Answer: SELECT region FROM table_name_89 WHERE catalog = \"alca-487\"\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r 70%|███████   | 7/10 [00:27<00:10,  3.41s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "SUCCESS!\n",
            "\n",
            "QUESTION: what season did the juventus win\n",
            "\n",
            "SCHEMA QUERY: CREATE TABLE table_name_54 (season VARCHAR, champion VARCHAR)\n",
            "Table Created successfully\n",
            "\n",
            "Generated Answer: SELECT season FROM table_name_54 WHERE champion = \"juventus\"\n",
            "TABLE IS EMPTY\n",
            "\n",
            "Original Answer: SELECT season FROM table_name_54 WHERE champion = \"juventus\"\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r 80%|████████  | 8/10 [00:28<00:05,  2.71s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "SUCCESS!\n",
            "\n",
            "QUESTION: How many film are there?\n",
            "\n",
            "SCHEMA QUERY: CREATE TABLE film (Id VARCHAR)\n",
            "Table Created successfully\n",
            "\n",
            "Generated Answer: SELECT COUNT(*) FROM film\n",
            "TABLE IS EMPTY\n",
            "\n",
            "Original Answer: SELECT COUNT(*) FROM film\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r 90%|█████████ | 9/10 [00:33<00:03,  3.27s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "SUCCESS!\n",
            "\n",
            "QUESTION: What is the episode number that was first broadcast on 3 September 2010?\n",
            "\n",
            "SCHEMA QUERY: CREATE TABLE table_23575917_4 (episode VARCHAR, first_broadcast VARCHAR)\n",
            "Table Created successfully\n",
            "\n",
            "Generated Answer: SELECT episode FROM table_23575917_4 WHERE first_broadcast = \"3 September 2010\"\n",
            "TABLE IS EMPTY\n",
            "\n",
            "Original Answer: SELECT episode FROM table_23575917_4 WHERE first_broadcast = \"3 September 2010\"\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 10/10 [00:38<00:00,  3.83s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "SUCCESS!\n",
            "\n",
            "QUESTION: If the water square milage is 1.701, what is the land square milage?\n",
            "\n",
            "SCHEMA QUERY: CREATE TABLE table_18600760_7 (land___sqmi__ VARCHAR, water__sqmi_ VARCHAR)\n",
            "Table Created successfully\n",
            "\n",
            "Generated Answer: SELECT land___sqmi__ FROM table_18600760_7 WHERE water__sqmi_ = \"1.701\"\n",
            "TABLE IS EMPTY\n",
            "\n",
            "Original Answer: SELECT land___sqmi__ FROM table_18600760_7 WHERE water__sqmi_ = \"1.701\"\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print()\n",
        "#print(f\"Accuracy: {accuracy*100:.2f}%\")\n",
        "print(f\"Accuracy (Eval dataset and predict) for a sample of {number_of_eval_samples}: {accuracy*100:.2f}%\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "U6ArykBRZGLp",
        "outputId": "1600671a-2e4a-45a4-98ce-ee323d2a994e"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Accuracy (Eval dataset and predict) for a sample of 10: 90.00%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Model Evaluation - Kernel"
      ],
      "metadata": {
        "id": "wEsmtqGJ1pxE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Count Hidden Layers and Neurons (Before Evaluation)\n",
        "if hasattr(model, 'base_model'):\n",
        "    llama_model = model.base_model\n",
        "else:\n",
        "    llama_model = model\n",
        "\n",
        "# Count hidden layers of type LlamaDecoderLayer\n",
        "num_hidden_layers = llama_model.config.num_hidden_layers\n",
        "#print(num_hidden_layers)\n",
        "\n",
        "# Estimate neurons (this is very simplified, as explained earlier)\n",
        "num_neurons = num_hidden_layers * llama_model.config.hidden_size\n",
        "\n",
        "print(f\"Number of hidden layers in the model: {num_hidden_layers}\")\n",
        "print(f\"Approximate number of neurons (simplified): {num_neurons}\")"
      ],
      "metadata": {
        "id": "7kU-lpcA1i3j",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c2b39af0-e491-4728-d305-c6dc21fb8862"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of hidden layers in the model: 32\n",
            "Approximate number of neurons (simplified): 131072\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "torch.cuda.empty_cache()"
      ],
      "metadata": {
        "id": "Xl-M2zL02pnh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Load our test dataset\n",
        "eval_dataset = load_dataset(\"json\", data_files=\"/content/test_dataset.json\", split=\"train\")\n",
        "reduced_size = 10\n",
        "eval_dataset = eval_dataset.shuffle(seed=42).select(range(reduced_size))"
      ],
      "metadata": {
        "id": "MdkVe49IBkni"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "eval_dataset"
      ],
      "metadata": {
        "id": "zrFcALOL3Y4y",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ff99140d-57ac-49ad-b840-28cceb743fd4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Dataset({\n",
              "    features: ['messages'],\n",
              "    num_rows: 10\n",
              "})"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "eval_dataset[0][\"messages\"][0]['content']"
      ],
      "metadata": {
        "id": "2iyxLiME4r9s",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "be421042-0b57-4a56-917b-3efdfe31deb8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'You are an text to SQL query translator. Users will ask you questions in English and you will generate a SQL query based on the provided SCHEMA.\\nSCHEMA:\\nCREATE TABLE table_name_78 (position VARCHAR, player VARCHAR)'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "max_length = 10\n",
        "all_input_ids = []\n",
        "all_attention_masks = []\n",
        "\n",
        "for item in eval_dataset:\n",
        "    messages = item['messages']\n",
        "\n",
        "    # Concatenate the 'content' of all messages into a single string\n",
        "    text = \" \".join([msg['content'] for msg in messages])\n",
        "    #print()\n",
        "    #print(text)\n",
        "    #print()\n",
        "\n",
        "    tokenized = tokenizer(text, padding=\"max_length\", truncation=True, max_length=max_length, return_tensors=\"pt\")\n",
        "    all_input_ids.append(tokenized[\"input_ids\"][0])\n",
        "    all_attention_masks.append(tokenized[\"attention_mask\"][0])\n",
        "\n",
        "\n",
        "input_ids = torch.stack(all_input_ids)\n",
        "attention_masks = torch.stack(all_attention_masks)"
      ],
      "metadata": {
        "id": "84VruCFtLEXZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Now you have input_ids, attention_masks, and labels as tensors with compatible shapes\n",
        "print(input_ids.shape)\n",
        "print(attention_masks.shape)"
      ],
      "metadata": {
        "id": "-4eAUNAiLFrV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dcdaa965-a64e-453f-f45b-ac59fad6f324"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([10, 10])\n",
            "torch.Size([10, 10])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "torch.cuda.empty_cache()"
      ],
      "metadata": {
        "id": "wx1GS8ANZMLO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "del model\n",
        "del tokenizer\n",
        "torch.cuda.empty_cache()"
      ],
      "metadata": {
        "id": "meb7f_r6m0Bv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "source": [
        "import torch\n",
        "from torch.utils.data import DataLoader, TensorDataset\n",
        "from datasets import load_dataset\n",
        "from peft import PeftModel\n",
        "import evaluate\n",
        "from tqdm import tqdm\n",
        "from transformers import AutoTokenizer, AutoModelForCausalLM\n",
        "\n",
        "# Constants\n",
        "BATCH_SIZE = 8\n",
        "MAX_LENGTH = 10\n",
        "peft_model_id = \"frankmorales2020/Mistral-7B-text-to-sql-flash-attention-2-dataeval\"\n",
        "data_files = \"/content/test_dataset.json\"\n",
        "reduced_size = 10\n",
        "\n",
        "# Load tokenizer (using tokenizer from the PEFT model)\n",
        "tokenizer = AutoTokenizer.from_pretrained(peft_model_id)\n",
        "\n",
        "# Load base model\n",
        "base_model = AutoModelForCausalLM.from_pretrained(\n",
        "    \"mistralai/Mistral-7B-v0.3\",\n",
        "    torch_dtype=torch.float16,\n",
        "    device_map='auto'\n",
        ")\n",
        "\n",
        "# Resize the token embeddings to match the PEFT vocabulary\n",
        "base_model.resize_token_embeddings(len(tokenizer))\n",
        "\n",
        "# Load PEFT model (using the base_model object)\n",
        "model = PeftModel.from_pretrained(base_model, peft_model_id)\n",
        "model.eval()\n",
        "\n",
        "# Ensure model is on the GPU\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "model.to(device)\n",
        "\n",
        "# Load your test dataset\n",
        "eval_dataset = load_dataset(\"json\", data_files=data_files, split=\"train\")\n",
        "eval_dataset = eval_dataset.shuffle(seed=42).select(range(reduced_size))\n",
        "\n",
        "# Tokenization and Tensor Creation\n",
        "all_input_ids = []\n",
        "all_attention_masks = []\n",
        "for item in eval_dataset:\n",
        "    messages = item['messages']\n",
        "    # Concatenate the 'content' of all messages into a single string\n",
        "    text = \" \".join([msg['content'] for msg in messages])\n",
        "\n",
        "    tokenized = tokenizer(text, padding=\"max_length\", truncation=True, max_length=MAX_LENGTH, return_tensors=\"pt\")\n",
        "    all_input_ids.append(tokenized[\"input_ids\"][0])\n",
        "    all_attention_masks.append(tokenized[\"attention_mask\"][0])\n",
        "\n",
        "input_ids = torch.stack(all_input_ids)\n",
        "attention_masks = torch.stack(all_attention_masks)\n",
        "\n",
        "# Create TensorDataset from your tensors\n",
        "eval_dataset = TensorDataset(input_ids, attention_masks)\n",
        "\n",
        "# Create DataLoader\n",
        "eval_dataloader = DataLoader(eval_dataset, batch_size=BATCH_SIZE, shuffle=False)\n",
        "\n",
        "# Evaluation function (Manually calculating perplexity)\n",
        "def evaluate_model(model, eval_dataloader):\n",
        "    model.eval()\n",
        "    losses = []\n",
        "    for batch in tqdm(eval_dataloader, desc=\"Evaluating\"):\n",
        "        try:\n",
        "            # Convert batch to device (assuming it's a list/tuple of tensors)\n",
        "            batch = tuple(t.to(device) for t in batch)\n",
        "\n",
        "            with torch.no_grad():\n",
        "                outputs = model(input_ids=batch[0], attention_mask=batch[1], labels=batch[0])  # Adjust based on your batch structure\n",
        "            loss = outputs.loss\n",
        "            losses.append(loss.item())\n",
        "\n",
        "        except RuntimeError as e:\n",
        "            if \"out of memory\" in str(e):\n",
        "                print(\"WARNING: Ran out of memory. Consider reducing batch size or model complexity.\")\n",
        "                return None  # Exit early if out of memory\n",
        "            else:\n",
        "                raise e\n",
        "\n",
        "    try:\n",
        "        perplexity = torch.exp(torch.tensor(losses).mean())\n",
        "        return perplexity\n",
        "    except OverflowError:\n",
        "        print(\"WARNING: Overflow error while calculating perplexity. Loss values might be too large.\")\n",
        "        return None\n",
        "\n",
        "\n",
        "# Perform Evaluation\n",
        "results = evaluate_model(model, eval_dataloader)"
      ],
      "cell_type": "code",
      "metadata": {
        "id": "chWV971BnLqi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        " The perplexity of 10.40 achieved on the dataset indicates that the fine-tuned Mistral-7B model has a reasonable understanding of natural language and SQL syntax. However, further evaluation using task-specific metrics is necessary to fully assess the model's effectiveness in real-world scenarios."
      ],
      "metadata": {
        "id": "-3M0INi4Cyer"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(f\"Perplexity: {results:.2f}\")"
      ],
      "metadata": {
        "id": "oWgLgFuX0AzV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a8301b4c-14b8-4d93-dbf8-6dcb6cbbeec9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Perplexity: 10.40\n"
          ]
        }
      ]
    }
  ]
}