notebooks/llmu/Fine_Tuning_for_Classify.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/cohere-ai/notebooks/blob/main/notebooks/llmu/Fine_Tuning_for_Classify.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NOoAYv5ThsHg"
   },
   "source": [
    "# Fine-Tuning for Classify"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AVf0QmYJiJ22"
   },
   "source": [
    "Cohere's large language models (LLMs) have been pre-trained with a vast amount of training data, allowing them to capture how words are being used and how their meaning changes depending on the context. \n",
    "\n",
    "However, there are many cases in which getting the best model performance requires performing an additional round of training on custom user data. Creating a custom model using this process is called fine-tuning.\n",
    "\n",
    "Fine-tuning allows you to alter the model itself and customize it to excel at your specific task. In this notebook, you'll learn how to fine-tune a model for classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "We'll do the following steps:\n",
    "- **Step 1: Prepare the Dataset** - Download the dataset, select a subset, and prepare it for the Classify endpoint.\n",
    "- **Step 2: Fine-Tune the Model** - Kick off a fine-tuning job, and confirm when the model has completed training.\n",
    "- **Step 3: Use/Evaluate the Fine-Tuned Model** - Evaluate the fine-tuned model's performance on the test dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QVrCaQDaQNoM"
   },
   "source": [
    "## Setup\n",
    "\n",
    "We'll start by installing the tools we'll need and then importing them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "29uwe-jzJ9rh"
   },
   "outputs": [],
   "source": [
    "! pip install cohere -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "y9-RyLu7KHII"
   },
   "outputs": [],
   "source": [
    "# Import the required modules\n",
    "import os\n",
    "import json\n",
    "import cohere\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from cohere import ClassifyExample\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import f1_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qrdm__9-AT04"
   },
   "source": [
    "Fill in your Cohere API key in the next cell. To do this, begin by [signing up to Cohere](https://os.cohere.ai/) (for free!) if you haven't yet. Then get your API key [here](https://dashboard.cohere.com/api-keys)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "AW4RsVSE4j74"
   },
   "outputs": [],
   "source": [
    "# Paste your API key here. Remember to not share publicly\n",
    "co = cohere.Client(\"COHERE_API_KEY\") # Insert your Cohere API key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "laIkCcKa40PR"
   },
   "source": [
    "## Step 1: Prepare the Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0Fil-WkzipV_"
   },
   "source": [
    "We'll use the [Airline Travel Information System (ATIS)](https://www.kaggle.com/datasets/hassanamin/atis-airlinetravelinformationsystem?select=atis_intents_train.csv) intent classification dataset [[source](https://aclanthology.org/H90-1021/)]. For demonstration purposes, we’ll take just a small portion of the dataset: 1,000 data points in total."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 224
    },
    "id": "gI7wMIdOrbiK",
    "outputId": "5cb28300-2cbb-40da-80a4-fe9f22241f74"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 1000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>intent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>i want to fly from boston at 838 am and arriv...</td>\n",
       "      <td>atis_flight</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>what flights are available from pittsburgh to...</td>\n",
       "      <td>atis_flight</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>what is the arrival time in san francisco for...</td>\n",
       "      <td>atis_flight_time</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>cheapest airfare from tacoma to orlando</td>\n",
       "      <td>atis_airfare</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>round trip fares from pittsburgh to philadelp...</td>\n",
       "      <td>atis_airfare</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               query            intent\n",
       "0   i want to fly from boston at 838 am and arriv...       atis_flight\n",
       "1   what flights are available from pittsburgh to...       atis_flight\n",
       "2   what is the arrival time in san francisco for...  atis_flight_time\n",
       "3            cheapest airfare from tacoma to orlando      atis_airfare\n",
       "4   round trip fares from pittsburgh to philadelp...      atis_airfare"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the dataset to a dataframe\n",
    "df = pd.read_csv('https://raw.githubusercontent.com/cohere-ai/notebooks/main/notebooks/data/atis_subset.csv', names=['query','intent'])\n",
    "print(\"Number of entries:\", len(df))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CJIkKk9-AT07"
   },
   "source": [
    "The first thing we need is to create a training dataset, to be used for building the classifier, and a test dataset, to be used for testing the classifier performance. We will use 800 and 200 data points for these datasets, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "cM-2KGlQe80t"
   },
   "outputs": [],
   "source": [
    "# Split the dataset into training and test portions\n",
    "df_train, df_test = train_test_split(df, test_size=200, random_state=21)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xSfHL3OnAT07"
   },
   "source": [
    "Our goal is to train the classifier so it can predict the class of a new customer inquiry out of eight classes, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-J-T5ZKP5s0t",
    "outputId": "5d3eb630-2822-4643-f7ae-e5cd597a6365"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['atis_flight', 'atis_airfare', 'atis_ground_service', 'atis_flight_time', 'atis_airline', 'atis_quantity', 'atis_abbreviation', 'atis_aircraft']\n"
     ]
    }
   ],
   "source": [
    "# View the list of all available categories\n",
    "intents = df_train.intent.unique().tolist()\n",
    "print(intents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8k5CoStyVEGe"
   },
   "source": [
    "We transform the data to JSONL format to match the style expected by the Classification endpoint ([documentation](https://docs.cohere.com/docs/classify-preparing-the-data))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DidOiKmnU9nn",
    "outputId": "1947518f-ccdb-448e-e8c7-6b2e1b6404d8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating jsonl file ...\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "def create_classification_data(text, label):\n",
    "    formatted_data = {\n",
    "        \"text\": text,\n",
    "        \"label\": label\n",
    "    }\n",
    "    return formatted_data\n",
    "\n",
    "if not os.path.isfile(\"data.jsonl\"):\n",
    "    print(\"Creating jsonl file ...\")\n",
    "    with open(\"data.jsonl\", 'w+') as file:\n",
    "        for row in df_train.itertuples():\n",
    "            formatted_data = create_classification_data(row.query, row.intent)\n",
    "            file.write(json.dumps(formatted_data) + '\\n')\n",
    "        file.close()\n",
    "        print(\"Done\")\n",
    "else:\n",
    "    print(\"data.jsonl file already exists\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RkxARuVbZJww"
   },
   "source": [
    "The JSONL file has a row for each example.  Each example has \"text\" and \"label\" fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "-2l3HpVEZHEe",
    "outputId": "530880a8-6364-4fef-a605-f601aad990fa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"text\": \" what flights go from newark to boston after 5 pm\", \"label\": \"atis_flight\"}\n",
      "{\"text\": \" can you show me economy class one way fares for flights from oakland to dallas on december sixteenth\", \"label\": \"atis_airfare\"}\n",
      "{\"text\": \" show me the latest flight from salt lake city to phoenix\", \"label\": \"atis_flight\"}\n",
      "{\"text\": \" pittsburgh to denver\", \"label\": \"atis_flight\"}\n",
      "{\"text\": \" show me all flights from san francisco to atlanta\", \"label\": \"atis_flight\"}\n"
     ]
    }
   ],
   "source": [
    "# Load jsonl file and print first 5 lines\n",
    "N = 5\n",
    "with open(\"data.jsonl\") as f:\n",
    "    for i in range(0, N):\n",
    "        print(f.readline(), end = '')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2hg1F5GfZRZa"
   },
   "source": [
    "## Step 2: Fine-Tune the Model\n",
    "\n",
    "We kick off a fine-tuning job by navigating to the [fine-tuning tab of the Dashboard](https://dashboard.cohere.com/fine-tuning).  Under \"Classify\", click on \"Create a Classify model\".\n",
    "\n",
    "<img src=\"https://files.readme.io/48dad78-cohere_dashboard.png\">\n",
    "\n",
    "Next, upload the `.jsonl` file you just created as the training set by clicking on the \"TRAINING SET\" button. When ready, click on \"Review data\" to proceed to the next step.\n",
    "\n",
    "<img src=\"https://files.readme.io/9c83f64-classify_data.png\">\n",
    "\n",
    "Then, you'll see a preview of how the model will ingest your data. If anything is wrong with the data, the page will also provide suggested changes to fix the training file. Otherwise, if everything looks good, you can proceed to the next step.\n",
    "\n",
    "<img src=\"https://files.readme.io/d14803d-atis_review_data.png\">\n",
    "\n",
    "Finally, you'll see a page where you'll provide a nickname to your model. We used `atis-classify-ft` as the nickname for our model. Under \"BASE MODEL\", ensure \"english\" is selected.\n",
    "\n",
    "<img src=\"https://files.readme.io/456ba27-model_nickname_select.png\">\n",
    "\n",
    "Then click on \"Start training\" to kick off the fine-tuning process. This will navigate you to a page where you can monitor the status of the model. A model that has finished fine-tuning will show the status as `READY`.\n",
    "\n",
    "<img src=\"https://files.readme.io/48a2e02-atis_model_ready.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DJDDq9kdCMvH"
   },
   "source": [
    "## Step 3: Use/Evaluate the Fine-Tuned Model\n",
    "\n",
    "Once the model has finished fine-tuning, it’s time to evaluate its performance. Navigate to the API tab of the fine-tuned model. There, you'll see the model ID that you should use when calling `co.classify()`.\n",
    "\n",
    "<img src=\"https://files.readme.io/580ff8c-get_model_ID.png\">\n",
    "\n",
    "We fill in the model ID to generate test predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "Mq8sj5Fn0OuE"
   },
   "outputs": [],
   "source": [
    "# Generate classification predictions on the test dataset using the finetuned model\n",
    "\n",
    "# Classification function\n",
    "def classify_text_finetune(texts):\n",
    "    classifications = co.classify(\n",
    "        model='b2c94ac3-7a74-4de7-a11a-9808a3b8ef59-ft',\n",
    "        inputs=texts,\n",
    "        examples=None\n",
    "    ).classifications\n",
    "    return [c.predictions[0] for c in classifications]\n",
    "\n",
    "# Create batches of texts and classify them\n",
    "BATCH_SIZE = 90 # The API accepts a maximum of 96 inputs\n",
    "y_pred = []\n",
    "for i in range(0, len(df_test), BATCH_SIZE):\n",
    "    batch_texts = df_test[\"query\"][i:i+BATCH_SIZE].tolist()\n",
    "    y_pred.extend(classify_text_finetune(batch_texts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P0jYvZK9AT1E"
   },
   "source": [
    "Next, we calculate the model's test accuracy and F1 score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "qJ961ixH9kbf",
    "outputId": "610b9ae4-3c62-4315-a572-ec1c81167eec",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 98.00\n",
      "F1-score: 98.05\n"
     ]
    }
   ],
   "source": [
    "# Compute metrics on the test dataset\n",
    "accuracy = accuracy_score(df_test[\"intent\"], y_pred)\n",
    "f1 = f1_score(df_test[\"intent\"], y_pred, average='weighted')\n",
    "\n",
    "print(f'Accuracy: {100*accuracy:.2f}')\n",
    "print(f'F1-score: {100*f1:.2f}')"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "vscode": {
   "interpreter": {
    "hash": "1fb8019e3560b882083e525615cf48e713d3a7345a15eb723d805e91aa410aac"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}