Phi3_ORPO_Unsloth.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wGcFRVNqqxsc"
      },
      "source": [
        "**To** run this, press \"*Runtime*\" and press \"*Run all*\" on a **free** Tesla T4 Google Colab instance!\n",
        "<div class=\"align-center\">\n",
        "  <a href=\"https://github.com/unslothai/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\" width=\"115\"></a>\n",
        "  <a href=\"https://discord.gg/u54VK8m8tk\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Discord button.png\" width=\"145\"></a>\n",
        "  <a href=\"https://ko-fi.com/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Kofi button.png\" width=\"145\"></a></a> Join Discord if you need help + ⭐ <i>Star us on <a href=\"https://github.com/unslothai/unsloth\">Github</a> </i> ⭐\n",
        "</div>\n",
        "\n",
        "To install Unsloth on your own computer, follow the installation instructions on our Github page [here](https://github.com/unslothai/unsloth#installation-instructions---conda).\n",
        "\n",
        "You will learn how to do [data prep](#Data), how to [train](#Train), how to [run the model](#Inference), & [how to save it](#Save) (eg for Llama.cpp).\n",
        "\n",
        "[NEW] ORPO support is finally here thanks to [oKatanaaa](https://github.com/oKatanaaa) and [AT&Dev](https://huggingface.co/AtAndDev)\n",
        "\n",
        "ORPO merges the SFT and DPO steps into 1. Before one had to do a SFT, then DPO. ORPO now requires only 1 step."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_repo=\"your_username/phi-3-orpo-v8\""
      ],
      "metadata": {
        "id": "LoiX9EEttcXS"
      },
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8lSNa9FYvmA7"
      },
      "outputs": [],
      "source": [
        "#%%capture\n",
        "import torch\n",
        "major_version, minor_version = torch.cuda.get_device_capability()\n",
        "# Must install separately since Colab has torch 2.2.1, which breaks packages\n",
        "!pip install \"unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git\"\n",
        "if major_version >= 8:\n",
        "    # Use this for new GPUs like Ampere, Hopper GPUs (RTX 30xx, RTX 40xx, A100, H100, L40)\n",
        "    !pip install --no-deps packaging ninja einops flash-attn xformers trl peft accelerate bitsandbytes\n",
        "else:\n",
        "    # Use this for older GPUs (V100, Tesla T4, RTX 20xx)\n",
        "    !pip install --no-deps xformers==0.0.25.post1 trl peft accelerate bitsandbytes\n",
        "pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4dfwD72Ct1_T"
      },
      "source": [
        "* We support Llama, Mistral, CodeLlama, TinyLlama, Vicuna, Open Hermes etc\n",
        "* And Yi, Qwen ([llamafied](https://huggingface.co/models?sort=trending&search=qwen+llama)), Deepseek, all Llama, Mistral derived archs.\n",
        "* We support 16bit LoRA or 4bit QLoRA. Both 2x faster.\n",
        "* `max_seq_length` can be set to anything, since we do automatic RoPE Scaling via [kaiokendev's](https://kaiokendev.github.io/til) method.\n",
        "* [**NEW**] With [PR 26037](https://github.com/huggingface/transformers/pull/26037), we support downloading 4bit models **4x faster**! [Our repo](https://huggingface.co/unsloth) has Llama, Mistral 4bit models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2eSvM9zX_2d3"
      },
      "outputs": [],
      "source": [
        "from unsloth import FastLanguageModel\n",
        "import torch\n",
        "max_seq_length = 4096 # Choose any! We auto support RoPE Scaling internally!\n",
        "dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+\n",
        "load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.\n",
        "\n",
        "# 4bit pre quantized models we support for 4x faster downloading + no OOMs.\n",
        "fourbit_models = [\n",
        "    \"unsloth/mistral-7b-bnb-4bit\",\n",
        "    \"unsloth/mistral-7b-instruct-v0.2-bnb-4bit\",\n",
        "    \"unsloth/llama-3-8b-bnb-4bit\", # [NEW] 15 Trillion token Llama-3\n",
        "] # More models at https://huggingface.co/unsloth\n",
        "\n",
        "model, tokenizer = FastLanguageModel.from_pretrained(\n",
        "    model_name = \"vonjack/Phi-3-mini-4k-instruct-LLaMAfied\",\n",
        "    max_seq_length = max_seq_length,\n",
        "    dtype = dtype,\n",
        "    load_in_4bit = load_in_4bit,\n",
        "    # token = \"hf_...\", # use one if using gated models like meta-llama/Llama-2-7b-hf\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_H6EWbe-t_bk"
      },
      "source": [
        "We now add LoRA adapters so we only need to update 1 to 10% of all parameters!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "6bZsfBuZDeCL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f5a55dc0-c831-455b-a952-dfe8d2b09984"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Unsloth 2024.4 patched 32 layers with 32 QKV layers, 32 O layers and 32 MLP layers.\n"
          ]
        }
      ],
      "source": [
        "model = FastLanguageModel.get_peft_model(\n",
        "    model,\n",
        "    r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128\n",
        "    target_modules = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n",
        "                      \"gate_proj\", \"up_proj\", \"down_proj\",],\n",
        "    lora_alpha = 16,\n",
        "    lora_dropout = 0, # Supports any, but = 0 is optimized\n",
        "    bias = \"none\",    # Supports any, but = \"none\" is optimized\n",
        "    # [NEW] \"unsloth\" uses 30% less VRAM, fits 2x larger batch sizes!\n",
        "    use_gradient_checkpointing = \"unsloth\", # True or \"unsloth\" for very long context\n",
        "    random_state = 3407,\n",
        "    use_rslora = False,  # We support rank stabilized LoRA\n",
        "    loftq_config = None, # And LoftQ\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FNfc6OLOuuyZ"
      },
      "source": [
        "<a name=\"Data\"></a>\n",
        "### Data Prep\n",
        "We now use a special ORPO style dataset from [recipe-research](https://huggingface.co/datasets/reciperesearch/dolphin-sft-v0.1-preference).\n",
        "\n",
        "You need at least 3 columns:\n",
        "* Instruction\n",
        "* Accepted\n",
        "* Rejected\n",
        "\n",
        "For example:\n",
        "* Instruction: \"What is 2+2?\"\n",
        "* Accepted: \"The answer is 4\"\n",
        "* Rejected: \"The answer is 5\"\n",
        "\n",
        "The goal of ORPO is to penalize the \"rejected\" samples, and increase the likelihood of \"accepted\" samples. [recipe-research](https://huggingface.co/datasets/reciperesearch/dolphin-sft-v0.1-preference) essentially used Mistral to generate the \"rejected\" responses, and used GPT-4 to generated the \"accepted\" responses."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from unsloth.chat_templates import get_chat_template\n",
        "\n",
        "EOS_TOKEN = tokenizer.eos_token\n",
        "\n",
        "phi3_template = \\\n",
        "    \"{% if messages[0]['role'] == 'system' %}\"\\\n",
        "        \"{{ '<|system|>' + messages[0]['content'] + '<|end|>\\n' }}\"\\\n",
        "        \"{% set loop_messages = messages[1:] %}\"\\\n",
        "    \"{% else %}\"\\\n",
        "        \"{% set loop_messages = messages %}\"\\\n",
        "    \"{% endif %}\"\\\n",
        "    \"{% for message in loop_messages %}\"\\\n",
        "        \"{% if message['role'] == 'user' %}\"\\\n",
        "            \"{{ '<|user|>' + message['content'] + '<|end|>\\n' }}\"\\\n",
        "        \"{% elif message['role'] == 'assistant' %}\"\\\n",
        "            \"{{ '<|assistant|>' + message['content'] + '<|end|>\\n' }}\"\\\n",
        "        \"{% else %}\"\\\n",
        "            \"{{ raise_exception('Only system, user, and assistant roles are supported!') }}\"\\\n",
        "        \"{% endif %}\"\\\n",
        "    \"{% endfor %}\"\\\n",
        "    \"{% if add_generation_prompt %}\"\\\n",
        "        \"{{ '<|assistant|>' }}\"\\\n",
        "    \"{% endif %}\"\n",
        "\n",
        "tokenizer = get_chat_template(\n",
        "        tokenizer,\n",
        "        chat_template = (phi3_template, EOS_TOKEN,), # You must provide a template and EOS token\n",
        "        mapping={\"role\": \"role\", \"content\": \"content\", \"system\": \"system\", \"user\": \"user\", \"assistant\": \"assistant\"}, #\n",
        "        map_eos_token = True, # Maps <|im_end|> to </s> instead\n",
        "    )\n",
        "\n",
        "from datasets import load_dataset\n",
        "\n",
        "def format_prompt(sample):\n",
        "    system = sample[\"system\"]\n",
        "    question = sample[\"question\"]\n",
        "    accepted = sample[\"chosen\"] + EOS_TOKEN\n",
        "    rejected = sample[\"rejected\"] + EOS_TOKEN\n",
        "\n",
        "    conversation = [{\"role\": \"system\", \"content\": system}] if system.strip() else []\n",
        "    conversation.append({\"role\": \"user\", \"content\": question})\n",
        "\n",
        "    try:\n",
        "        prompt = tokenizer.apply_chat_template(conversation, phi3_template, tokenize=False, add_generation_prompt=True)\n",
        "        return {\"prompt\": prompt, \"chosen\": accepted, \"rejected\": rejected}\n",
        "    except Exception as e:\n",
        "        print(f\"Error processing conversation: {e}\\nConversation: {conversation}\")\n",
        "        return sample\n",
        "\n",
        "dataset = load_dataset(\"johannhartmann/mistralorpo\")[\"train\"]\n",
        "\n",
        "try:\n",
        "    dataset_mapped = dataset.map(format_prompt, remove_columns=dataset.column_names)\n",
        "except StopIteration as e:\n",
        "    print(str(e))\n",
        "\n",
        "dataset = dataset_mapped"
      ],
      "metadata": {
        "id": "K9dDSDrXlfMs"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7PH-7Cudu8Ja"
      },
      "source": [
        "Let's print out some examples to see how the dataset should look like"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oF63zQqNlNJC",
        "outputId": "8db74389-3877-443d-b1b8-3491a4b20849"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'chosen': 'Das Mittsommerhaus ist ein moderat preisiges Restaurant in der Nähe von All Bar One, bietet chinesisches Essen an und hat eine Kundenbewertung von 3 von 5.<|end|>', 'rejected': '\\nMidsummer House is a distinctive eatType restaurant, renowned for serving an exquisite range of Chinesish cuisine. The mid-priced, moderat Midsummer House price span appeals to a broad clientele, making it an accessible yet special dining destination. This establishment, located in proximity to All Bar One, boasts a customer rating of 3 out of 5 stars, indicating a satisfactory yet not exceptional dining experience.<|end|>', 'prompt': '<|system|>Sie sind ein KI-Assistent. Sie erhalten eine Aufgabe. Sie müssen eine detaillierte und lange Antwort generieren.<|end|>\\n<|user|>Generieren Sie einen Satz mit etwa fünfzehn Wörtern, der all diese Daten beschreibt: Midsummer House eatType restaurant; Mittsommerhaus-Essen Chinesisch; Mittsommerhaus-Preisspanne moderat; Midsummer House Kundenbewertung: 3 von 5; Mittsommerhaus in der Nähe von All Bar One<|end|>\\n<|assistant|>'}\n",
            "INSTRUCTION: ==================================================\n",
            "('<|system|>Sie sind ein KI-Assistent. Sie erhalten eine Aufgabe. Sie müssen '\n",
            " 'eine detaillierte und lange Antwort generieren.<|end|>\\n'\n",
            " '<|user|>Generieren Sie einen Satz mit etwa fünfzehn Wörtern, der all diese '\n",
            " 'Daten beschreibt: Midsummer House eatType restaurant; Mittsommerhaus-Essen '\n",
            " 'Chinesisch; Mittsommerhaus-Preisspanne moderat; Midsummer House '\n",
            " 'Kundenbewertung: 3 von 5; Mittsommerhaus in der Nähe von All Bar One<|end|>\\n'\n",
            " '<|assistant|>')\n",
            "ACCEPTED: ==================================================\n",
            "('Das Mittsommerhaus ist ein moderat preisiges Restaurant in der Nähe von All '\n",
            " 'Bar One, bietet chinesisches Essen an und hat eine Kundenbewertung von 3 von '\n",
            " '5.<|end|>')\n",
            "REJECTED: ==================================================\n",
            "('\\n'\n",
            " 'Midsummer House is a distinctive eatType restaurant, renowned for serving an '\n",
            " 'exquisite range of Chinesish cuisine. The mid-priced, moderat Midsummer '\n",
            " 'House price span appeals to a broad clientele, making it an accessible yet '\n",
            " 'special dining destination. This establishment, located in proximity to All '\n",
            " 'Bar One, boasts a customer rating of 3 out of 5 stars, indicating a '\n",
            " 'satisfactory yet not exceptional dining experience.<|end|>')\n"
          ]
        }
      ],
      "source": [
        "import pprint\n",
        "row = dataset[1]\n",
        "print (row)\n",
        "print('INSTRUCTION: ' + '=' * 50)\n",
        "pprint.pprint(row[\"prompt\"])\n",
        "print('ACCEPTED: ' + '=' * 50)\n",
        "pprint.pprint(row[\"chosen\"])\n",
        "print('REJECTED: ' + '=' * 50)\n",
        "pprint.pprint(row[\"rejected\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "oqyleKojqgDq"
      },
      "outputs": [],
      "source": [
        "# Enable reward modelling stats\n",
        "from unsloth import PatchDPOTrainer\n",
        "PatchDPOTrainer()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J1ZlnJpkxIuV"
      },
      "source": [
        "<a name=\"Train\"></a>\n",
        "### Train the model\n",
        "Now let's use Huggingface TRL's `ORPOTrainer`! More docs here: [TRL ORPO docs](https://huggingface.co/docs/trl/main/en/orpo_trainer). We do 60 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`. We also support TRL's `DPOTrainer`!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "TkDMnoPiwH3P"
      },
      "outputs": [],
      "source": [
        "turns = 100\n",
        "dataset = dataset.select(range(turns))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 120,
          "referenced_widgets": [
            "2d3b783e5bbd4f8da1c1de4e1e348e70",
            "3095238a31bb450ab2c73a25adcc5987",
            "a95cc5a7a1834733b6ff6b85b5f0ef7a",
            "f11b16391ce5461c8a0b233477597fe6",
            "3d86c8bf6f3949f8baffb0ff7b6727a7",
            "8fc4d0ad47e349a3a3f5c1c675553b48",
            "e9fb2ecf86164771ad02772930286249",
            "e0ff5af098bc4d82b7107e89a11bdef9",
            "28e8c875bb4f4314a0685f81e740fe76",
            "4548a8c803114d2dacd1b8fd5b840403",
            "8a67559f27ef46ce8c0d9d88f9d5777a"
          ]
        },
        "id": "QtoqUw80QDV0",
        "outputId": "2145ed1d-3b62-4bdb-9930-c57cb377459c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/trl/trainer/orpo_trainer.py:247: UserWarning: When using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your TrainingArguments we have set it for you, but you should do it yourself in the future.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Map:   0%|          | 0/100 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "2d3b783e5bbd4f8da1c1de4e1e348e70"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "max_steps is given, it will override any value given in num_train_epochs\n"
          ]
        }
      ],
      "source": [
        "from trl import ORPOConfig, ORPOTrainer\n",
        "\n",
        "orpo_trainer = ORPOTrainer(\n",
        "    model = model,\n",
        "    train_dataset = dataset,\n",
        "    tokenizer = tokenizer,\n",
        "    args = ORPOConfig(\n",
        "        max_length = max_seq_length,\n",
        "        max_prompt_length = max_seq_length//2,\n",
        "        max_completion_length = max_seq_length//2,\n",
        "        per_device_train_batch_size = 2,\n",
        "        gradient_accumulation_steps = 4,\n",
        "        beta = 0.1,\n",
        "        logging_steps = 1,\n",
        "        optim = \"adamw_8bit\",\n",
        "        lr_scheduler_type = \"linear\",\n",
        "        max_steps = turns, # Change to num_train_epochs = 1 for full training runs\n",
        "        fp16 = not torch.cuda.is_bf16_supported(),\n",
        "        bf16 = torch.cuda.is_bf16_supported(),\n",
        "        output_dir = \"outputs\",\n",
        "    ),\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "EWGFqAo5Q2me",
        "outputId": "7b9e5b93-d07f-44b3-be91-3656c83d52ff"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "==((====))==  Unsloth - 2x faster free finetuning | Num GPUs = 1\n",
            "   \\\\   /|    Num examples = 100 | Num Epochs = 9\n",
            "O^O/ \\_/ \\    Batch size per device = 2 | Gradient Accumulation steps = 4\n",
            "\\        /    Total batch size = 8 | Total steps = 100\n",
            " \"-____-\"     Number of trainable parameters = 29,884,416\n",
            "Could not estimate the number of tokens of the input, floating-point operations will not be computed\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='35' max='100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [ 35/100 11:47 < 23:13, 0.05 it/s, Epoch 2.72/9]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Step</th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>rewards / chosen</th>\n",
              "      <th>rewards / rejected</th>\n",
              "      <th>rewards / accuracies</th>\n",
              "      <th>rewards / margins</th>\n",
              "      <th>logps / rejected</th>\n",
              "      <th>logps / chosen</th>\n",
              "      <th>logits / rejected</th>\n",
              "      <th>logits / chosen</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>4.746200</td>\n",
              "      <td>-0.105510</td>\n",
              "      <td>-0.135138</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.029629</td>\n",
              "      <td>-1.351384</td>\n",
              "      <td>-1.055096</td>\n",
              "      <td>-0.756783</td>\n",
              "      <td>-0.966098</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>5.120300</td>\n",
              "      <td>-0.124053</td>\n",
              "      <td>-0.123623</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>-0.000430</td>\n",
              "      <td>-1.236231</td>\n",
              "      <td>-1.240529</td>\n",
              "      <td>-0.793058</td>\n",
              "      <td>-1.048098</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>5.837100</td>\n",
              "      <td>-0.123863</td>\n",
              "      <td>-0.344523</td>\n",
              "      <td>0.750000</td>\n",
              "      <td>0.220659</td>\n",
              "      <td>-3.445227</td>\n",
              "      <td>-1.238633</td>\n",
              "      <td>-2.104153</td>\n",
              "      <td>-1.479414</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>5.279800</td>\n",
              "      <td>-0.146216</td>\n",
              "      <td>-0.159627</td>\n",
              "      <td>0.625000</td>\n",
              "      <td>0.013411</td>\n",
              "      <td>-1.596274</td>\n",
              "      <td>-1.462160</td>\n",
              "      <td>-1.534064</td>\n",
              "      <td>-1.044438</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>5</td>\n",
              "      <td>4.559300</td>\n",
              "      <td>-0.165567</td>\n",
              "      <td>-0.131282</td>\n",
              "      <td>0.750000</td>\n",
              "      <td>-0.034285</td>\n",
              "      <td>-1.312823</td>\n",
              "      <td>-1.655671</td>\n",
              "      <td>-2.347234</td>\n",
              "      <td>-1.601332</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>6</td>\n",
              "      <td>4.092200</td>\n",
              "      <td>-0.081874</td>\n",
              "      <td>-0.113994</td>\n",
              "      <td>0.750000</td>\n",
              "      <td>0.032120</td>\n",
              "      <td>-1.139941</td>\n",
              "      <td>-0.818739</td>\n",
              "      <td>-2.527887</td>\n",
              "      <td>-2.060605</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>7</td>\n",
              "      <td>3.452700</td>\n",
              "      <td>-0.125774</td>\n",
              "      <td>-0.133338</td>\n",
              "      <td>0.625000</td>\n",
              "      <td>0.007564</td>\n",
              "      <td>-1.333382</td>\n",
              "      <td>-1.257744</td>\n",
              "      <td>-2.218135</td>\n",
              "      <td>-1.810420</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>8</td>\n",
              "      <td>2.314100</td>\n",
              "      <td>-0.116242</td>\n",
              "      <td>-0.144191</td>\n",
              "      <td>0.750000</td>\n",
              "      <td>0.027949</td>\n",
              "      <td>-1.441913</td>\n",
              "      <td>-1.162422</td>\n",
              "      <td>-2.987442</td>\n",
              "      <td>-2.334255</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>9</td>\n",
              "      <td>1.940800</td>\n",
              "      <td>-0.098408</td>\n",
              "      <td>-0.129020</td>\n",
              "      <td>0.750000</td>\n",
              "      <td>0.030613</td>\n",
              "      <td>-1.290205</td>\n",
              "      <td>-0.984076</td>\n",
              "      <td>-2.085018</td>\n",
              "      <td>-1.838334</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>10</td>\n",
              "      <td>1.920100</td>\n",
              "      <td>-0.150198</td>\n",
              "      <td>-0.138865</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>-0.011333</td>\n",
              "      <td>-1.388647</td>\n",
              "      <td>-1.501975</td>\n",
              "      <td>-2.300640</td>\n",
              "      <td>-1.942458</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>11</td>\n",
              "      <td>1.441600</td>\n",
              "      <td>-0.142856</td>\n",
              "      <td>-0.125309</td>\n",
              "      <td>0.375000</td>\n",
              "      <td>-0.017546</td>\n",
              "      <td>-1.253094</td>\n",
              "      <td>-1.428555</td>\n",
              "      <td>-1.871226</td>\n",
              "      <td>-2.054025</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>12</td>\n",
              "      <td>1.378700</td>\n",
              "      <td>-0.133618</td>\n",
              "      <td>-0.161189</td>\n",
              "      <td>0.750000</td>\n",
              "      <td>0.027571</td>\n",
              "      <td>-1.611885</td>\n",
              "      <td>-1.336175</td>\n",
              "      <td>-2.412627</td>\n",
              "      <td>-1.882052</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>13</td>\n",
              "      <td>1.189600</td>\n",
              "      <td>-0.131696</td>\n",
              "      <td>-0.156640</td>\n",
              "      <td>0.625000</td>\n",
              "      <td>0.024944</td>\n",
              "      <td>-1.566398</td>\n",
              "      <td>-1.316961</td>\n",
              "      <td>-2.735764</td>\n",
              "      <td>-3.141084</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>14</td>\n",
              "      <td>1.567200</td>\n",
              "      <td>-0.157121</td>\n",
              "      <td>-0.135459</td>\n",
              "      <td>0.375000</td>\n",
              "      <td>-0.021662</td>\n",
              "      <td>-1.354586</td>\n",
              "      <td>-1.571206</td>\n",
              "      <td>-1.927677</td>\n",
              "      <td>-2.384028</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>15</td>\n",
              "      <td>1.205300</td>\n",
              "      <td>-0.161441</td>\n",
              "      <td>-0.124607</td>\n",
              "      <td>0.750000</td>\n",
              "      <td>-0.036834</td>\n",
              "      <td>-1.246072</td>\n",
              "      <td>-1.614408</td>\n",
              "      <td>-2.941623</td>\n",
              "      <td>-2.910793</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>16</td>\n",
              "      <td>1.245700</td>\n",
              "      <td>-0.161125</td>\n",
              "      <td>-0.131071</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>-0.030055</td>\n",
              "      <td>-1.310707</td>\n",
              "      <td>-1.611255</td>\n",
              "      <td>-2.527095</td>\n",
              "      <td>-2.796366</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>17</td>\n",
              "      <td>1.352200</td>\n",
              "      <td>-0.106095</td>\n",
              "      <td>-0.277463</td>\n",
              "      <td>0.875000</td>\n",
              "      <td>0.171368</td>\n",
              "      <td>-2.774630</td>\n",
              "      <td>-1.060945</td>\n",
              "      <td>-2.039414</td>\n",
              "      <td>-1.208907</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>18</td>\n",
              "      <td>1.223200</td>\n",
              "      <td>-0.107352</td>\n",
              "      <td>-0.118422</td>\n",
              "      <td>0.625000</td>\n",
              "      <td>0.011071</td>\n",
              "      <td>-1.184223</td>\n",
              "      <td>-1.073518</td>\n",
              "      <td>-3.417062</td>\n",
              "      <td>-1.675549</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>19</td>\n",
              "      <td>1.275500</td>\n",
              "      <td>-0.087783</td>\n",
              "      <td>-0.113576</td>\n",
              "      <td>0.625000</td>\n",
              "      <td>0.025793</td>\n",
              "      <td>-1.135760</td>\n",
              "      <td>-0.877827</td>\n",
              "      <td>-1.752435</td>\n",
              "      <td>-1.421326</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>20</td>\n",
              "      <td>1.211300</td>\n",
              "      <td>-0.112256</td>\n",
              "      <td>-0.160242</td>\n",
              "      <td>0.625000</td>\n",
              "      <td>0.047986</td>\n",
              "      <td>-1.602419</td>\n",
              "      <td>-1.122562</td>\n",
              "      <td>-2.978828</td>\n",
              "      <td>-2.326180</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>21</td>\n",
              "      <td>1.177500</td>\n",
              "      <td>-0.111365</td>\n",
              "      <td>-0.124940</td>\n",
              "      <td>0.750000</td>\n",
              "      <td>0.013575</td>\n",
              "      <td>-1.249397</td>\n",
              "      <td>-1.113652</td>\n",
              "      <td>-1.949482</td>\n",
              "      <td>-2.166897</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>22</td>\n",
              "      <td>1.253300</td>\n",
              "      <td>-0.100421</td>\n",
              "      <td>-0.133920</td>\n",
              "      <td>0.875000</td>\n",
              "      <td>0.033500</td>\n",
              "      <td>-1.339202</td>\n",
              "      <td>-1.004207</td>\n",
              "      <td>-1.711888</td>\n",
              "      <td>-1.915395</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>23</td>\n",
              "      <td>1.203100</td>\n",
              "      <td>-0.141344</td>\n",
              "      <td>-0.117473</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>-0.023871</td>\n",
              "      <td>-1.174730</td>\n",
              "      <td>-1.413438</td>\n",
              "      <td>-2.180020</td>\n",
              "      <td>-1.887465</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>24</td>\n",
              "      <td>1.257800</td>\n",
              "      <td>-0.114876</td>\n",
              "      <td>-0.144538</td>\n",
              "      <td>0.750000</td>\n",
              "      <td>0.029662</td>\n",
              "      <td>-1.445379</td>\n",
              "      <td>-1.148759</td>\n",
              "      <td>-3.178006</td>\n",
              "      <td>-1.569383</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>25</td>\n",
              "      <td>1.267800</td>\n",
              "      <td>-0.125704</td>\n",
              "      <td>-0.141110</td>\n",
              "      <td>0.875000</td>\n",
              "      <td>0.015406</td>\n",
              "      <td>-1.411097</td>\n",
              "      <td>-1.257035</td>\n",
              "      <td>-3.118785</td>\n",
              "      <td>-1.520048</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>26</td>\n",
              "      <td>1.130400</td>\n",
              "      <td>-0.158023</td>\n",
              "      <td>-0.104329</td>\n",
              "      <td>0.250000</td>\n",
              "      <td>-0.053694</td>\n",
              "      <td>-1.043292</td>\n",
              "      <td>-1.580231</td>\n",
              "      <td>-2.158500</td>\n",
              "      <td>-2.697563</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>27</td>\n",
              "      <td>1.243600</td>\n",
              "      <td>-0.113218</td>\n",
              "      <td>-0.130714</td>\n",
              "      <td>0.625000</td>\n",
              "      <td>0.017496</td>\n",
              "      <td>-1.307144</td>\n",
              "      <td>-1.132185</td>\n",
              "      <td>-2.515060</td>\n",
              "      <td>-1.747510</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>28</td>\n",
              "      <td>1.024300</td>\n",
              "      <td>-0.094745</td>\n",
              "      <td>-0.115618</td>\n",
              "      <td>0.750000</td>\n",
              "      <td>0.020873</td>\n",
              "      <td>-1.156181</td>\n",
              "      <td>-0.947448</td>\n",
              "      <td>-1.269313</td>\n",
              "      <td>-2.547213</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>29</td>\n",
              "      <td>1.310900</td>\n",
              "      <td>-0.103419</td>\n",
              "      <td>-0.118267</td>\n",
              "      <td>0.750000</td>\n",
              "      <td>0.014848</td>\n",
              "      <td>-1.182669</td>\n",
              "      <td>-1.034194</td>\n",
              "      <td>-1.982093</td>\n",
              "      <td>-1.303439</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>30</td>\n",
              "      <td>1.223400</td>\n",
              "      <td>-0.101349</td>\n",
              "      <td>-0.156568</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.055218</td>\n",
              "      <td>-1.565676</td>\n",
              "      <td>-1.013491</td>\n",
              "      <td>-2.890398</td>\n",
              "      <td>-1.294924</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>31</td>\n",
              "      <td>1.149000</td>\n",
              "      <td>-0.090349</td>\n",
              "      <td>-0.109970</td>\n",
              "      <td>0.875000</td>\n",
              "      <td>0.019621</td>\n",
              "      <td>-1.099698</td>\n",
              "      <td>-0.903490</td>\n",
              "      <td>-2.245007</td>\n",
              "      <td>-1.665482</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>32</td>\n",
              "      <td>1.160200</td>\n",
              "      <td>-0.095986</td>\n",
              "      <td>-0.101751</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>0.005765</td>\n",
              "      <td>-1.017509</td>\n",
              "      <td>-0.959855</td>\n",
              "      <td>-1.555219</td>\n",
              "      <td>-1.161400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>33</td>\n",
              "      <td>1.138000</td>\n",
              "      <td>-0.084847</td>\n",
              "      <td>-0.122109</td>\n",
              "      <td>0.750000</td>\n",
              "      <td>0.037261</td>\n",
              "      <td>-1.221086</td>\n",
              "      <td>-0.848475</td>\n",
              "      <td>-1.244303</td>\n",
              "      <td>-1.206149</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "orpo_trainer.train()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FgEvCW76xblp"
      },
      "source": [
        "<a name=\"Inference\"></a>\n",
        "### Inference\n",
        "Let's run the model! You can change the instruction and input - leave the output blank!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0DJPbbtGxcFJ",
        "outputId": "6b25cdda-c86f-4f25-99e7-90c170d97197"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "['<|begin_of_text|>Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\\n\\n### Instruction:\\nContinue the fibonnaci sequence.\\n\\n### Input:\\n1, 1, 2, 3, 5, 8\\n\\n### Response:\\n13<|end_of_text|>']"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# alpaca_prompt = Copied from above\n",
        "FastLanguageModel.for_inference(model) # Enable native 2x faster inference\n",
        "inputs = tokenizer(\n",
        "[\n",
        "    alpaca_prompt.format(\n",
        "        \"Continue the fibonnaci sequence.\", # instruction\n",
        "        \"1, 1, 2, 3, 5, 8\", # input\n",
        "        \"\", # output - leave this blank for generation!\n",
        "    )\n",
        "], return_tensors = \"pt\").to(\"cuda\")\n",
        "\n",
        "outputs = model.generate(**inputs, max_new_tokens = 64, use_cache = True)\n",
        "tokenizer.batch_decode(outputs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "absTV8M1xzwz"
      },
      "source": [
        " You can also use a `TextStreamer` for continuous inference - so you can see the generation token by token, instead of waiting the whole time!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xKzWRN1Px0Bg",
        "outputId": "fcbe0342-a227-4a1d-be7b-039f37193ec3"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<|begin_of_text|>Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n",
            "\n",
            "### Instruction:\n",
            "Continue the fibonnaci sequence.\n",
            "\n",
            "### Input:\n",
            "1, 1, 2, 3, 5, 8\n",
            "\n",
            "### Response:\n",
            "13<|end_of_text|>\n"
          ]
        }
      ],
      "source": [
        "# alpaca_prompt = Copied from above\n",
        "FastLanguageModel.for_inference(model) # Enable native 2x faster inference\n",
        "inputs = tokenizer(\n",
        "[\n",
        "    alpaca_prompt.format(\n",
        "        \"Continue the fibonnaci sequence.\", # instruction\n",
        "        \"1, 1, 2, 3, 5, 8\", # input\n",
        "        \"\", # output - leave this blank for generation!\n",
        "    )\n",
        "], return_tensors = \"pt\").to(\"cuda\")\n",
        "\n",
        "from transformers import TextStreamer\n",
        "text_streamer = TextStreamer(tokenizer)\n",
        "_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y_3rdZXmx3Hh"
      },
      "source": [
        "<a name=\"Save\"></a>\n",
        "### Saving, loading finetuned models\n",
        "To save the final model as LoRA adapters, either use Huggingface's `push_to_hub` for an online save or `save_pretrained` for a local save.\n",
        "\n",
        "**[NOTE]** This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HJbRqLynx3a8"
      },
      "outputs": [],
      "source": [
        "model.save_pretrained(\"lora_model\") # Local saving\n",
        "tokenizer.save_pretrained(\"lora_model\")\n",
        "\n",
        "from google.colab import userdata\n",
        "hf_token = userdata.get('HF_TOKEN')\n",
        "\n",
        "model.push_to_hub(model_repo, token = hf_token) # Online saving\n",
        "tokenizer.push_to_hub(model_repo, token = hf_token) # Online saving"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mwIRb8DByBGg"
      },
      "source": [
        "Now if you want to load the LoRA adapters we just saved for inference, set `False` to `True`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UPHJs9wDyBbN"
      },
      "outputs": [],
      "source": [
        "if False:\n",
        "    from unsloth import FastLanguageModel\n",
        "    model, tokenizer = FastLanguageModel.from_pretrained(\n",
        "        model_name = \"lora_model\", # YOUR MODEL YOU USED FOR TRAINING\n",
        "        max_seq_length = max_seq_length,\n",
        "        dtype = dtype,\n",
        "        load_in_4bit = load_in_4bit,\n",
        "    )\n",
        "    FastLanguageModel.for_inference(model) # Enable native 2x faster inference\n",
        "\n",
        "# alpaca_prompt = You MUST copy from above!\n",
        "\n",
        "inputs = tokenizer(\n",
        "[\n",
        "    alpaca_prompt.format(\n",
        "        \"What is a famous tall tower in Paris?\", # instruction\n",
        "        \"\", # input\n",
        "        \"\", # output - leave this blank for generation!\n",
        "    )\n",
        "], return_tensors = \"pt\").to(\"cuda\")\n",
        "\n",
        "outputs = model.generate(**inputs, max_new_tokens = 64, use_cache = True)\n",
        "tokenizer.batch_decode(outputs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JPSrEDuoyE5_"
      },
      "source": [
        "You can also use Hugging Face's `AutoModelForPeftCausalLM`. Only use this if you do not have `unsloth` installed. It can be hopelessly slow, since `4bit` model downloading is not supported, and Unsloth's **inference is 2x faster**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sTO5VnItyC08"
      },
      "outputs": [],
      "source": [
        "if False:\n",
        "    # I highly do NOT suggest - use Unsloth if possible\n",
        "    from peft import AutoPeftModelForCausalLM\n",
        "    from transformers import AutoTokenizer\n",
        "    model = AutoPeftModelForCausalLM.from_pretrained(\n",
        "        \"lora_model\", # YOUR MODEL YOU USED FOR TRAINING\n",
        "        load_in_4bit = load_in_4bit,\n",
        "    )\n",
        "    tokenizer = AutoTokenizer.from_pretrained(\"lora_model\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l10uNsFYyGav"
      },
      "source": [
        "### Saving to float16 for VLLM\n",
        "\n",
        "We also support saving to `float16` directly. Select `merged_16bit` for float16 or `merged_4bit` for int4. We also allow `lora` adapters as a fallback. Use `push_to_hub_merged` to upload to your Hugging Face account! You can go to https://huggingface.co/settings/tokens for your personal tokens."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fZR6n7DsyHu6"
      },
      "outputs": [],
      "source": [
        "# Merge to 16bit\n",
        "model.save_pretrained_merged(\"model_16\", tokenizer, save_method = \"merged_16bit\",)\n",
        "model.push_to_hub_merged(model_repo+\"_16\", tokenizer, save_method = \"merged_16bit\", token = hf_token)\n",
        "\n",
        "# Merge to 4bit\n",
        "model.save_pretrained_merged(\"model_4\", tokenizer, save_method = \"merged_4bit_forced\",)\n",
        "model.push_to_hub_merged(model_repo+\"_4\", tokenizer, save_method = \"merged_4bit_forced\", token = hf_token)\n",
        "\n",
        "# Just LoRA adapters\n",
        "model.save_pretrained_merged(\"model_lora_m\", tokenizer, save_method = \"lora\",)\n",
        "model.push_to_hub_merged(model_repo+\"_lora\", tokenizer, save_method = \"lora\", token = hf_token)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ssjW3ST2yI1L"
      },
      "source": [
        "### GGUF / llama.cpp Conversion\n",
        "To save to `GGUF` / `llama.cpp`, we support it natively now! We clone `llama.cpp` and we default save it to `q8_0`. We allow all methods like `q4_k_m`. Use `save_pretrained_gguf` for local saving and `push_to_hub_gguf` for uploading to HF.\n",
        "\n",
        "Some supported quant methods (full list on our [Wiki page](https://github.com/unslothai/unsloth/wiki#gguf-quantization-options)):\n",
        "* `q8_0` - Fast conversion. High resource use, but generally acceptable.\n",
        "* `q4_k_m` - Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q4_K.\n",
        "* `q5_k_m` - Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q5_K."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FKVTNAwyyKFR"
      },
      "outputs": [],
      "source": [
        "# Save to 8bit Q8_0\n",
        "if False: model.save_pretrained_gguf(\"model\", tokenizer,)\n",
        "if False: model.push_to_hub_gguf(\"hf/model\", tokenizer, token = \"\")\n",
        "\n",
        "# Save to 16bit GGUF\n",
        "if False: model.save_pretrained_gguf(\"model\", tokenizer, quantization_method = \"f16\")\n",
        "if False: model.push_to_hub_gguf(\"hf/model\", tokenizer, quantization_method = \"f16\", token = \"\")\n",
        "\n",
        "# Save to q4_k_m GGUF\n",
        "if False: model.save_pretrained_gguf(\"model\", tokenizer, quantization_method = \"q4_k_m\")\n",
        "if False: model.push_to_hub_gguf(\"hf/model\", tokenizer, quantization_method = \"q4_k_m\", token = \"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TL-PCYb1yMzx"
      },
      "source": [
        "Now, use the `model-unsloth.gguf` file or `model-unsloth-Q4_K_M.gguf` file in `llama.cpp` or a UI based system like `GPT4All`. You can install GPT4All by going [here](https://gpt4all.io/index.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hp2Lo5-0yNPB"
      },
      "source": [
        "And we're done! If you have any questions on Unsloth, we have a [Discord](https://discord.gg/u54VK8m8tk) channel! If you find any bugs or want to keep updated with the latest LLM stuff, or need help, join projects etc, feel free to join our Discord!\n",
        "\n",
        "Some other links:\n",
        "1. Zephyr DPO 2x faster [free Colab](https://colab.research.google.com/drive/15vttTpzzVXv_tJwEk-hIcQ0S9FcEWvwP?usp=sharing)\n",
        "2. Llama 7b 2x faster [free Colab](https://colab.research.google.com/drive/1lBzz5KeZJKXjvivbYvmGarix9Ao6Wxe5?usp=sharing)\n",
        "3. TinyLlama 4x faster full Alpaca 52K in 1 hour [free Colab](https://colab.research.google.com/drive/1AZghoNBQaMDgWJpi4RbffGM1h6raLUj9?usp=sharing)\n",
        "4. CodeLlama 34b 2x faster [A100 on Colab](https://colab.research.google.com/drive/1y7A0AxE3y8gdj4AVkl2aZX47Xu3P1wJT?usp=sharing)\n",
        "5. Mistral 7b [free Kaggle version](https://www.kaggle.com/code/danielhanchen/kaggle-mistral-7b-unsloth-notebook)\n",
        "6. We also did a [blog](https://huggingface.co/blog/unsloth-trl) with 🤗 HuggingFace, and we're in the TRL [docs](https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth)!\n",
        "7. `ChatML` for ShareGPT datasets, [conversational notebook](https://colab.research.google.com/drive/1Aau3lgPzeZKQ-98h69CCu1UJcvIBLmy2?usp=sharing)\n",
        "8. Text completions like novel writing [notebook](https://colab.research.google.com/drive/1ef-tab5bhkvWmBOObepl1WgJvfvSzn5Q?usp=sharing)\n",
        "\n",
        "<div class=\"align-center\">\n",
        "  <a href=\"https://github.com/unslothai/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\" width=\"115\"></a>\n",
        "  <a href=\"https://discord.gg/u54VK8m8tk\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Discord.png\" width=\"145\"></a>\n",
        "  <a href=\"https://ko-fi.com/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Kofi button.png\" width=\"145\"></a></a> Support our work if you can! Thanks!\n",
        "</div>"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.13"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "2d3b783e5bbd4f8da1c1de4e1e348e70": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3095238a31bb450ab2c73a25adcc5987",
              "IPY_MODEL_a95cc5a7a1834733b6ff6b85b5f0ef7a",
              "IPY_MODEL_f11b16391ce5461c8a0b233477597fe6"
            ],
            "layout": "IPY_MODEL_3d86c8bf6f3949f8baffb0ff7b6727a7"
          }
        },
        "3095238a31bb450ab2c73a25adcc5987": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8fc4d0ad47e349a3a3f5c1c675553b48",
            "placeholder": "​",
            "style": "IPY_MODEL_e9fb2ecf86164771ad02772930286249",
            "value": "Map: 100%"
          }
        },
        "a95cc5a7a1834733b6ff6b85b5f0ef7a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e0ff5af098bc4d82b7107e89a11bdef9",
            "max": 100,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_28e8c875bb4f4314a0685f81e740fe76",
            "value": 100
          }
        },
        "f11b16391ce5461c8a0b233477597fe6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4548a8c803114d2dacd1b8fd5b840403",
            "placeholder": "​",
            "style": "IPY_MODEL_8a67559f27ef46ce8c0d9d88f9d5777a",
            "value": " 100/100 [00:01&lt;00:00, 70.77 examples/s]"
          }
        },
        "3d86c8bf6f3949f8baffb0ff7b6727a7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8fc4d0ad47e349a3a3f5c1c675553b48": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e9fb2ecf86164771ad02772930286249": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e0ff5af098bc4d82b7107e89a11bdef9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "28e8c875bb4f4314a0685f81e740fe76": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "4548a8c803114d2dacd1b8fd5b840403": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8a67559f27ef46ce8c0d9d88f9d5777a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}