GPT2-finetuning2-774M.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "GPT-2 Fine Tuning Notebook",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jkraybill/gpt-2/blob/finetuning/GPT2-finetuning2-774M.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EHiwt3Ww7FF6",
        "colab_type": "text"
      },
      "source": [
        "To try out GPT-2, do this:\n",
        "\n",
        "- go to the \"Runtime\" menu and click \"Change runtime type\" and make sure this is a Python 3 notebook, running with GPU hardware acceleration.\n",
        "- use the \"Files\" section to the left to upload a text file called \"train.txt\" which contains all the text you want to train on, and  a text file called \"val.txt\" which contains all the text you want to validate on.\n",
        "- run the steps below in order."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JE_fFgQ8a_Dd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import json\n",
        "import random\n",
        "import re"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "etM-i8RwbcTH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!git clone https://github.com/jkraybill/gpt-2.git"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LVTyGrhsbdep",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "cd gpt-2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "azu6KCOHbhIy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip3 install -r requirements.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ecmuIWxFgFBz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!sh download_model.sh 774M"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-OhJ6ka_DD94",
        "colab_type": "text"
      },
      "source": [
        "The below step encodes your corpus into \"NPZ\" tokenized format for GPT-2.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZzHsNeMNenxR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!PYTHONPATH=src ./encode.py --in-text ../train.txt --out-npz train.txt.npz --model_name 774M\n",
        "!PYTHONPATH=src ./encode.py --in-text ../val.txt --out-npz val.txt.npz --model_name 774M"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "94oLzOq23474",
        "colab_type": "text"
      },
      "source": [
        "Training is below. It will auto-stop after val loss stops going down for a while, but you may want to interrupt it early.\n",
        "\n",
        "\"sample_every\" controls how often you get sample output from the trained model.\n",
        "\n",
        "\"save_every\" controls how often the model is saved.\n",
        "\n",
        "\"learning_rate\" is the AI learning rate. 0.00005 is the rate I've gotten the best results with, but I think most people are running with significantly higher rates, so you could try adjusting it.\n",
        "\n",
        "\"layers_to_train\" controls the number of layers to fine-tune. Reduce this number if you are getting Out of Memory type errors. I hit them in Colab at values larger than 336 for the 774M model.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QbBJ6JoufBGQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!PYTHONPATH=src ./trainval.py --dataset train.txt.npz --valset val.txt.npz --sample_every=1000 --save_every=25 --learning_rate=0.00005 --stop_after=60000 --model_name=774M --batch_length=512 --layers_to_train=336"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GUa2LujW82cy",
        "colab_type": "text"
      },
      "source": [
        "The step below simply copies your trained model to the model directory, so the output will use your training. If you don't do this, you will be running against the trained GPT-2 model without your finetuning training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mNXhOM22fNHJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cp -r /content/gpt-2/checkpoint/run1/* /content/gpt-2/models/774M/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uvtv6NUj5eXU",
        "colab_type": "text"
      },
      "source": [
        "Run the below step to generate unconditional samples (i.e. \"dream mode\").\n",
        "\n",
        "\"top_k\" controls how many options to consider per word (the larger, the more \"diverse\" the output - anything from 1 to about 50 usually works, I think values around 10 are pretty good).\n",
        "\n",
        "\"temperature\" controls the sampling of the words, from 0 to 1 where 1 is the most \"random\".\n",
        "\n",
        "\"length\" controls the number of words in each sample output.\n",
        "\n",
        "This command will run continuously until you turn it off."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2csc2bZHfrXd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python3 src/generate_unconditional_samples.py --top_k 20 --temperature 0.8 --length=300 --model_name=774M"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IwZl7JAn6d5y",
        "colab_type": "text"
      },
      "source": [
        "Run the command below to run in interactive / \"completion\" mode. You will get a prompt; just type in whatever prompt text you want, and the model will attempt to complete it \"nsamples\" times.\n",
        "\n",
        "\"top_k\", \"length\", and \"temperature\" work as specified above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ugmKkFv__NI_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python3 src/interactive_conditional_samples.py --top_k 1 --length=30 --temperature 0.1 --nsamples 3 --model_name=774M"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}