DhammaBerg_GPT2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "DhammaBerg.AI",
      "version": "0.3.2",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [
        "RMi46NNXcJkI",
        "PXgEKCOjdWjf",
        "yPfJ5b3CQXqr",
        "OxXYe-ZN9Oj4"
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "2hxLV4YTmcL7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# GPT-2 117M Fine Tuning\n",
        "\n",
        "Adaptation of https://github.com/ak9250/gpt-2-colab\n",
        "\n",
        "Includes code for building a single huge English training text file based on Project Gutenberg (books) or Access to Insight (Dharma texts). ATI seems to overfit quite rapidly since it's only 6 MB, but a 99 MB subset of Gutenberg came out quite convincing after only 4 hours of training. Your mileage may vary."
      ]
    },
    {
      "metadata": {
        "id": "Pzxl1vYX-1kk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Setup\n",
        "\n",
        "1) Make sure GPU is enabled, go to edit->notebook settings->Hardware Accelerator GPU\n",
        "\n",
        "2) make a copy to your google drive, click on copy to drive in panel"
      ]
    },
    {
      "metadata": {
        "id": "iW0abT07ZkhZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Note: colab will reset after 12 hours make sure to save your model checkpoints to google drive around 10-11 hours mark or before, then go to runtime->reset all runtimes. Now copy your train model back into colab and start training again from the previous checkpoint."
      ]
    },
    {
      "metadata": {
        "id": "Z4DHSFr6cRF8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Initialize"
      ]
    },
    {
      "metadata": {
        "id": "iLXW02eIYpcB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "clone and cd into repo"
      ]
    },
    {
      "metadata": {
        "id": "ICYu3w9hIJkC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/nshepperd/gpt-2.git"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Qtn1qZPgZLb0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "install requirements"
      ]
    },
    {
      "metadata": {
        "id": "434oOx0bZH6J",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip3 install --upgrade tensorflow-gpu beautifulsoup4\n",
        "!pip3 install -r requirements.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6eEIs3ApZUVO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "cd gpt-2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "o1hrgeKFYsuE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "download the model"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "A498TySgHYyF",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!python3 download_model.py 117M"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "eyeiSvqmfZNV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "set encoding"
      ]
    },
    {
      "metadata": {
        "id": "7oJPQtdLbbeK",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!export PYTHONIOENCODING=UTF-8"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RMi46NNXcJkI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Mount Google Drive"
      ]
    },
    {
      "metadata": {
        "id": "WvUQhgK3PQ4L",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "mount drive to access google drive for saving and accessing checkpoints later"
      ]
    },
    {
      "metadata": {
        "id": "FNpf6R4ahYSN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0KzSbAvePgsI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "(optional) fetch checkpoints if you have them saved in google drive"
      ]
    },
    {
      "metadata": {
        "id": "cA2Wk7yIPmS6",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!cp -r /content/drive/My\\ Drive/checkpoint/ /content/gpt-2/ "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NSc7_rYkcZbk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Download + Prepare Training Data"
      ]
    },
    {
      "metadata": {
        "id": "8NGvwu_Ucefp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "k-o_fWOVFrEA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "cd /content/gpt-2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "AKAOgbcmczdE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "mkdir data"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Oo0Q5GWLc1Iu",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "cd data"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0p--9zwqQRTc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Project Gutenberg\n",
        "\n",
        "Download through their [Robot Access](http://www.gutenberg.org/wiki/Gutenberg:Information_About_Robot_Access_to_our_Pages)"
      ]
    },
    {
      "metadata": {
        "id": "IpArPqxBPY3y",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "mkdir books"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Oi69zY2Jiu4b",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "download and unzip (this will take a while)"
      ]
    },
    {
      "metadata": {
        "id": "QOCvrs-DHvxa",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!wget -w 0.5 -m -H \"http://www.gutenberg.org/robot/harvest?filetypes[]=txt&langs[]=en\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "K7WfbFtqmJ3n",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!find . -name \"*[!-8].zip\" | while read filename; do unzip -o -d \"`basename -s .zip \"$filename\"`\" \"$filename\"; done;"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "rWTB_loVdJgs",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "collect text files for conversion"
      ]
    },
    {
      "metadata": {
        "id": "ohiK40aamuNJ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!find . -name \"*.txt\" | while read filename; do cp $filename /content/gpt-2/books/; done;"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FW9yT-3OPmNL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "change charset to utf8"
      ]
    },
    {
      "metadata": {
        "id": "ENxoKbCYkW8T",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!find . -name \"*.txt\" | while read filename; do iconv -f ascii -t utf8 $filename > $filename-utf8.txt ; done;"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "59bfzKEJPplb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "combine into single text file"
      ]
    },
    {
      "metadata": {
        "id": "-iAs2XT61sU9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "cat *utf8.txt >> allbooks-utf8.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PXgEKCOjdWjf",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Access to Insight\n",
        "\n",
        "Bulk download from [this page](https://accesstoinsight.org/tech/download/bulk.html)"
      ]
    },
    {
      "metadata": {
        "id": "_kZEr0r2djTr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!wget \"http://accesstoinsight.org/tech/download/ati.zip\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "a5FH7EAEeBVG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "unzip the archive"
      ]
    },
    {
      "metadata": {
        "id": "Az4sTE34eDCO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!unzip ati.zip"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "MiJPoejOeIWG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "cd /content/gpt-2/data/ati"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "69NoNXPB29Id",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Convert HTML to Text"
      ]
    },
    {
      "metadata": {
        "id": "xJcDmfJ2bdmz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Parser for Access to Insight HTML dump based on [this script](https://codereview.stackexchange.com/questions/128515/parsing-locally-stored-html-files) and [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/index.html)"
      ]
    },
    {
      "metadata": {
        "id": "VmZgVcsV15Id",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from bs4 import BeautifulSoup\n",
        "import glob\n",
        "import os\n",
        "import re\n",
        "import contextlib\n",
        "\n",
        "\n",
        "@contextlib.contextmanager\n",
        "def stdout2file(fname):\n",
        "    import sys\n",
        "    f = open(fname, 'w')\n",
        "    sys.stdout = f\n",
        "    yield\n",
        "    sys.stdout = sys.__stdout__\n",
        "    f.close()\n",
        "\n",
        "\n",
        "def parser():\n",
        "    # os.chdir(r\"/contents/gpt-2/data/ati/)\n",
        "    with stdout2file(\"dhamma_cleaned.txt\"):\n",
        "        for file in glob.iglob('tipitaka/**/*.html', recursive=True):\n",
        "            with open(file, encoding=\"utf8\") as f:\n",
        "                contents = f.read()\n",
        "                soup = BeautifulSoup(contents, \"html.parser\")\n",
        "                for item in soup.find_all([\"blockquote\",\"h4\",\"p\"]):\n",
        "                    print(item.get_text())\n",
        "                    print('\\n')\n",
        "                # break\n",
        "parser()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "yPfJ5b3CQXqr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Train the Model"
      ]
    },
    {
      "metadata": {
        "id": "yft7g5w2bOX4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "enter the directory"
      ]
    },
    {
      "metadata": {
        "id": "DCoJhKk-2Bx9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "cd gpt-2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "CBiLbT80bMoM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "initiate training (set to data file created in previous step)"
      ]
    },
    {
      "metadata": {
        "id": "pEn_ihcGI00T",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!PYTHONPATH=src ./train.py --dataset /content/gpt-2/data/dhamma_cleaned.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vS1RJJDFOPnb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "save our checkpoints to start training again later"
      ]
    },
    {
      "metadata": {
        "id": "JretqG1zOXdi",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!cp -r /content/gpt-2/checkpoint/ /content/drive/My\\ Drive/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6D-i7vERWbNS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "copy re-trained (fine-tuned) model into the main directory"
      ]
    },
    {
      "metadata": {
        "id": "VeETvWvrbKga",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!cp -r /content/gpt-2/checkpoint/run1/* /content/gpt-2/models/117M/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "OxXYe-ZN9Oj4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Use the Trained Model\n",
        "\n",
        "There are a few flags available, with a default value:\n",
        "\n",
        "* `seed = None` || a random value is generated unless specified. give a specific integer value if you want to reproduce same results in the future.\n",
        "* `nsamples = 1` || specify the number of samples you want to print\n",
        "* `length = None` || number of tokens (words) to print on each sample.\n",
        "* `batch_size= 1` || how many inputs you want to process simultaneously. doesn't seem to affect the results.\n",
        "* `temperature = 1` || scales logits before sampling prior to softmax.\n",
        "* `top_k = 0` || truncates the set of logits considered to those with the highest values."
      ]
    },
    {
      "metadata": {
        "id": "GmnSrXqtfRbq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Conditional samples"
      ]
    },
    {
      "metadata": {
        "id": "utJj-iY4gHwE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!python3 /content/gpt-2/src/interactive_conditional_samples.py --top_k=40 --nsamples=3 --temperature=0.7 --length=100"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "K8rSqkGxg5OK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Unconditional samples"
      ]
    },
    {
      "metadata": {
        "id": "LaQUEnRxWc3c",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!python3 src/generate_unconditional_samples.py | tee /tmp/samples"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}