Copy_of_Train_a_GPT_2_Text_Generating_Model_w_GPU.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of Train a GPT-2 Text-Generating Model w/ GPU",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Deadiy/CBR_Article_Generator/blob/master/Copy_of_Train_a_GPT_2_Text_Generating_Model_w_GPU.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H7LoMj4GA4n_",
        "colab_type": "text"
      },
      "source": [
        "#  Train a GPT-2 Text-Generating Model w/ GPU For Free \n",
        "\n",
        "by [Max Woolf](http://minimaxir.com)\n",
        "\n",
        "*Last updated: November 10th, 2019*\n",
        "\n",
        "Retrain an advanced text generating neural network on any text dataset **for free on a GPU using Collaboratory** using `gpt-2-simple`!\n",
        "\n",
        "For more about `gpt-2-simple`, you can visit [this GitHub repository](https://github.com/minimaxir/gpt-2-simple). You can also read my [blog post](https://minimaxir.com/2019/09/howto-gpt2/) for more information how to use this notebook!\n",
        "\n",
        "\n",
        "To get started:\n",
        "\n",
        "1. Copy this notebook to your Google Drive to keep it and save your changes. (File -> Save a Copy in Drive)\n",
        "2. Make sure you're running the notebook in Google Chrome.\n",
        "3. Run the cells below:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KBkpRgBCBS2_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%tensorflow_version 1.x\n",
        "!pip install -q gpt-2-simple\n",
        "import gpt_2_simple as gpt2\n",
        "from datetime import datetime\n",
        "from google.colab import files"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bj2IJLHP3KwE",
        "colab_type": "text"
      },
      "source": [
        "## GPU\n",
        "\n",
        "Colaboratory uses either a Nvidia T4 GPU or an Nvidia K80 GPU. The T4 is slightly faster than the old K80 for training GPT-2, and has more memory allowing you to train the larger GPT-2 models and generate more text.\n",
        "\n",
        "You can verify which GPU is active by running the cell below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sUmTooTW3osf",
        "colab_type": "code",
        "outputId": "b08024d9-8b4a-47a5-ac3a-3298c8d159b7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 301
        }
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mon May 11 12:28:51 2020       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 440.82       Driver Version: 418.67       CUDA Version: 10.1     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   57C    P0    27W /  70W |   8489MiB / 15079MiB |      0%      Default |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                       GPU Memory |\n",
            "|  GPU       PID   Type   Process name                             Usage      |\n",
            "|=============================================================================|\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0wXB05bPDYxS",
        "colab_type": "text"
      },
      "source": [
        "## Downloading GPT-2\n",
        "\n",
        "If you're retraining a model on new text, you need to download the GPT-2 model first. \n",
        "\n",
        "There are three released sizes of GPT-2:\n",
        "\n",
        "* `124M` (default): the \"small\" model, 500MB on disk.\n",
        "* `355M`: the \"medium\" model, 1.5GB on disk.\n",
        "* `774M`: the \"large\" model, cannot currently be finetuned with Colaboratory but can be used to generate text from the pretrained model (see later in Notebook)\n",
        "* `1558M`: the \"extra large\", true model. Will not work if a K80 GPU is attached to the notebook. (like `774M`, it cannot be finetuned).\n",
        "\n",
        "Larger models have more knowledge, but take longer to finetune and longer to generate text. You can specify which base model to use by changing `model_name` in the cells below.\n",
        "\n",
        "The next cell downloads it from Google Cloud Storage and saves it in the Colaboratory VM at `/models/<model_name>`.\n",
        "\n",
        "This model isn't permanently saved in the Colaboratory VM; you'll have to redownload it if you want to retrain it at a later time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P8wSlgXoDPCR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 141
        },
        "outputId": "a3efe731-8416-473a-838e-e1d747afc01c"
      },
      "source": [
        "gpt2.download_gpt2(model_name=\"124M\")"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fetching checkpoint: 1.05Mit [00:00, 630Mit/s]                                                      \n",
            "Fetching encoder.json: 1.05Mit [00:00, 113Mit/s]                                                    \n",
            "Fetching hparams.json: 1.05Mit [00:00, 281Mit/s]                                                    \n",
            "Fetching model.ckpt.data-00000-of-00001: 498Mit [00:03, 135Mit/s]                                   \n",
            "Fetching model.ckpt.index: 1.05Mit [00:00, 252Mit/s]                                                \n",
            "Fetching model.ckpt.meta: 1.05Mit [00:00, 151Mit/s]                                                 \n",
            "Fetching vocab.bpe: 1.05Mit [00:00, 185Mit/s]                                                       \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N8KXuKWzQSsN",
        "colab_type": "text"
      },
      "source": [
        "## Mounting Google Drive\n",
        "\n",
        "The best way to get input text to-be-trained into the Colaboratory VM, and to get the trained model *out* of Colaboratory, is to route it through Google Drive *first*.\n",
        "\n",
        "Running this cell (which will only work in Colaboratory) will mount your personal Google Drive in the VM, which later cells can use to get data in/out. (it will ask for an auth code; that auth is not saved anywhere)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "puq4iC6vUAHc",
        "colab_type": "code",
        "outputId": "c5013bfd-d8a7-49e2-fad3-495e39a75f7b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "gpt2.mount_gdrive()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BT__brhBCvJu",
        "colab_type": "text"
      },
      "source": [
        "## Uploading a Text File to be Trained to Colaboratory\n",
        "\n",
        "In the Colaboratory Notebook sidebar on the left of the screen, select *Files*. From there you can upload files:\n",
        "\n",
        "![alt text](https://i.imgur.com/TGcZT4h.png)\n",
        "\n",
        "Upload **any smaller text file**  (<10 MB) and update the file name in the cell below, then run the cell."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6OFnPCLADfll",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "file_name = \"cbr_treated.txt\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HeeSKtNWUedE",
        "colab_type": "text"
      },
      "source": [
        "If your text file is larger than 10MB, it is recommended to upload that file to Google Drive first, then copy that file from Google Drive to the Colaboratory VM."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-Z6okFD8VKtS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gpt2.copy_file_from_gdrive(file_name)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LdpZQXknFNY3",
        "colab_type": "text"
      },
      "source": [
        "## Finetune GPT-2\n",
        "\n",
        "The next cell will start the actual finetuning of GPT-2. It creates a persistent TensorFlow session which stores the training config, then runs the training for the specified number of `steps`. (to have the finetuning run indefinitely, set `steps = -1`)\n",
        "\n",
        "The model checkpoints will be saved in `/checkpoint/run1` by default. The checkpoints are saved every 500 steps (can be changed) and when the cell is stopped.\n",
        "\n",
        "The training might time out after 4ish hours; make sure you end training and save the results so you don't lose them!\n",
        "\n",
        "**IMPORTANT NOTE:** If you want to rerun this cell, **restart the VM first** (Runtime -> Restart Runtime). You will need to rerun imports but not recopy files.\n",
        "\n",
        "Other optional-but-helpful parameters for `gpt2.finetune`:\n",
        "\n",
        "\n",
        "*  **`restore_from`**: Set to `fresh` to start training from the base GPT-2, or set to `latest` to restart training from an existing checkpoint.\n",
        "* **`sample_every`**: Number of steps to print example output\n",
        "* **`print_every`**: Number of steps to print training progress.\n",
        "* **`learning_rate`**:  Learning rate for the training. (default `1e-4`, can lower to `1e-5` if you have <1MB input data)\n",
        "*  **`run_name`**: subfolder within `checkpoint` to save the model. This is useful if you want to work with multiple models (will also need to specify  `run_name` when loading the model)\n",
        "* **`overwrite`**: Set to `True` if you want to continue finetuning an existing model (w/ `restore_from='latest'`) without creating duplicate copies. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aeXshJM-Cuaf",
        "colab_type": "code",
        "outputId": "f52314e1-332c-4cc3-f99b-f1989916ffe1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "sess = gpt2.start_tf_sess()\n",
        "\n",
        "gpt2.finetune(sess,\n",
        "              dataset=file_name,\n",
        "              model_name='124M',\n",
        "              steps=1000,\n",
        "              restore_from='latest',\n",
        "              run_name='run2',\n",
        "              print_every=50,\n",
        "              sample_every=100,\n",
        "\t\t\t        overwrite= 'true',\n",
        "              save_every=100\n",
        "              )"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:17: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "Loading checkpoint checkpoint/run2/model-2051\n",
            "INFO:tensorflow:Restoring parameters from checkpoint/run2/model-2051\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\r  0%|          | 0/1 [00:00<?, ?it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Loading dataset...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 1/1 [00:00<00:00,  3.96it/s]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "dataset has 32400 tokens\n",
            "Training...\n",
            "Saving checkpoint/run2/model-2051\n",
            "[2100 | 114.54] loss=0.22 avg=0.22\n",
            "Saving checkpoint/run2/model-2100\n",
            "WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/saver.py:963: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to delete files with this prefix.\n",
            "======== SAMPLE 1 ========\n",
            "05/Tammy-Kills-Birdperson-Cropped.jpg' /></figure><p><span style=\"font-weight: 400;\">Like Saitama, Tammy may have been a while ago, but she still holds a strong place in the series' lore. As such, it's easy for enemy players to overanalyze her, but knowing how she does damage in the hero role as well should. Graves is by no means dumb, but the explanation of optimal damage for an enemy and an enemy role is by no means simple. During the Heroes Association Championships, <a href=\"https://www.cbr.com/tag/ha-hey-res/\"> Heroes Association</a> members took to the airwaves this summer to discuss how popular they were, both because of their hero role and because of their role as an enemy hero. Well done, friends. </span></p><p><span class=\"related-single\">Related: <a href=\"https://www.cbr.com/harry-is-pot-of-greed-most-popular-sees/\">Harry Potter And The Order Of St. Dumas</a></span></p><p>Winter Is Order Of St. Dumas' Snowman</h2><figure><img src='https://static2.cbrimages.com/wordpress/wp-content/uploads/2020/05/lucia-catwoman-costume- Winter-Ista-Mustache-US.jpg' /></figure><p>Damian Wayne is all things Victoria Pavilion and rock star Caroline Dries, who has a lot on her and I've known Donna Troy but not Donna Troy, Joe Ra. In this rendition of the \"Sweet Caroline\" blooper, proverbial wisdom as underpinning every iconic moment in every episode, but the truly gorgeous moments come from producership. The episode has been completed with social distancing in mind, with red carpet-style effects and effects during the episode's production. I encourage you to check it out and let me know what you think. </p>\n",
            "                    <h2 class=\"item-title\"><span class=\"item-num\">10 </span>Parody Of The <a href=\"https://www.cbr.com/tag/batman-meaghan-rath-daughter/\">Heroes</a> Family</a>, particularly Bruce Wayne, where he apparently shares a poignant moment with his half-sister Rachel, which stars Chloe Coleman</span></p><p>Per <a href=\"https://www.cbr.com/tag/batman-daughter-of-darkness/\">Camrus' Catwoman</a>, <em>Lamia</em> may not be her real name, but the mother seems to hold a candle to every line of dialogue in the episode. A hint sent out by <em>Batman: The Animated Series</em> to <a href=\"https://www.cbr.com/batman-and-araheim-rosebudapestries-pot-of-greed-meme-from-boston-statehouse.html\" target=\"_blank\" rel=\"noopener noreferrer\">comic book adaptation</a> of <em>Batman: The Animated Series</em> cartoon characters, <a href=\"https://www.cbr.com/tag/batman-meaghan-rath-bearing-child/\">Batman</a> is to every child the most insecure of all superheroes. As such, he must learn to mistrust and fear the school system as a whole. By being willing to tread into the line of Harry Potter's darkness and come out on the team as something scary, it can save the day for the young heroes on the team.</p><p><span class=\"related-single\">RELATED: <a href=\"https://www.cbr.com/darkest-superheroes-wh-extreme/\">Darkest Heroes</a></span></p><p>That line of thinking isn't always reinforced by fact, as it seems every hero on theteam is protected by a strong inner circle.</p><p>While not a lot is known about their upbringing, <a href=\"https://www.cbr.com/strongest-teen-titans/\">Teen Titans</a> have everything</ Sphere had going for it, the young heroes were always going to have weaknesses and strengths, and while most characters have strengths and vulnerabilities, within the team they're most interested in being part of.</p><h2 class=\"item-title\"><span class=\"item-num\">9 </span>Yang</h2><figure><img src='https://static3.cbrimages.com/wordpress/wp-content\n",
            "\n",
            "[2150 | 244.20] loss=0.12 avg=0.17\n",
            "[2200 | 360.29] loss=0.03 avg=0.12\n",
            "Saving checkpoint/run2/model-2200\n",
            "======== SAMPLE 1 ========\n",
            " E&#039;s decision to work with Lex Luthor.\n",
            "\n",
            "Body: WARNING: This article contains major spoilers for \"The Missing Link,\" the penultimate episode of <em>Supergirl</em> Season 5.</strong></p><p><a href=\"https://www.cbr.com/supergirl-episodes-brainy-dangerous-situation/\"><em>Supergirl</em></a> just bid goodbye to the DEO headquarters.</p><p>In \"The Missing Link,\" the penultimate episode of Season 5, the Super Friends tracked down Rama Khan just as he attempted to activate a fault line in Portland, Oregon. They managed to capture him and bring him back to the DEO -- but, as it turns out, that's exactly what he wanted.</p><p><span class=\"related-single\">RELATED: <a class=\"bc-title-link\" title=\"Supergirl's Next Two Episodes Put [Spoiler] in a Dangerous Situation\" href=\"https://www.cbr.com/supergirl-episodes-brainy-dangerous-situation/\">Supergirl's Next Two Episodes Put [Spoiler] In A Dangerous Situation</a></span></p><p>When Brainy interrogated Rama Khan, he attempted to make a bargain: Rama Khan's freedom for a chance to board Leviathan's ship. However, Rama Khan quickly made it clear he was there only because he wanted to be. In this, boy wonder, all the world gives you a raise. You have been chosen because you have shown that you are worthy of the world's best, and they have all been given a fair amount of heart by the fact that they have all been cast in theinesculptressive gender binary.</p><p>They are not, to be fair, naive. As life goes, the macros on which the characters at large are based are obscenely large, and they are a surefire way to lose your mind. There is little mystery as to why crime rates went down.</p><h2 class=\"item-title\"><span class=\"item-num\">8 </span>What's His Successor Rank?</h2><figure><img src='https://static3.cbrimages.com/wordpress/wp-content/uploads/2020/04/my-hero-academia-one-for-all.jpg' /></figure><p>One For All was born from an unusual situation where All for One gifted his younger brother the ability to stockpile power. For any regular person, this really wouldn't mean anything. However, whether All for One knew it or not, this perfectly complimented a hidden ability within his younger brother to hand down Quirks to other people.</p><p>Thus, One for All is born. Counting the original user, One for All has been held by seven other people before it reached All Might, making him the eighth and greatest holder until now. Midway through their research, Geoffrey came across a gem that, with his knowledge of the gem, made his intention clear. So with a little knowledge and a smile on his face, Geoffrey took to the subreddit to complain about the forum. You can see the lighthearted tone with which he does things, gives away a lot about his character character, and explains what it does. There might be no better use of Geoffrey than this.</p><h2 class=\"item-title\"><span class=\"item-num\">7 </span>What's His Real Name?</h2><figure><img src='https://static0.cbrimages.com/wordpress/wp-content/uploads/2018/12/Webp.net-resizeimage-2.jpg' /></figure><p>The series made it clear that the characters in the anime learn their insecurities in school. Only them.</p><p>Such disparity in learning is part of the mystery as to why so many of the series' heroes end up as the infamous Squelch Man.</p><h2 class=\"item-title\"><span class=\"item-num\">6 </span>What's His Real Name?</h2><figure><img src='https://static3.cbrimages.com/wordpress/wp-content/uploads/2020/04/Webp.net-resizeimage-2.jpg' /></figure><p>The series made it clear that the characters who learn their real name nowadays are either dead or in deep thought,s or just plain not there. The Social Network knew this and have since deleted their social features from the internet, making it hard for those who otherwise would have been friends with anyone who knew.</p><p>The series made it clear that the characters who learn their real names nowadays are either dead or in deep thought,s or just plain not there. The majority of characters nowadays just stay hidden from the world. However,\n",
            "\n",
            "[2250 | 488.83] loss=0.02 avg=0.10\n",
            "[2300 | 604.65] loss=0.02 avg=0.08\n",
            "Saving checkpoint/run2/model-2300\n",
            "======== SAMPLE 1 ========\n",
            "><span class=\"related-single\">RELATED: <a href=\"https://www.cbr.com/dc-comics-batman-robin-jason-todd-damian-wayne-best-worst/\">Robin War: 5 Reasons Why Jason Todd Is The Better Robin (&amp; 5 It's Damian Wayne)</a></span></p></p><p>Raven may have been a superhero longer, but Damian has been fighting against people who were trying to kill him for the majority of his life, giving him an advantage here.</p><h2 class=\"item-title\"><span class=\"item-num\">6 </span>Allies: Raven</h2><figure><img src='https://static3.cbrimages.com/wordpress/wp-content/uploads/2020/05/Teen-Titans-Classic-Lineup.jpg' /></figure><p>Damian's brash attitude means he doesn't make friends easily. His position in the Bat Family might make it seem like he could call on a lot of help, but only Dick and Batman actually like him. And even then, they probably wouldn't get involved. If Superboy wasn't in the 31st century he might help Damian... or just clown on him.</p><p>While both are members of the Teen Titans, people actually like Raven whereas they tolerate Damian. Their mutual teammates would side with her, and that's not even counting her older friends like Donna Troy or Wally West.</p><h2 class=\"item-title\"><span class=\"item-num\">5 </span>Defensive Abilities: Raven</h2><figure><img src='https://static3.cbrimages.com/wordpress/wp-content/uploads/2020/04/Raven.jpg' /></figure><p>As an expert martial artist, Damian knows how to defend himself from a variety of attacks. His reflexes have been honed from years of training.</p><p><p style=\"text-align: left;\"><span class=\"related-single\">RELATED: <a href=\"https://www.cbr.com/jean-grey-vs-raven-most-powerful/\">Jean Grey Vs. Raven: Who Is The Most Powerful? </a></span></p></p><p>Raven's flight and teleportation are much better for avoiding attacks than anything that Damian can do. Plus, because of her mental abilities, Raven can sense when an enemy is going to attack her before they do it. This allows her to take preemptive action, making her defense much better.</p><h2 class=\"item-title\"><span class=\"item-num\">4 </span>Durability: Raven</h2><figure><img src='https://static1.cbrimages.com/wordpress/wp-content/uploads/2020/04/Raven-2.jpg' /></figure><p>Damian has all the durability of a normal human. Being who he is, his costume is definitely in some way reinforced so he can take attacks that get past his guard.</p><p>Raven's durability is also that of a normal human, but her magic and her soulself have allowed her to take more damage than should be possible; plus she's been shown to be able heal herself and others. These abilities give her a bit of an advantage in this department.</p><h2 class=\"item-title\"><span class=\"item-num\">3 </span>Strength: Damian Wayne</h2><figure><img src='https://static2.cbrimages.com/wordpress/wp-content/uploads/2017/09/robin-damian-wayne.jpg' /></figure><p>While he is quite young, Damian is very strong. In his line of work, he'd have to be. He's been training since he was very small and he's been shown beating up adults pretty easily over the years, which is no small feat for someone as young as him.</p><p>Raven isn't much of a physical combatant, and while older than Damian, she's physically weaker than him. That's not to say she's weak - she's still a superhero and has some physical training, just not as much Damian.</p><h2 class=\"item-title\"><span class=\"item-num\">2 </span>Attitude: Raven</h2><figure><img src='https://static3.cbrimages.com/wordpress/wp-content/uploads/2018/11/Raven-1.jpg' /></figure><p>Damian is brash, arrogant, and headstrong. He has an inflated opinion of himself and this can definitely be taken advantage of. Damian is much too emotional for his own good and while he's very good at what he does, he's not as good as he thinks he is. That's a liability.</p><p\n",
            "\n",
            "[2350 | 733.10] loss=0.02 avg=0.07\n",
            "[2400 | 848.78] loss=0.01 avg=0.06\n",
            "Saving checkpoint/run2/model-2400\n",
            "======== SAMPLE 1 ========\n",
            "p><img src=\"https://static3.cbrimages.com/wordpress/wp-content/uploads/2020/05/Tammy-Kills-Birdperson-Cropped.jpg\"/></figure><p>A huge part of Birdperson's gimmick is that he has a very <a href=\"https://www.cbr.com/rick-morty-alien-races-ranked/\">alien</a> yet stable demeanor as an emotionless, logical warrior. Much like his <em>Buck Rogers</em> inspiration or <em>Star Trek</em>'s Spock, Birdperson is reserved and monotoned person who shows very little facial affectation.</p><p>This changes in the Season 2 finale, where Tammy's betrayal and attack cause him to look worried and confused during his final moments. While the relationship wasn't prominent in the story, this scene spoke levels to how much it meant to Birdperson himself.</p><h2 class=\"item-title\"><span class=\"item-num\">6 </span>Voiced By Dan Harmon</h2><figure><img src='https://static0.cbrimages.com/wordpress/wp-content/uploads/2018/09/Dan-Harmon-1.jpg' /></figure><p>Perhaps finding kinship with the emotionless and overly-logical sidekick, Dan Harmon took it upon himself to voice Birdperson. In contrast to his silly-voiced partner, Justin Roiland, Dan Harmon has rarely voiced anyone in the series, taking only to one-off characters such as Anatomy Park's Alejandro or Ice-T.</p><p><span class=\"related-single\">RELATED: <a href=\"https://www.cbr.com/rick-morty-jokes-funniest-extreme/\">Rick And Morty: 5 Of The Funniest Jokes (&amp; 5 That Took It Too Far)</a></span></p><p>Birdperson is perhaps his most active role behind the mic, and fans will hopefully get to hear his sultry tones as the character in the future.</p><h2 class=\"item-title\"><span class=\"item-num\">5 </span>Was In A Band With Rick</h2><figure><img src='https://static2.cbrimages.com/wordpress/wp-content/uploads/2020/05/Rick-and-Morty-Band-Photo-Cropped.jpg' /></figure><p>Birdperson was once Rick's partner in crime and a leading revolutionary in the greater battle against the Galactic Federation. However, that deep and tumultuous history doesn't hold a candle to the fact that he was once part of an epic band alongside Rick and Squanchy. During the trio's early days, they were apparently part of a rock band named \"The Flesh Curtains,\" in which, Birdperson was the lead vocals.</p><p>If the series is going to delve in anything in Rick's past, it has to be his old band. Birdperson singing has to become a thing outside of the musical gag in \"Never Ricking Morty.\"</p><h2 class=\"item-title\"><span class=\"item-num\">4 </span>Was Saved By Rick Long Ago</h2><figure><img src='https://static1.cbrimages.com/wordpress/wp-content/uploads/2018/07/rick-morty-soundtrack-cropped.jpg' /></figure><p>Being revolutionaries in war against a galaxy wide government is going to have some casualties. Birdperson has been in war and survived on his back legs. For those keeping up (being revolutionaries at heart isn't always about who he works with)</p><p>Rick has had his fair share of alien blood on his hands and has turned a blind-eye to the deaths of thousands of other characters. The fact that he was willing to save anyone at all, outside of the Smith family, is kind of a big deal for his history.</p><h2 class=\"item-title\"><span class=\"item-num\">3 </span>May Be A Bad Birdperson</h2><figure><img src='https://static3.cbrimages.com/wordpress/wp-content/uploads/2020/05/Rick-and-Morty-Bar-Photo-Cropped.jpg' /></figure><p>While Birdperson's exploits against the Galactic Federation may have been for freedom, that doesn't inherently mean that he was always the noble hero.</p><p>Evident in his drunken talk during \"The Wedding Squanchers,\" Rick, Squanchy, and, surprisingly, Birdperson have all committed a vast array of \"atrocities\" during their battles. Exactly what this means is left in mystery; but knowing what Rick does\n",
            "\n",
            "[2450 | 977.24] loss=0.02 avg=0.06\n",
            "[2500 | 1093.08] loss=0.01 avg=0.05\n",
            "Saving checkpoint/run2/model-2500\n",
            "======== SAMPLE 1 ========\n",
            "=\"https://static1.cbrimages.com/wordpress/wp-content/uploads/2020/04/All-Might-Hero-Form.jpg' /></figure><p>One of the first things that an anime fan may notice and even be driven mad by is the sound of a <a href=\"https://www.cbr.com/anime-characters-with-same-voice-actor/\">familiar voice in another anime</a>. When <em>My Hero Academia</em> first premiered with a cast of voice acting veterans, tons of people played the \"Who's Who?\" game. This is no different for All Might, whose strong voice and writing demanded some attention.</p><p>His Japanese voice actor is Kenta Miyake, who has a lot of supporting roles as a voice actor but really got to break out as All Might. The same can't be said for All Might's English voice actor, Christopher Sabat, who's voiced iconic characters such as Major Armstrong, Vegeta, and Roronoa Zoro.\n",
            "\n",
            "Title: Teen Titans: Damian Wayne Vs. Raven: Who Would Win? | CBR\n",
            "\n",
            "SubTitle: It&#039;s time for a Teen Titan rumble. If Damien Wayne took on Raven, which one of them would end up winning?\n",
            "\n",
            "Body: The <a href=\"https://screenrant.com/teen-titans-plotlines-see-titans/\">Teen Titans</a> are a storied franchise in <a href=\"https://www.cbr.com/tag/dc/\">DC</a> history. The team debuted in the '60s, teaming the sidekicks of DC's heroes together. The concept would be resurrected in the '80s, introducing new characters to the team and becoming one of the top-selling titles in the industry. One of those new characters was Raven, a powerful magic user with a dark past.</p><p style=\"text-align: left;\"><span class=\"related-single\">RELATED: <a href=\"https://www.cbr.com/batman-things-didnt-know-about-damian-wayne-robin-secrets-facts/\">Batman: 10 Things You Didn't Know About Damian Wayne </a></span></p><p>Robins have always been a part of the <a href=\"https://www.cbr.com/tag/teen-titans/\">Teen Titans</a> and nowadays, that means Damian Wayne. The son of Batman and Talia al Ghul, this arrogant young man is one of the most skilled Robins ever. If he and Raven were to come to blows, who would win?</p>\n",
            "                           <h2 class=\"item-title\"><span class=\"item-num\">11 </span>Fighting Skill: Damian Wayne</h2><figure><img src='https://static1.cbrimages.com/wordpress/wp-content/uploads/2020/04/Damian-Wayne-Cropped.jpg' /></figure><p>This one is a no brainer- Damian is a much better fighter than Raven. Trained by the League of Assassins from a young age, he also learned from two of the best crimefighters out there, Dick Grayson and Batman.</p><p>Raven's powers means she's never really had to worry very much about hand to hand combat. She's probably picked up a thing or two over the years, but not nearly enough against someone of Damian's caliber.</p><h2 class=\"item-title\"><span class=\"item-num\">10 </span>Long Range Attacks: Raven</h2><figure><img src='https://static2.cbrimages.com/wordpress/wp-content/uploads/2020/05/Raven-showing-off-her-powers.jpg' /></figure><p>Damian, like most members of the Bat Family, has access to a wide variety of weapons from his utility belt - batarangs, explosives, grappling guns, and others. He's an expert in using these weapons in combat.</p><p><p style=\"text-align: left;\"><span class=\"related-single\">RELATED: <a href=\"https://www.cbr.com/dc-comics-characters-respect-raven-dont/\">5 DC Characters Who Respect Raven (&amp; 5 Who Don't)</a></span></p></p><p>Raven, though, has her magical abilities and magic, trumping just about any weapon that Damian can bring to bear. She can also use her mental powers offensively, allowing her to knock out her opponents with just a thought.</p><h2 class=\"item\n",
            "\n",
            "[2550 | 1221.48] loss=0.01 avg=0.05\n",
            "[2600 | 1337.21] loss=0.02 avg=0.04\n",
            "Saving checkpoint/run2/model-2600\n",
            "======== SAMPLE 1 ========\n",
            " his son to take care of their father's pedestal and keep it clean since the person who protects them sits there. </span></p><p><span style=\"font-weight: 400;\">The location of the pedestal is similar to that of Hachiko, Japan’s most famous dog, who was known for the way he loyally waited for his owner outside of the train station, even after the owner died. </span></p><h2 class=\"item-title\"><span class=\"item-num\">1 </span>Prone To Boredom</h2><figure><img src='https://static3.cbrimages.com/wordpress/wp-content/uploads/2020/04/Watchdog-Man-Sits-From-One-Punch-Man-Cropped.jpg' /></figure><p><span style=\"font-weight: 400;\">Like Saitama, Watchdog Man is also very prone to getting bored, especially when it comes to combat. He doesn’t find much of anything very impressive, and he doesn’t care much for the fighting styles of others. </span></p><p><span style=\"font-weight: 400;\">When he leaves Q-City, he often finds reasons to go back, especially if he’s mostly been seeing the combat techniques of others.\n",
            "\n",
            "Title: My Hero Academia: How Strong Is All Might? (&amp; 9 Other Questions About The Character, Answered)\n",
            "\n",
            "SubTitle: All Might is one of the most iconic heroes from My Hero Academia, and here are 10 questions that even his fans have about him, answered.\n",
            "\n",
            "Body: My Hero Academia</a></em> may be one of the biggest Shonen series out nowadays, but that doesn't mean that even some of the most hardened fans still don't have questions about it. The series has been going on since 2014 and has introduced a myriad of different characters, each with a deep history, if not just plenty of intrigue with their character design and <a href=\"https://screenrant.com/my-hero-academia-worst-quirks-ranked-tape-invisibility/\" target=\"_blank\" rel=\"noopener noreferrer\">Quirks alone</a>.</p><p><span class=\"related-single\">RELATED: <a href=\"https://www.cbr.com/my-hero-academia-mha-characters-wh-surpass-greater-weaker-all-might/\">My Hero Academia: 5 Characters Who Can Surpass All Might (&amp; 5 Who Already Failed)</a></span></p><p>As such, it's easy for character details to be lost in the shuffle, as the series is constantly moving and shifting between different focuses. One such character that fans are still trying to get to know is <a href=\"https://www.cbr.com/my-hero-all-might-memes/\">All Might</a>. He's Izuku's main mentor whose past and even current agenda are shrouded in some mystery. For new fans trying to get into the series or old ones who may have missed something, here are 10 questions answered about All Might.</p>\n",
            "                         <h2 class=\"item-title\"><span class=\"item-num\">10 </span>How Strong Is He?</h2><figure><img src='https://static1.cbrimages.com/wordpress/wp-content/uploads/2020/04/my-hero-academia-all-might-1.jpg' /></figure><p>All Might immediately premiered in the Shonen world as one of its strongest figures. His initial appearance literally had him punch so hard that he was able to summon a storm. This begged the question of how strong he is exactly. Creator Kohei Horikoshi hasn't provided any solid stats as of this writing, but that hasn't stopped wide speculation, especially by the analytical nerds over at Death Battle.</p><p><span class=\"related-single\">RELATED: <a href=\"https://www.cbr.com/anime-heroes-powerful-decade-ranked/\">The 10 Most Powerful Anime Heroes Of The Decade, Ranked</a></span></p><p>They posited that the power of the storm creating punch was equivalent to a little more than 24 gigatons of TNT. Going by All Might's estimate, that he could've defeated the USJ Nomu with five punches instead of 300, and they also predicted that his pre-injury self was about 60x stronger. Doing the math, there's a whole lot of American behind All Might's fists.</p><h\n",
            "\n",
            "[2650 | 1465.56] loss=0.01 avg=0.04\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AD-D9WIvNne_",
        "colab_type": "code",
        "outputId": "bbcc77b1-894b-49b1-d429-9a2cd5fa6d38",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IXSuTNERaw6K",
        "colab_type": "text"
      },
      "source": [
        "After the model is trained, you can copy the checkpoint folder to your own Google Drive.\n",
        "\n",
        "If you want to download it to your personal computer, it's strongly recommended you copy it there first, then download from Google Drive. The checkpoint folder is copied as a `.rar` compressed file; you can download it and uncompress it locally."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VHdTL8NDbAh3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gpt2.copy_checkpoint_to_gdrive(run_name='run2')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qQJgV_b4bmzd",
        "colab_type": "text"
      },
      "source": [
        "You're done! Feel free to go to the **Generate Text From The Trained Model** section to generate text based on your retrained model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pel-uBULXO2L",
        "colab_type": "text"
      },
      "source": [
        "## Load a Trained Model Checkpoint\n",
        "\n",
        "Running the next cell will copy the `.rar` checkpoint file from your Google Drive into the Colaboratory VM."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DCcx5u7sbPTD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gpt2.copy_checkpoint_from_gdrive(run_name='run2')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RTa6zf3e_9gV",
        "colab_type": "text"
      },
      "source": [
        "The next cell will allow you to load the retrained model checkpoint + metadata necessary to generate text.\n",
        "\n",
        "**IMPORTANT NOTE:** If you want to rerun this cell, **restart the VM first** (Runtime -> Restart Runtime). You will need to rerun imports but not recopy files."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-fxL77nvAMAX",
        "colab_type": "code",
        "outputId": "0cdf5db7-f8b8-449e-84ba-2b719e2b4264",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "sess = gpt2.start_tf_sess()\n",
        "gpt2.load_gpt2(sess, run_name='run2')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loading checkpoint checkpoint/run2/model-1051\n",
            "INFO:tensorflow:Restoring parameters from checkpoint/run2/model-1051\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ClJwpF_ACONp",
        "colab_type": "text"
      },
      "source": [
        "## Generate Text From The Trained Model\n",
        "\n",
        "After you've trained the model or loaded a retrained model from checkpoint, you can now generate text. `generate` generates a single text from the loaded model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4RNY6RBI9LmL",
        "colab_type": "code",
        "outputId": "d29faa92-cec4-4863-fae5-4af14221c554",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "gpt2.generate(sess, run_name='run2')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "One of the strangest things about Japan's #1 Hero is the fact that he just doesn't look Japanese. All Might is heavily modeled off of the American sensibility of superheroes, as well as Japan's perception of the ideal American as a whole, i.e. large, buff, blonde, and blue-eyed. A lot of this can be chocked up to Kohei Horikoshi's Marvel influences and the current boom of the superhero genre.</p><p>In story, however, All Might did work and train in Los Angeles, California before he took his mantle in Japan. The American influences are apparent in his costume and move set, with attacks like Texas and Oklahoma Smash, attacks that generate tornadoes, actually correlating with the U.S.'s tornado heavy territories.</p><h2 class=\"item-title\"><span class=\"item-num\">4 </span>How Old Is He?</h2><figure><img src='https://static1.cbrimages.com/wordpress/wp-content/uploads/2019/06/Young-All-Might-My-Hero-Academia.jpg' /></figure><p>Given his abilities and unique condition, All Might's appearance varies wildly across the series. In his hero form, he's large and healthy, practically brimming with life. However, in his regular form, he looks like he really should be in a hospital. He's incredibly skinny, constantly coughing up blood, and has abyss like eyes that should haunt children.</p><p>Such disparity in his appearance makes it hard for fans to gauge how old he really is. However, working with the story's timeline and some analysis by <a href=\"https://www.youtube.com/channel/UCB9_VH_CNbbH4GfKu8qh63w\" target=\"_blank\" rel=\"noopener noreferrer\">Death Battle</a> and a really detailed <a href=\"https://www.quora.com/How-old-is-All-Might-from-My-Hero-Academia\">Quora post</a>, All Might is most likely 48 at the beginning of the series and 49 in the current story.</p><h2 class=\"item-title\"><span class=\"item-num\">3 </span>Is He Going To Die?</h2><figure><img src='https://static0.cbrimages.com/wordpress/wp-content/uploads/2020/04/All-Might-Nighteye-Flashbackkk.jpg' /></figure><p>All Might is one of those characters that just waves a death flag as soon as he premieres. Out of a lot of anime characters, he probably has the least ambiguity given his regular form's demeanor and the comedic tendency to constantly be coughing up blood. Many thought that his recent encounter with All for One would be his last, but he was fortunately able to pull through. However, the appearance of his old sidekick, Sir Nighteye, would soon bring fans back to earth.</p><p><span class=\"related-single\">RELATED: <a href=\"https://www.cbr.com/death-flags-anime-character-die/\">10 Death Flags That Mean An Anime Character is Probably Going To Die</a></span></p><p>In a flashback to the events after All Might beat All for One for the first time, Nighteye predicted that he would die in six to seven years to a powerful villain. About five years had passed between then and the moment that he met Midoriya Izuku. Despite his major victory and his propensity to remain in the sidelines, All Might's reckoning is coming very soon.</p><h2 class=\"item-title\"><span class=\"item-num\">2 </span>What's His Relationship With Detective Naomasa?</h2><figure><img src='https://static0.cbrimages.com/wordpress/wp-content/uploads/2020/04/Naomasa-with-backup.jpg' /></figure><p>One of All Might's most understated and mysterious relationships is with one Detective Naomasa. After the events of the Stain Arc, Midoriya was surprised to find that a regular detective not only knew All Might's secrets, but was one of his closest friends. Since then, Naomasa hasn't really been brought up much, leaving fans to speculate how close he and the #1 really were and if he would turn up as the infamous traitor.</p><p>However, <em>My Hero Academia: Vigilantes</em> would stifle some of those worries, as it would actually go into detail on their history. Naomasa is actually a detective who happened to discover All Might's secret identity after a comedic turn of events. However, he not only promised to keep All Might's secret\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oF4-PqF0Fl7R",
        "colab_type": "text"
      },
      "source": [
        "If you're creating an API based on your model and need to pass the generated text elsewhere, you can do `text = gpt2.generate(sess, return_as_list=True)[0]`\n",
        "\n",
        "You can also pass in a `prefix` to the generate function to force the text to start with a given character sequence and generate text from there (good if you add an indicator when the text starts).\n",
        "\n",
        "You can also generate multiple texts at a time by specifing `nsamples`. Unique to GPT-2, you can pass a `batch_size` to generate multiple samples in parallel, giving a massive speedup (in Colaboratory, set a maximum of 20 for `batch_size`).\n",
        "\n",
        "Other optional-but-helpful parameters for `gpt2.generate` and friends:\n",
        "\n",
        "*  **`length`**: Number of tokens to generate (default 1023, the maximum)\n",
        "* **`temperature`**: The higher the temperature, the crazier the text (default 0.7, recommended to keep between 0.7 and 1.0)\n",
        "* **`top_k`**: Limits the generated guesses to the top *k* guesses (default 0 which disables the behavior; if the generated output is super crazy, you may want to set `top_k=40`)\n",
        "* **`top_p`**: Nucleus sampling: limits the generated guesses to a cumulative probability. (gets good results on a dataset with `top_p=0.9`)\n",
        "* **`truncate`**: Truncates the input text until a given sequence, excluding that sequence (e.g. if `truncate='<|endoftext|>'`, the returned text will include everything before the first `<|endoftext|>`). It may be useful to combine this with a smaller `length` if the input texts are short.\n",
        "*  **`include_prefix`**: If using `truncate` and `include_prefix=False`, the specified `prefix` will not be included in the returned text."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DKMc0fiej4N",
        "colab_type": "code",
        "outputId": "942a86dc-1fe0-43f0-a467-473fac678022",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 370
        }
      },
      "source": [
        "gpt2.generate(sess,\n",
        "              length=250,\n",
        "              temperature=0.7,\n",
        "              prefix=\"Title:\",\n",
        "              nsamples=5,\n",
        "              batch_size=5\n",
        "              )"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "FileNotFoundError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-23-2870898db7b5>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      4\u001b[0m               \u001b[0mprefix\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"Title:\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m               \u001b[0mnsamples\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m               \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m               )\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/gpt_2_simple/gpt_2.py\u001b[0m in \u001b[0;36mgenerate\u001b[0;34m(sess, run_name, checkpoint_dir, model_name, model_dir, sample_dir, return_as_list, truncate, destination_path, sample_delim, prefix, seed, nsamples, batch_size, length, temperature, top_k, top_p, include_prefix)\u001b[0m\n\u001b[1;32m    439\u001b[0m         \u001b[0mcheckpoint_path\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcheckpoint_dir\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrun_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    440\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 441\u001b[0;31m     \u001b[0menc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mencoder\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_encoder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcheckpoint_path\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    442\u001b[0m     \u001b[0mhparams\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdefault_hparams\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    443\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcheckpoint_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'hparams.json'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/encoder.py\u001b[0m in \u001b[0;36mget_encoder\u001b[0;34m(checkpoint_path)\u001b[0m\n\u001b[1;32m    107\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    108\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mget_encoder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcheckpoint_path\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 109\u001b[0;31m     \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcheckpoint_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'encoder.json'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'r'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    110\u001b[0m         \u001b[0mencoder\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mjson\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    111\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcheckpoint_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'vocab.bpe'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'r'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"utf-8\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'checkpoint/run1/encoder.json'"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zjjEN2Tafhl2",
        "colab_type": "text"
      },
      "source": [
        "For bulk generation, you can generate a large amount of text to a file and sort out the samples locally on your computer. The next cell will generate a generated text file with a unique timestamp.\n",
        "\n",
        "You can rerun the cells as many times as you want for even more generated texts!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fa6p6arifSL0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gen_file = 'gpt2_CBR_{:%Y%m%d_%H%M%S}.txt'.format(datetime.utcnow())\n",
        "\n",
        "gpt2.generate_to_file(sess,\n",
        "                      destination_path=gen_file,\n",
        "                      length=500,\n",
        "                      run_name='run2',\n",
        "                      temperature=0.7,\n",
        "                      nsamples=20,\n",
        "\t\t                  prefix=\"Title:\",\n",
        "                      batch_size=1\n",
        "                      )"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0-LRex8lfv1g",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# may have to run twice to get file to download\n",
        "files.download(gen_file)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QQAN3M6RT7Kj",
        "colab_type": "text"
      },
      "source": [
        "## Generate Text From The Pretrained Model\n",
        "\n",
        "If you want to generate text from the pretrained model, not a finetuned model, pass `model_name` to `gpt2.load_gpt2()` and `gpt2.generate()`.\n",
        "\n",
        "This is currently the only way to generate text from the 774M or 1558M models with this notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hsUd_jHgUZnD",
        "colab_type": "code",
        "outputId": "4e0c8a3f-3527-41c4-e3fe-3357f3f8f6c2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 158
        }
      },
      "source": [
        "model_name = \"774M\"\n",
        "\n",
        "gpt2.download_gpt2(model_name=model_name)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fetching checkpoint: 1.05Mit [00:00, 354Mit/s]                                                      \n",
            "Fetching encoder.json: 1.05Mit [00:00, 131Mit/s]                                                    \n",
            "Fetching hparams.json: 1.05Mit [00:00, 279Mit/s]                                                    \n",
            "Fetching model.ckpt.data-00000-of-00001: 3.10Git [00:23, 131Mit/s]                                  \n",
            "Fetching model.ckpt.index: 1.05Mit [00:00, 380Mit/s]                                                \n",
            "Fetching model.ckpt.meta: 2.10Mit [00:00, 226Mit/s]                                                 \n",
            "Fetching vocab.bpe: 1.05Mit [00:00, 199Mit/s]                                                       \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BAe4NpKNUj2C",
        "colab_type": "code",
        "outputId": "b09bfe1d-2ff8-4b8a-fffb-273d28d5d4ae",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        }
      },
      "source": [
        "sess = gpt2.start_tf_sess()\n",
        "\n",
        "gpt2.load_gpt2(sess, model_name=model_name)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0828 18:37:58.571830 139905369159552 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to check for files with this prefix.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Loading pretrained model models/774M/model.ckpt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-xInIZKaU104",
        "colab_type": "code",
        "outputId": "56348e28-7d08-45e3-c859-f26c0efd066d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 797
        }
      },
      "source": [
        "gpt2.generate(sess,\n",
        "              model_name=model_name,\n",
        "              prefix=\"The secret of life is\",\n",
        "              length=100,\n",
        "              temperature=0.7,\n",
        "              top_p=0.9,\n",
        "              nsamples=5,\n",
        "              batch_size=5\n",
        "              )"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The secret of life is that it's really easy to make it complicated,\" said Bill Nye, the host of the popular science show \"Bill Nye the Science Guy.\" \"And this is one of the reasons why we all need to be smarter about science, because we can't keep up with the amazing things that are going on all the time.\"\n",
            "\n",
            "While Nye is correct that \"everything that's going on all the time\" is making the world a better place, he misses the point. This is not\n",
            "====================\n",
            "The secret of life is in the rhythm of the universe. It's not a mystery. It's not a mystery to me. It's the nature of the universe. It's the beauty of the universe. It's the way the universe works. It's the way the universe is. It's the way the universe is going to work. It's the way the universe is. It's the way the universe is. It's the way the universe is. It's the way the universe is. It's the way\n",
            "====================\n",
            "The secret of life is in the universe.\n",
            "\n",
            "\n",
            "-\n",
            "\n",
            "The Red Devil\n",
            "\n",
            "It's the end of the world as we know it, and the only thing that can save us is a band of super-powered individuals known as the Red Devil.\n",
            "\n",
            "\n",
            "The Red Devil is a group of super-powered individuals who are seeking the secret of life and the only way they know how to do it is by taking on the roles of a variety of different super-powered individuals, each of which has their own\n",
            "====================\n",
            "The secret of life is in the mixing of the elements, and it is the mixing of the elements that makes life possible.\"\n",
            "\n",
            "But in the world of food science, the idea of a \"complex\" or \"complexity\" is almost entirely imaginary.\n",
            "\n",
            "As a scientist, I'm fascinated by the question of how life first began.\n",
            "\n",
            "It's the question that drives my work and the work of the scientists who work on it.\n",
            "\n",
            "My current research is exploring how microbes work in the first moments\n",
            "====================\n",
            "The secret of life is the journey of life, the search for the truth.\n",
            "\n",
            "4.4.2. The last thing you know\n",
            "\n",
            "There is nothing more important than the last thing you know.\n",
            "\n",
            "4.4.3. The little things that make all the difference\n",
            "\n",
            "The little things that make all the difference.\n",
            "\n",
            "4.4.4. The truth is the best teacher\n",
            "\n",
            "The truth is the best teacher.\n",
            "\n",
            "4.4.5. The truth is what\n",
            "====================\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ig-KVgkCDCKD",
        "colab_type": "text"
      },
      "source": [
        "# Etcetera\n",
        "\n",
        "If the notebook has errors (e.g. GPU Sync Fail), force-kill the Colaboratory virtual machine and restart it with the command below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rIHiVP53FnsX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!kill -9 -1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wmTXWNUygS5E",
        "colab_type": "text"
      },
      "source": [
        "# LICENSE\n",
        "\n",
        "MIT License\n",
        "\n",
        "Copyright (c) 2019 Max Woolf\n",
        "\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE."
      ]
    }
  ]
}