Train_a_GPT_2_Text_Generating_Model_w_GPU.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H7LoMj4GA4n_"
      },
      "source": [
        "#  Train a GPT-2 Text-Generating Model w/ GPU For Free \n",
        "\n",
        "by [Max Woolf](http://minimaxir.com)\n",
        "\n",
        "*Last updated: October 17th, 2021*\n",
        "\n",
        "Retrain an advanced text generating neural network on any text dataset **for free on a GPU using Collaboratory** using `gpt-2-simple`!\n",
        "\n",
        "For more about `gpt-2-simple`, you can visit [this GitHub repository](https://github.com/minimaxir/gpt-2-simple). You can also read my [blog post](https://minimaxir.com/2019/09/howto-gpt2/) for more information how to use this notebook!\n",
        "\n",
        "\n",
        "To get started:\n",
        "\n",
        "1. Copy this notebook to your Google Drive to keep it and save your changes. (File -> Save a Copy in Drive)\n",
        "2. Make sure you're running the notebook in Google Chrome.\n",
        "3. Run the cells below:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KBkpRgBCBS2_",
        "outputId": "9836f297-331c-4761-f14d-7aadd3c2cb47",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "!pip install -q gpt-2-simple\n",
        "import gpt_2_simple as gpt2\n",
        "from datetime import datetime\n",
        "from google.colab import files"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  Building wheel for gpt-2-simple (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bj2IJLHP3KwE"
      },
      "source": [
        "## GPU\n",
        "\n",
        "Colaboratory uses either a Nvidia T4 GPU or an Nvidia K80 GPU. The T4 is slightly faster than the old K80 for training GPT-2, and has more memory allowing you to train the larger GPT-2 models and generate more text.\n",
        "\n",
        "You can verify which GPU is active by running the cell below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sUmTooTW3osf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0ab66156-cc87-4408-9833-f0a9d35bf734"
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fri Nov 18 16:34:20 2022       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   67C    P8    11W /  70W |      0MiB / 15109MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0wXB05bPDYxS"
      },
      "source": [
        "## Downloading GPT-2\n",
        "\n",
        "If you're retraining a model on new text, you need to download the GPT-2 model first. \n",
        "\n",
        "There are three released sizes of GPT-2:\n",
        "\n",
        "* `124M` (default): the \"small\" model, 500MB on disk.\n",
        "* `355M`: the \"medium\" model, 1.5GB on disk.\n",
        "* `774M`: the \"large\" model, cannot currently be finetuned with Colaboratory but can be used to generate text from the pretrained model (see later in Notebook)\n",
        "* `1558M`: the \"extra large\", true model. Will not work if a K80/P4 GPU is attached to the notebook. (like `774M`, it cannot be finetuned).\n",
        "\n",
        "Larger models have more knowledge, but take longer to finetune and longer to generate text. You can specify which base model to use by changing `model_name` in the cells below.\n",
        "\n",
        "The next cell downloads it from Google Cloud Storage and saves it in the Colaboratory VM at `/models/<model_name>`.\n",
        "\n",
        "This model isn't permanently saved in the Colaboratory VM; you'll have to redownload it if you want to retrain it at a later time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P8wSlgXoDPCR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0040cae3-0bc1-4daa-ac86-2207d7d84641"
      },
      "source": [
        "gpt2.download_gpt2(model_name=\"124M\")"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Fetching checkpoint: 1.05Mit [00:00, 361Mit/s]                                                      \n",
            "Fetching encoder.json: 1.05Mit [00:00, 2.69Mit/s]\n",
            "Fetching hparams.json: 1.05Mit [00:00, 435Mit/s]                                                    \n",
            "Fetching model.ckpt.data-00000-of-00001: 498Mit [00:19, 26.1Mit/s]                                  \n",
            "Fetching model.ckpt.index: 1.05Mit [00:00, 625Mit/s]                                                \n",
            "Fetching model.ckpt.meta: 1.05Mit [00:00, 3.64Mit/s]\n",
            "Fetching vocab.bpe: 1.05Mit [00:00, 3.16Mit/s]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N8KXuKWzQSsN"
      },
      "source": [
        "## Mounting Google Drive\n",
        "\n",
        "The best way to get input text to-be-trained into the Colaboratory VM, and to get the trained model *out* of Colaboratory, is to route it through Google Drive *first*.\n",
        "\n",
        "Running this cell (which will only work in Colaboratory) will mount your personal Google Drive in the VM, which later cells can use to get data in/out. (it will ask for an auth code; that auth is not saved anywhere)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "puq4iC6vUAHc"
      },
      "source": [
        "gpt2.mount_gdrive()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BT__brhBCvJu"
      },
      "source": [
        "## Uploading a Text File to be Trained to Colaboratory\n",
        "\n",
        "In the Colaboratory Notebook sidebar on the left of the screen, select *Files*. From there you can upload files:\n",
        "\n",
        "![alt text](https://i.imgur.com/TGcZT4h.png)\n",
        "\n",
        "Upload **any smaller text file**  (<10 MB) and update the file name in the cell below, then run the cell."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6OFnPCLADfll"
      },
      "source": [
        "file_name = \"jvg.txt\""
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HeeSKtNWUedE"
      },
      "source": [
        "If your text file is larger than 10MB, it is recommended to upload that file to Google Drive first, then copy that file from Google Drive to the Colaboratory VM."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-Z6okFD8VKtS"
      },
      "source": [
        "gpt2.copy_file_from_gdrive(file_name)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LdpZQXknFNY3"
      },
      "source": [
        "## Finetune GPT-2\n",
        "\n",
        "The next cell will start the actual finetuning of GPT-2. It creates a persistent TensorFlow session which stores the training config, then runs the training for the specified number of `steps`. (to have the finetuning run indefinitely, set `steps = -1`)\n",
        "\n",
        "The model checkpoints will be saved in `/checkpoint/run1` by default. The checkpoints are saved every 500 steps (can be changed) and when the cell is stopped.\n",
        "\n",
        "The training might time out after 4ish hours; make sure you end training and save the results so you don't lose them!\n",
        "\n",
        "**IMPORTANT NOTE:** If you want to rerun this cell, **restart the VM first** (Runtime -> Restart Runtime). You will need to rerun imports but not recopy files.\n",
        "\n",
        "Other optional-but-helpful parameters for `gpt2.finetune`:\n",
        "\n",
        "\n",
        "*  **`restore_from`**: Set to `fresh` to start training from the base GPT-2, or set to `latest` to restart training from an existing checkpoint.\n",
        "* **`sample_every`**: Number of steps to print example output\n",
        "* **`print_every`**: Number of steps to print training progress.\n",
        "* **`learning_rate`**:  Learning rate for the training. (default `1e-4`, can lower to `1e-5` if you have <1MB input data)\n",
        "*  **`run_name`**: subfolder within `checkpoint` to save the model. This is useful if you want to work with multiple models (will also need to specify  `run_name` when loading the model)\n",
        "* **`overwrite`**: Set to `True` if you want to continue finetuning an existing model (w/ `restore_from='latest'`) without creating duplicate copies. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aeXshJM-Cuaf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "60addaf4-034b-4dc1-abe2-131f0a845bba"
      },
      "source": [
        "sess = gpt2.start_tf_sess()\n",
        "\n",
        "gpt2.finetune(sess,\n",
        "              dataset=file_name,\n",
        "              model_name='124M',\n",
        "              steps=1000,\n",
        "              restore_from='fresh',\n",
        "              run_name='run1',\n",
        "              print_every=10,\n",
        "              sample_every=200,\n",
        "              save_every=500\n",
        "              )"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loading checkpoint models/124M/model.ckpt\n",
            "Loading dataset...\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 1/1 [00:00<00:00,  1.75it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "dataset has 100058 tokens\n",
            "Training...\n",
            "[10 | 27.32] loss=3.42 avg=3.42\n",
            "[20 | 49.70] loss=3.36 avg=3.39\n",
            "[30 | 72.42] loss=3.38 avg=3.39\n",
            "[40 | 94.72] loss=3.12 avg=3.32\n",
            "[50 | 116.97] loss=2.90 avg=3.24\n",
            "[60 | 139.43] loss=2.75 avg=3.15\n",
            "[70 | 161.88] loss=2.70 avg=3.09\n",
            "[80 | 184.21] loss=2.72 avg=3.04\n",
            "[90 | 206.58] loss=2.71 avg=3.00\n",
            "[100 | 229.00] loss=2.63 avg=2.96\n",
            "[110 | 251.42] loss=2.39 avg=2.91\n",
            "[120 | 273.81] loss=2.25 avg=2.85\n",
            "[130 | 296.21] loss=2.13 avg=2.79\n",
            "[140 | 318.55] loss=2.27 avg=2.75\n",
            "[150 | 340.93] loss=1.60 avg=2.67\n",
            "[160 | 363.31] loss=2.29 avg=2.64\n",
            "[170 | 385.67] loss=1.64 avg=2.58\n",
            "[180 | 408.06] loss=1.96 avg=2.54\n",
            "[190 | 430.47] loss=1.36 avg=2.47\n",
            "[200 | 452.86] loss=1.31 avg=2.41\n",
            "======== SAMPLE 1 ========\n",
            "t that little violets; the girl was so sweet I can’t put my hand on. I’d rather be dissatisfied.\n",
            "I’ve decided to go on a suicide note until tomorrow, to see whether I can’t finish my diary. It would be a great shame if I did, because I’m busy and it would be foolish for me to stop at nothing. I’m going out too much and am going through an awful lot. But I still feel happy, I’d rather go home to my darling wife – and then I don’t work, I work, I work because I love her deeply, because she’s a human being who’s worth more than all the money she could earn. If I could only count to one, five things: mother nature, kindness, compassion, simplicity, goodness, genius, wit, she would be a star, and there would be love, happiness – but I don’t know her. I could make up for everything by growing older – oh, no, I could do it for the child – I still have so much to do, but I don’t know how.\n",
            "I wish I knew the English language better now. The first thing I did was to select five books that I think will improve my understanding of English: first they ask me to imagine a sentence and then they translate it and read it out in front of me, so that the reader is not so easily deceived: if the author, the translator, dares ask the question, the reader doesn’t know, but after all the questions are obvious – and the character of the book is not one for ambiguous or ambiguous answers either; the reader is warned beforehand of dangers, and warned again when events appear stranger and stranger than they could possibly be! But I can’t do it.\n",
            "I’m dissatisfied with myself because I’ve kept so much confidential about myself for so long. I’ve spent so much time looking after Miss Gard; she’s got everything they need – everything they never see. (Well, I’ll have to do it all by myself if I’m to be truly well these days – but I’m not being selfish, I want to keep all the secrets and pleasures I’ve brought with me into the society.) There’s that nice, long, very, very long, you see.\n",
            "I’m in such a sad mood today. I think that Lena left school in earnest because her father didn’t like her, she was quiet and boring – I would rather go to another girl, but I prefer quieter and nicer companions. Oh, I hope tomorrow that I’ll have a lovely day with her – I’d like to be beside myself.\n",
            "I’ve just sat in the old wooden table that I sat in the old day in the old spot – Lena’s drawing room with the music room behind it. We sat together cosily enough – but it was the old, sad dream that made me want to burst into tears. Lena was sweet and quiet, we also knew one another through our books and our dancing – and her face did the talking – her eyes did the talking – it made me sad, because she did not recall my first time seeing her.\n",
            "It has been a long time since I wrote anything good. We went to the Vechon fair, but the first time, when we arrived at the station, the tram was already half past five and we were tired and a bit bad; after that we walked home at half past six, but it was already dark at half past six. Then we went to the Botorg and went to the Vosgeshe – then to the museum. Everything was the same – dark and deserted streets – and the most beautiful music and flowers and everything looked exactly like Paris. Oh, what shall I do without my notebooks, where are they now? Another year has passed without me writing anything good. I think it’s a wonderful life full of possibility; I’d rather go away now than remain in my old, dismal way, for once in my life I’ll start learning and living a good life, I want to go out for a walk and enjoy nature – but that’s only a diversion, that’s foolish. Now a few months ago – a few months – empty spirits around me, thoughts drifting about – that’s it. It’s over – I’ll go – I have nothing left to do – I spent the summer with my godson – and now life on earth is drifting by and fro around me. He was still a little boy at the time of the death of Macaulay Culkin, who was buried in a small grave in the churchyard beside the grave of his great-great-grandson, which had an elaborate\n",
            "\n",
            "[210 | 488.17] loss=0.75 avg=2.32\n",
            "[220 | 510.61] loss=1.34 avg=2.27\n",
            "[230 | 533.06] loss=1.43 avg=2.23\n",
            "[240 | 555.49] loss=0.97 avg=2.17\n",
            "[250 | 577.90] loss=0.77 avg=2.11\n",
            "[260 | 600.31] loss=0.66 avg=2.05\n",
            "[270 | 622.73] loss=0.63 avg=1.99\n",
            "[280 | 645.11] loss=0.43 avg=1.92\n",
            "[290 | 667.50] loss=0.57 avg=1.87\n",
            "[300 | 689.88] loss=0.45 avg=1.82\n",
            "[310 | 712.26] loss=0.28 avg=1.76\n",
            "[320 | 734.65] loss=0.22 avg=1.70\n",
            "[330 | 757.04] loss=0.27 avg=1.65\n",
            "[340 | 779.43] loss=0.30 avg=1.61\n",
            "[350 | 801.81] loss=0.22 avg=1.56\n",
            "[360 | 824.20] loss=0.37 avg=1.52\n",
            "[370 | 846.59] loss=0.22 avg=1.48\n",
            "[380 | 868.97] loss=0.12 avg=1.43\n",
            "[390 | 891.35] loss=0.19 avg=1.40\n",
            "[400 | 913.74] loss=0.15 avg=1.36\n",
            "======== SAMPLE 1 ========\n",
            " appearance that we shall have to live for the enjoyment of others, not so much for the enjoyment of myself as for other men.– It’s not a pleasant thought to have to say: “You are such a strange creature, I hardly know you’s name; do you consider yourself as a person, then, to be with all men a little unkind, sometimes so very foolish?” That perhaps you’ll find it difficult to understand. I feel as if I’m not a person myself; I’ve changed from before, but there’s a reason, as Misthpern has said before, and that’s something that’s for which all people are guilty; I’m better this year. Oh, how I’ve come to understand that I’ve already lived my happiness for a long time, that feeling of complete fulfilment is coming over and above all my bad, selfish ways. I’ve changed from before, but not so that I can’t express how I feel now; I’m more mature, much more sensible, more sensible I’m going. The sense of duty towards the children is nice, but I still feel a little nasty and wrong, bad in a good way; I think if I do this a lot, I just should a) do my best not to do what others me, b) resist what I do and do my best not to do what others say I can’t do, but I mustn’t be so petty, I must be firm but firm. I must be careful not to do what others say I can, I must be very honest with myself, very simple, very good. The matter of the Coenmarkt has been brought to my attention, I shallanswer soon. The real purpose of this journey is to arrive at “the ideal wife”, not just a writer but one who represents the ideal of nature, that is, a married woman will not only have an unhappy marriage but could well destroy the couple’s happiness. The question of the fate of the Beagle is a sensitive one, for as David Copperfield once said, all doubts and fears and anxieties are now resolved in the eyes of the people. This great spiritual man, who raised the young Eagle above the lowly elders, says: “Who will succeed him” Who will save the little boy? The little boy will be the beginning of a fine life, yes it is I who say to myself: how can I make a person happy? But the child is the soul of this happy Union, “he who builds and sustains homes for the little,”””” and every hour he takes his lamp and his fan, it lives and grows and will yet forever and ever its inhabitants. The Lord has shown me the way. Amen. Joshua VII: (chap. iv) The Kingdom Come Kingdom (chap. v) The New Jerusalem and the new year with me, oh I do so much resent the day of summer and winter, when we once again spilled over us both the glorious New Year and the divine spring. And I have such a bad idea about it,^ I’m tired, I don’t know what to do with myself; what shall I do? But what can I do, so I can –?\n",
            "I’m longing for a photograph, what better opportunity can I present myself than by way of a candid picture of my little face?\n",
            "A letter from Dries came today, my dear Dries, oh I know that sometimes we see each other quite happily together, but then again this morning we had a dance, but that did me a lot of good for the evening.\n",
            "It’s been a sad day for me. I’ve felt so unhappy and unhappy I haveの眴miscolledmyreligion ,andodicyclics on purpose , but what good does it do me, I don’t know. I’ve also had dreams, but they were very unpleasant and I’ve had myself carefully considered and treated for my dream–dreaming.\n",
            "It’s over, it seems over, I was sad yesterday and I offered my resignation, but the young woman you married, before she died, then she left and I did not accept her at first. What will I do now?\n",
            "I’m going to read over what I wrote about him yesterday, to reflect.\n",
            "I went skating with Annie this evening, for a walk through the woods in the early evening. We listened to Mother Shelley in her gloriously beautiful sings, tired, hopelessly sad, then we walked together in the evening to the National Gallery, where I was received admirably. In the evening we danced for a while and then got on quite well, but I really wanted to regress into my old ways\n",
            "\n",
            "[410 | 947.60] loss=0.12 avg=1.32\n",
            "[420 | 969.99] loss=0.11 avg=1.29\n",
            "[430 | 992.40] loss=0.09 avg=1.25\n",
            "[440 | 1014.81] loss=0.18 avg=1.22\n",
            "[450 | 1037.21] loss=0.09 avg=1.19\n",
            "[460 | 1059.59] loss=0.11 avg=1.16\n",
            "[470 | 1081.97] loss=0.10 avg=1.13\n",
            "[480 | 1104.36] loss=0.09 avg=1.11\n",
            "[490 | 1126.74] loss=0.10 avg=1.08\n",
            "[500 | 1149.13] loss=0.09 avg=1.06\n",
            "Saving checkpoint/run1/model-500\n",
            "[510 | 1174.22] loss=0.07 avg=1.03\n",
            "[520 | 1196.68] loss=0.09 avg=1.01\n",
            "[530 | 1219.09] loss=0.07 avg=0.99\n",
            "[540 | 1241.45] loss=0.08 avg=0.96\n",
            "[550 | 1263.82] loss=0.07 avg=0.94\n",
            "[560 | 1286.22] loss=0.08 avg=0.92\n",
            "[570 | 1308.61] loss=0.06 avg=0.90\n",
            "[580 | 1331.01] loss=0.09 avg=0.88\n",
            "[590 | 1353.39] loss=0.08 avg=0.87\n",
            "[600 | 1375.77] loss=0.08 avg=0.85\n",
            "======== SAMPLE 1 ========\n",
            " so that is one of the attractions of Mona Lisa. I also enjoyed my visit to the museum, which was actually Miss World wearing his best, and the dinner was ahead of me, not too full and fresh but still consisting of very good food and people, and a very pretty, fine evening. To-night I must set to work in elevating and painting our houses.I won’t be able to do all that embroidery, I’m much too lazy and shouldn’t be making anything for years to come. I must make enough of the rustic old frames for my lamp and the steam powered frames for the steam powered stove. I must make plenty of room in the ceilings and so on.I’ve just finished my portrait of Pauline Bayle. I’ve lived here for more than a year, I have not been accustomed to everything, but it seems to me quiet and calm here. I wonder how it is that after so long lives here, my curiosity reawakened itself very late.\n",
            "For the first time I’ll be able to write here again after the whole day of hard work which I spent doing nothing for nothing in the house. Sunday morning is going away, I haven’t been at all nice, to go to church, be there with my mouth to nothing. I had a dreadful hour on Sunday where I cut down a sad tree and found one of those little pink kittens, oh I wish I was with them in my heart, I wish I could comfort them. I just feel my first long thirst, oh how silly and selfish I sound.\n",
            " iPPOE and its related books have been hugely influence on me. Sunday mornings are the most important religious day of my life; I sit so quiet and do nothing, and on Sunday afternoon I read something in the newspaper that said: OBAMA! \n",
            "I’ve never really known a reason why the British people should approve or disapprove of a posthumous award of the like. Now I’m finally going to get down to work, so I shall return to Waterloo at once. summertime here, I appreciate its solitude and serenity much, but then it comes to the normal routine of a small family, and I shouldn’t want to repeat myself. I now have work in me again, but it must be fun, and dangerous, and in that I shan’t be dull; I feel that I must create my own happiness! If only I could do something to make it pleasant here, for I’m already tired of the idea that I’d become a little grumpy again. I can’t even do simple work; I want to create for myself something nice and relaxing here, instead of doing my work for the few who are here and who want to, I’ll do it for them myself. I can’t do simple householdwork; I want to express myself in something original and good, so that when people see me in public who are actually I’s friends, I can’t manage to do that at all; then I’ll be much more cool and shy then, and people will think I’m vain or stupid instead of me, so I can’t be serious in the most vulgar, most simple things. I’ve started a boardinghouse in Elburg and, like many poor artists, it’s not built on nor that of cedar, so there’s scummy acacia in the handle, and all the furniture is from a long time ago. The building is elegant and relatively new, the ceilings are beautiful, the marble from old factories looks beautiful and fresh, the windows are beautiful and the lighting works look beautiful and so on and so on. The building, I’d expect from a Dickens novel, is a bustling boarding house with elegant brandy in the windows; a healthy, healthy many who come here. The building is like a large noble house in a Dickens novel, with its happy patients among the trees; an old woman with a simple red coat, she looks sad and quiet, and the man with a scarlet scarf is kneeling down and smiles winceingly as he looks at him. The crowd of people gathering around the fire makes a racket when it touches us; a couple stand at the window and stand very close to each other, mouth agape as they do, while a very large crowd gather around the fire, some even sit together in front of the fire, others sit side by side with us and inhale deeply. The fat man in the white shirt and the slim, thin one, handsomely dressed. with velvet tie and a purple tie. He seems to be explaining something to us, but he doesn’t say what it is, and sometimes he gives it to us with a smile, or signals with his fingers what it is he is – then he puts it back in his mouth and\n",
            "\n",
            "[610 | 1409.65] loss=0.05 avg=0.83\n",
            "[620 | 1432.03] loss=0.07 avg=0.82\n",
            "[630 | 1454.45] loss=0.06 avg=0.80\n",
            "[640 | 1476.87] loss=0.06 avg=0.78\n",
            "[650 | 1499.30] loss=0.06 avg=0.77\n",
            "[660 | 1521.72] loss=0.06 avg=0.75\n",
            "[670 | 1544.11] loss=0.06 avg=0.74\n",
            "[680 | 1566.51] loss=0.05 avg=0.73\n",
            "[690 | 1588.91] loss=0.07 avg=0.71\n",
            "[700 | 1611.28] loss=0.05 avg=0.70\n",
            "[710 | 1633.65] loss=0.06 avg=0.69\n",
            "[720 | 1656.03] loss=0.05 avg=0.67\n",
            "[730 | 1678.39] loss=0.07 avg=0.66\n",
            "[740 | 1700.76] loss=0.05 avg=0.65\n",
            "[750 | 1723.11] loss=0.06 avg=0.64\n",
            "[760 | 1745.48] loss=0.05 avg=0.63\n",
            "[770 | 1767.88] loss=0.07 avg=0.62\n",
            "[780 | 1790.27] loss=0.07 avg=0.61\n",
            "[790 | 1812.64] loss=0.06 avg=0.60\n",
            "[800 | 1835.03] loss=0.05 avg=0.59\n",
            "======== SAMPLE 1 ========\n",
            " a friend in the churchyard with her little child, gave me a letter in which he confirmed my belief that he loves me and wants to marry me. He works with me – he wants to spend his whole life with me – oh, it’s such a wonderful dream, where everything is empty and forgetful – where everything is clear and comforting – oh, when he comes to me he kisses me, touches me and says: “I’ve done a lot of good, I thank you for that papa’s note – oh, that’s good.– I thank him for the note and oh, that is the last straw for me – where was my peace for a long time? That man is gone – gone mad, who knows what he thought of me – but above all that poor young girl – and above all – I, who love him heart and soul – I no longer believe him – I no longer trust him – oh, if only I could do something – to be more true to myself – but to wait until after work! This must be it!\n",
            "Oh, I wouldn’t like to have it so bad – processed and pasted together so that the end justifies the means – impossible to us mortals.– Who would have thought that a wandering, unhappy soul could travel the world, open new vistas, become richer and richer – oh, we human beings of our time, we must have that goal in mind.\n",
            "I haven’t written for days and yet I managed to write my last – at the beginning of December – and now! December! – No, I’m not going to repeat it, because that’s how it goes – everything is situational – I had a wonderful life before, but now – my last – happy one.\n",
            "How I Love My Mother – by Huet\n",
            "As I leafed through the book I came across a beautiful image. High on my head these words were whispered about me – and yet I felt so free, so normal – there was something so small thatmeans so much, something thateveryman – if you will just live through it. And how beautiful it is in the sunshine, when people are askew and play sound, and people are askew and whine too. But it was more than I expected – I was weak, nothing else I could do. And I, I am so weak! I want to upset my nature, make it sweeter, something for my heart. Huet!\n",
            "I haven’t written for long, but I have had a great many letters – from all over the world – and from many sympathetic mouths – so it was a good day today. That’s what I do – I read everything, even though it’s not much, I read what he says about culture and literature, I also read his little book, but it really doesn’t do me any good.\n",
            "What I enjoyed most of yesterday were the streets of Amsterdam – bustling with people enjoying themselves and all the atmosphere was very good. I usually stay at home in the evening – till one evening, suddenly I am not in a good mood again and go to bed very early. I don’t know what is, but when I go upstairs I find Mien too miserable to go to the window or at least sit down in my chair. I wish the painting was soon gone, it was unpleasant of me.\n",
            "What a series of lovely evenings. Yesterday I received two lovely letters, one from Lize van Gogh, another from André and Mon, from André I think; a very kind, nice letter, only I don’t have a very good one from them yet; at least I couldn’t send one myself. And I have to keep looking for work; after all I can do nothing but lie in wait and wait. Then in the morning we go to Chaudfontaine and Spa, and then it all starts again in a week. Oh, if only I could just go to Chaudfontaine again and get some rest; I wish Chaudfontaine was a nice resort city; it makes me more cheerful, I want to go there. And on the other hand I don’t know how to sing because of the lack of sounds for the violin lesson, so I just don’t know how I should sing. I wonder how the evening would have gone. Oh, I could have danced for a while at home, with my father or with my little sister; I doubt it. I should also like to see the pictures of Piet and Agnes with the Three Sisters; that mustn’t happen again. Last week I had a long talk with Mrs Doorman and had a wonderful talk; it made me happy again. I haven’t seen Gidius for a long time, but when he came to see us in the evening he made me feel special, something seemed to me as if it was\n",
            "\n",
            "[810 | 1868.83] loss=0.05 avg=0.58\n",
            "[820 | 1891.21] loss=0.05 avg=0.57\n",
            "[830 | 1913.58] loss=0.05 avg=0.56\n",
            "[840 | 1935.94] loss=0.07 avg=0.55\n",
            "[850 | 1958.34] loss=0.06 avg=0.54\n",
            "[860 | 1980.72] loss=0.05 avg=0.53\n",
            "[870 | 2003.10] loss=0.06 avg=0.53\n",
            "[880 | 2025.47] loss=0.05 avg=0.52\n",
            "[890 | 2047.84] loss=0.05 avg=0.51\n",
            "[900 | 2070.24] loss=0.08 avg=0.50\n",
            "[910 | 2092.63] loss=0.07 avg=0.50\n",
            "[920 | 2115.02] loss=0.05 avg=0.49\n",
            "[930 | 2137.42] loss=0.05 avg=0.48\n",
            "[940 | 2159.80] loss=0.04 avg=0.47\n",
            "[950 | 2182.20] loss=0.05 avg=0.47\n",
            "[960 | 2204.61] loss=0.06 avg=0.46\n",
            "[970 | 2227.01] loss=0.05 avg=0.45\n",
            "[980 | 2249.44] loss=0.05 avg=0.45\n",
            "[990 | 2271.87] loss=0.05 avg=0.44\n",
            "[1000 | 2294.28] loss=0.04 avg=0.44\n",
            "Saving checkpoint/run1/model-1000\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/training/saver.py:1068: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to delete files with this prefix.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IXSuTNERaw6K"
      },
      "source": [
        "After the model is trained, you can copy the checkpoint folder to your own Google Drive.\n",
        "\n",
        "If you want to download it to your personal computer, it's strongly recommended you copy it there first, then download from Google Drive. The checkpoint folder is copied as a `.rar` compressed file; you can download it and uncompress it locally."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VHdTL8NDbAh3"
      },
      "source": [
        "gpt2.copy_checkpoint_to_gdrive(run_name='run1')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qQJgV_b4bmzd"
      },
      "source": [
        "You're done! Feel free to go to the **Generate Text From The Trained Model** section to generate text based on your retrained model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pel-uBULXO2L"
      },
      "source": [
        "## Load a Trained Model Checkpoint\n",
        "\n",
        "Running the next cell will copy the `.rar` checkpoint file from your Google Drive into the Colaboratory VM."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DCcx5u7sbPTD"
      },
      "source": [
        "gpt2.copy_checkpoint_from_gdrive(run_name='run1')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RTa6zf3e_9gV"
      },
      "source": [
        "The next cell will allow you to load the retrained model checkpoint + metadata necessary to generate text.\n",
        "\n",
        "**IMPORTANT NOTE:** If you want to rerun this cell, **restart the VM first** (Runtime -> Restart Runtime). You will need to rerun imports but not recopy files."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-fxL77nvAMAX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8938432a-3b86-4102-f32b-362721ecb897"
      },
      "source": [
        "sess = gpt2.start_tf_sess()\n",
        "gpt2.load_gpt2(sess, run_name='run1')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loading checkpoint checkpoint/run1/model-1000\n",
            "INFO:tensorflow:Restoring parameters from checkpoint/run1/model-1000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ClJwpF_ACONp"
      },
      "source": [
        "## Generate Text From The Trained Model\n",
        "\n",
        "After you've trained the model or loaded a retrained model from checkpoint, you can now generate text. `generate` generates a single text from the loaded model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4RNY6RBI9LmL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cd9574f8-1069-4249-bfd1-a52269186247"
      },
      "source": [
        "gpt2.generate(sess, run_name='run1')"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Artificial light – the future is always beautiful – but there are also thousands of other wonderful things – people, castles, vaults, metallurgy, iron and steel, plumbing, everything imaginable, we live in – and yet there’s something so dull and drab, so peaceful and peaceful here.\n",
            "Another wonderful thing is the view from the sea from shore. If I’d seen anything like it here before, it’d be a full moon today, then I would be steamroasted – I don’t know how – there was something very livelier than here, afternoons of bliss and nightmares slept in the trees, read Languedocs and Isaacs and now finally this evening – oh that first night when my eyes were dry and wet and empty mouths begging me to join them, and I woke up cold and decided to never think about anything like this again.\n",
            "The most delightful hours were these, I fly high when I have nothing but work to do and pleasure to experience – Nell Lacy for the first time, oh, how beautiful it is, how pleasurable it must be to feel something.\n",
            "Well, my bad, my bad day. It was a good day, a good day. We did our homines number in, because I get a good book and I’m good for a living, but before we begin to read our book review, I want to try to remember a small ray of light that went over the cover of your book. It was published in the London Review of Books, \"The Life of Charlotte Brontë,\" and it made the morning more delightful than ever.\n",
            "I had a lovely day with Marie yesterday, it was very pleasant, as usual, and I was very glad for her. I shan’t write much in my diary, though, there’s not much opportunity here. Dries and Annie are going to stay and I shall do my best to stay above all others.\n",
            "I had a lovely day with Anne on Sunday, I really enjoyed them both. I shan’t write much in my diary, though, there’s not much opportunity here.\n",
            "The domesticity and luxury of bourgeois life are not things to which I can devote a whole volume, but I have experienced so much domestic oppression! If I were free myself, one could say that I was held against my will, and that was the worst of all. I had my firstand was treated as a second-class citizen, I could also be beaten, robbed of my freedom, my everything. I had his books and his books – his house and his everything – he had them on his own, he took them and gave it to me, oh, it was so beautiful, it seemed so beautiful, it seemed soian, it was as if our hands were in harmony, like flowers thatossom together. I had my first justice for my cause, and I rise from my apace free and clear.\n",
            "Kenelm Chillingly is the best book I know; Gidius really must read it, I shall be interested to hear what he thinks of it.\n",
            "I had a nice day with Anne on Sunday, I really like her. They’re both very good friends, one is witty and amusing and the other profoundly serious, they both possess the spirit to go further than most people. Gidius is better at conversing with his companions.\n",
            "I went to the masked ball as Little Red Riding Hood; what a difference with the ladies like that. The only thing I didn’t like was the sight of the masked woman at the ready with her butterbeer that was all over the table, I don’t think I will take that with me for long.\n",
            "I lunched to-day at Mr De Clercq’s. A most comfortable, elegant, refined, surrounding; how nice for Mary to live there always! \n",
            "Nearly a month that I have not written. It is true no event occurred that day as I had expected, but it was very nice; besides the usual refreshments there were also some sweets, a couple of pence biscuits, one, a coffee tree and one for the child. Miss Gard also died of a self-imposed cancer, another of which she was ridiculed. That being so, it was a nice end to a month. That I shall never forget it.\n",
            "In the mean time my thoughts are with Marie and with all those who have supported me recently. I feel so much at home here than I did a year ago; I write to everyone I can find, to make sure everyone has something to look forward to when I return. I heard a beautiful sermon by Stopford Brooke to-day, there was a lady with a well-known silver and goldsmith’s shop, they talked about their wonderful trips to the West Indies and their visit to that great spot in South America. How often I have thought of Brooke, how\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oF4-PqF0Fl7R"
      },
      "source": [
        "If you're creating an API based on your model and need to pass the generated text elsewhere, you can do `text = gpt2.generate(sess, return_as_list=True)[0]`\n",
        "\n",
        "You can also pass in a `prefix` to the generate function to force the text to start with a given character sequence and generate text from there (good if you add an indicator when the text starts).\n",
        "\n",
        "You can also generate multiple texts at a time by specifing `nsamples`. Unique to GPT-2, you can pass a `batch_size` to generate multiple samples in parallel, giving a massive speedup (in Colaboratory, set a maximum of 20 for `batch_size`).\n",
        "\n",
        "Other optional-but-helpful parameters for `gpt2.generate` and friends:\n",
        "\n",
        "*  **`length`**: Number of tokens to generate (default 1023, the maximum)\n",
        "* **`temperature`**: The higher the temperature, the crazier the text (default 0.7, recommended to keep between 0.7 and 1.0)\n",
        "* **`top_k`**: Limits the generated guesses to the top *k* guesses (default 0 which disables the behavior; if the generated output is super crazy, you may want to set `top_k=40`)\n",
        "* **`top_p`**: Nucleus sampling: limits the generated guesses to a cumulative probability. (gets good results on a dataset with `top_p=0.9`)\n",
        "* **`truncate`**: Truncates the input text until a given sequence, excluding that sequence (e.g. if `truncate='<|endoftext|>'`, the returned text will include everything before the first `<|endoftext|>`). It may be useful to combine this with a smaller `length` if the input texts are short.\n",
        "*  **`include_prefix`**: If using `truncate` and `include_prefix=False`, the specified `prefix` will not be included in the returned text."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DKMc0fiej4N",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "48679c06-dd39-481f-9159-c8e79313fbdb"
      },
      "source": [
        "gpt2.generate(sess,\n",
        "              length=250,\n",
        "              temperature=0.7,\n",
        "              prefix=\"When I reflect upon Vincent's legacy, I\",\n",
        "              nsamples=5,\n",
        "              batch_size=5\n",
        "              )"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "When I reflect upon Vincent's legacy, I see that it’s not about him showing off (what he’s good at), it’s about showing the way. If people think about it all the time, they’ll get it wrong. I think about it all the time today – how prideful and how prosperous and prosperous Theo always was – that’s how he’d do it. He worked hard, wrote much, studied, was busy and I still think about him – today – how happy and grateful I’d been recently. I shall try to be a little bit kinder and better this week and then we’ll see how it goes.\n",
            "It’s the evening before Whitsun, how we’d see him at home – I couldn’t put my hand around it so that it didn’t turn out poorly, but still I felt happy and grateful for the wonderful view from his window. Oh, that beautiful, wide-eyed child – how real that feeling was for me – oh, that sweet, pure, pure feeling that was Theo. I’ve lost him, my dear husband – oh, my loss.\n",
            "Winter is approaching – and I’m\n",
            "====================\n",
            "When I reflect upon Vincent's legacy, I see that it’s not about one iota, but over a lifetime. I may be poor now, I may be rich, I may be in love and marriage right at this moment in time to which he’s so keen – I may even be married at last – but that’s another story.\n",
            "Tomorrow I’m going to Anne’s – that makes me the mother’s right of free choice.\n",
            "I didn’t have time to write any more this afternoon and I wasn’t in a good mood either; Anne is so sweet, I can’t help thinking about her. It’s certain that one thing in particular binds us – our hands together so tightly; I can’t help thinking about Topsy too.\n",
            "\n",
            "The Christmas holidays are over and we’re back in La Hague. We were both from Amsterdam and went to St Paul’s Church a number of times; it was a lovely little church, the congregation was very nice, it was pleasant in the small way that I was led, but the real fun was in the drawing room and the singing room. Oh, how I’ve enjoyed myself recently,\n",
            "====================\n",
            "When I reflect upon Vincent's legacy, I see that it’s not so very easy to understand. He saw that his heroes are difficult souls, but nevertheless fall short of sacrificing their own worth and worthlessness for the sake of that of another. Oh why, why is it so difficult to understand, when one can see it in action, how precisely this must be the case for Vincent. He’s spent his whole life, indeed he’s already lived for almost a generation to perfection, but when he looks back now, he feels so ashamed, so despondent, he couldn’t do it to him. I think: he made a person ugly, rich and noble, he made him a coward, a dissembler, who lacks all the self-knowledge and self-knowledge of a dissembler. Not a whit of that noble, noble, this is precisely the reason why I’ve called him a fool and a dissembler. Oh, if only I could always know what I’re talking about – if I said that having one little bit of common with you is a good thing, I’d be condemned. But oh, I don’t have anything better to do than write again.\n",
            "I\n",
            "====================\n",
            "When I reflect upon Vincent's legacy, I see that it’s not easy to remember the golden years. When I was in London I applied and was confirmed and I think: it couldn’t have been more cheerful and kind when she was with Anne and Marie. They invited me to join them to London – which was a wonderful idea, as it gave me safe passage to England for the purpose of writing. But I’m too stupid to apply and get on; at least I can’t begin the English examination. I would be foolish not to apply; at least I shall go to London in a week. I would also really rather go to Paris, I would at least get my examination started on the right track.\n",
            "I am packing my box for the examination. I wrote “midnight reflection” on one side of my face and a note “I don’t really feel very much like going to the opera with my husband” on the other. I would be foolish not to go. I am very sorry for Marie, she’s so worldly and cultured; I feel much more at ease with her. I haven’t heard from Gidius in the last few days; has he left town? Or\n",
            "====================\n",
            "When I reflect upon Vincent's legacy, I see that it’s not just a name change, either – he went on to become a householder, a nobleman, and one of the most learned people I’ve known. He was the first to realize that living by rules is bad, and he developed a position of great intellectual and moral importance that remains ever so small. He also knew that living by them all the time, he named after him – Beb. and his three small children: Johannes, Lien and E.; he also raised chickens and lent them to children and grandchildren. He is greater than life and above all, above all, on earth – and above all, what can I do for him?\n",
            "I’m not a brand new person, it seems as though I’ve changed recently; new clothes, nicer homes, more attractions, more people, what can I do?. Old clothes and old man – oh, it gets so cold and bitter sometimes. But he’s the same as always, I’ve changed – I’ve moved from being a noble, respectable, respectable person to something a bit more like a lamb who’s been ground to a lamb’s liver – by D\n",
            "====================\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "gpt2.generate(sess,\n",
        "              length=250,\n",
        "              temperature=0.7,\n",
        "              prefix=\"When I reflect upon my legacy, I\",\n",
        "              nsamples=5,\n",
        "              batch_size=5\n",
        "              )"
      ],
      "metadata": {
        "id": "EsH_80bHxJK0",
        "outputId": "77f4bc05-a3d1-46a3-d97b-846ad3f91cda",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "When I reflect upon my legacy, I remember the exhilarating sense of victory over my oppressor; to leave this life I feel only as if I were taken from there, I mustn’t for one moment fancy to be taken – but then I would not feel so victorious, not now, not ever. Oh, I shouldn’t have done that, I should have been afraid, I should have been so proud, it would have been so wonderful to her, it would have been so wonderful to her, it would have been so wonderful, oh, it’s all a dream, it can’t be true, it can’t be true, is it true, is it true what Shakespeare had in mind? I ask myself this evening, what wonderful thoughts and feelings I’ve had. Dreams of something good, of becoming. A person like that doesn’t work.A dreamer, on the other hand. Heists, conquers,missions,rosences,heirs, he’s just been patient enough. He’s paid a visit to Martha, he’s having a chat about her “love” on his own, hasn’t talked to her for years, and\n",
            "====================\n",
            "When I reflect upon my legacy, I believe that it’s all but empty. I shall go and study hard, so that my time will pass quickly.\n",
            "Afterwards we had tea, and then went for a walk through the Boschjes searching for the moon that our host had to study one more time because he couldn’t find it in one of his books. It was divine in the Boschjes.\n",
            "I’m back in Amsterdam. I suddenly got such a longing that I couldn’t stay any longer. It was very stupid of me.\n",
            "I went to the masked ball and played in it for the first time for the first time for the first time for the first time! Only I didn’t know what to do. I was so longing for a picture of the hero with Ajax, or Marie Antoinette, or any of those little girls, they looked so insignificant in front of them, or sat looking at him with desire in his quiet little room. I couldn’t understand myself; was I really thinking of being a bit ugly, too, or was my mind made up? I don’t really like myself in general, and I was so very foolish this whole time.\n",
            "It\n",
            "====================\n",
            "When I reflect upon my legacy, I couldn’t resist. Graduation is a bitter memory for me. I expected so much from my work, I was so delighted to see Gidius again, so warm and happy again. But I couldn’t bear the thought of working hard and sacrificing so much. I hope that our happiness will never be as that; it’s just a dream, a fickle person he does not want to be with anyone. I can understand why people are so fond of Marie; it’s not because of her clothes, she’s so elegant, so elegant. I could make a beautiful dress out of her; that would be a wonderful idea, but I don’t know how to go about it. The most difficult thing is finding the right tone and melody for it. If Dries were here now he would soon be music for the ears of everyone, I hope he will at least be heard.\n",
            "It’s strange working with Dries, it makes me better, but I wish that he would start studying music, I always feel better. How strange it is, being here so long, so far away, yet I can’t find the words for it. It can�\n",
            "====================\n",
            "When I reflect upon my legacy, I maintain it will last a long time. Gradually, I’ll get to know myself a bit, taste a bit and perhaps a bitdefine my mind, it will be a lot to repay God by making a person happy.A lot of people have congratulated me on this, but I don’t feel very happy. I would have liked to say that I shan’t be a hero, I would have been looked down on as a fool and I would have been looked down on as a coward, but I don’t feel unhappy. I also know what it is like to be a small child; there’s nothing pleasanter than that! A few days ago we sat together in the twilight at a table and had a nice little chat; he looked so cheerful and kind as he ate; then he left. I shall never forget that look of triumph on his face when he saw that little boy’s eyes would open and he would say: “I have very little to tell you”. \n",
            "Yesterday night we went for a beautiful walk in the Bosjes River; the lighting and smell of the flowers glorious, the fragrance of the fruit trees; the setting gave a\n",
            "====================\n",
            "When I reflect upon my legacy, I remember the friendly, affectionate greeting I received at the start of the new year; it made me happy to hear that it would eventually be a happy one.\n",
            "It’s unfair to award everything to Marie because she stayed with me for a short while and then left again – I would have been well off with that kind of a man, I would have been happy – and now I have nothing to which to turn back. What a strange feeling it is to have been with such a person, to have made that decision, to have made that choice for so long, to have lost him. It makes me tremble with sadness at the mere thought of what I’ll do under such a false name.\n",
            "I didn’t write anything all that evening, but still, I didn’t want to be ungrateful – and now I have to stop. It doesn’t matter what happens with me after this year; I’ll still love and care for him, we still have years left on our hands, and he’ll still be young and beautiful and rich and powerful.\n",
            "I’m in such a foolish mood today. Marie and I laughed like a pair of silly children\n",
            "====================\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "gpt2.generate(sess,\n",
        "              length=250,\n",
        "              temperature=0.7,\n",
        "              prefix=\"I sometimes ponder my greatest regret.\",\n",
        "              nsamples=5,\n",
        "              batch_size=5\n",
        "              )"
      ],
      "metadata": {
        "id": "9-ry26vxxI7b",
        "outputId": "f3d36850-2c7e-40d0-9dd3-6cf7619418cf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "I sometimes ponder my greatest regret. If I had not spent so many wonderful years growing and keeping beautiful memories of my youth in Dulwich – I might have lived in Dulwich! \n",
            "It was a delightful young life at school – at least this year. Why was it so? What was the difference between the short time they went without and the whole life in which they spent it? I always asked myself this question when I was in London – and little Wight was the first to notice it. \n",
            "In London I had a very good time – at the recent Christmas party in the Museum – but there was something oppressive – almost unbearable. There was something oppressive at the table – I almost gave in to it. I am so used to it – so used to everything – it took me longer than normal. Now I have to do it again; I must do my best to be good as soon as possible.\n",
            "It was a lovely day at Christmas; I sat on the floor in front of the bookcase and wrote my notes – after having spent the whole day in the country, Ihit and Ihitaka came to collect us. We were two lovely children – I felt so proud of him; I saw him in the midst of all the children and staff and\n",
            "====================\n",
            "I sometimes ponder my greatest regret.\n",
            "It’s true that one can’t truly regret anything one’s done; the day after the operation the day after the funeral, the following day, the following week, but it was still a wonderful, wonderful life. And then there was the reception, the famous Charlie Eliot, with all those good people, I think, and then that visit to Nunspeet, where I stayed with Mrs Van den Bergh. Oh, those three weeks, those three months, I was still young, I still had many, many things to live for, I still had my childhood holidays to go back to Amsterdam, I still remembered many things from that visit, from that visit to the library, from that visit to the church, oh, my life has not gone sooynously any of that.\n",
            "Now I’ve had my fun and now it’s over for good. My life is in turmoil, as I thought when Dries left and I never thought that I might soon realize anything that I wrote about him. First I had to accept that I had suffered, oh, my life, and I only had one hour left! And that hour was coming, and I’ve already lived\n",
            "====================\n",
            "I sometimes ponder my greatest regret. When I was in London I had a lovely evening with Marie, it was a wonderful evening. They gave me a Marie Gonne – and I was very happy. How long I haven’t told them about our engagement, or about Anne, or about our dear Rie, or about our dear Rie, or about all the little things. I always forget what they are; I really don’t understand how it happens.\n",
            "Now I’ve read Margarethe’s diary in the Sabbath des Levens, I feel for the first time what is missing in mine; after all, she has a monotonous life, too, and yet she writes with such a brisker, more poetic tone. She writes about being at home with Dries, or rather I did not find the book for quite some time, but for several days and in all that time there has been no one to help me and in that time there has been struggle and strife, strife without strife. For if one is not a person at all and one cannot be anything one wants one cannot delight in, in those days there have been moments of great, great happiness, for I’ve felt it myself. And those\n",
            "====================\n",
            "I sometimes ponder my greatest regret. When I was in London I had a lovely evening hour with Annie; I think they both had some lovely words for me. I can’t decide though “Fine, I’ll miss her a lot” or sorry, sorry, I can’t say either. Annie’s an ordinary, insignificant little girl; poor, poor, poor. She can’t stand it, she can’t stand it, she’s too indifferent to everyone in her house, too indifferent to money, she can’t do anything about it, she’s an object of desire only; in my opinion my dear little girl, my dear little love, my darling, I can’t stop myself.\n",
            "I’m not back yet. You can’t send me a book, I’ve to start a book now – yes, I shall read over my first book but it’s so pleasant now. I read too much sentimentalism now, it’s good to have some order.\n",
            "I have no concept of decor and no notion of proper feeling – oh, what can I improve on once in a while still having this feeling of being superior\n",
            "====================\n",
            "I sometimes ponder my greatest regret. How well I remember the day on which I knelt before the throne of the church of St Paul! I felt the indescribable pain of that day, and the indescribable pleasure of that day. I would have liked to be a writer, but I don’t know how. It can’t be helped. And now I shall stop writing and earnestly think about something other than school and duty. I must also think about my health. Would I really be a bit weak and listless? There’s nothing worse than being weak and miserable, and I’m so very ill now that I don’t know how I’ll manage without it for a long time. It will get better once in a while I say something about my letter to Dries, but it’s useless now.\n",
            "I’m in such a foolish mood today. Marie and I laughed like a pair of silly children, but it does us good.\n",
            "I’m in a sad mood again today. Ours was not bliss – happy and prosperous for our people – a world in which one can’t help but be unhappy. And it does us good.\n",
            "I\n",
            "====================\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zjjEN2Tafhl2"
      },
      "source": [
        "For bulk generation, you can generate a large amount of text to a file and sort out the samples locally on your computer. The next cell will generate a generated text file with a unique timestamp.\n",
        "\n",
        "You can rerun the cells as many times as you want for even more generated texts!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fa6p6arifSL0"
      },
      "source": [
        "gen_file = 'gpt2_gentext_{:%Y%m%d_%H%M%S}.txt'.format(datetime.utcnow())\n",
        "\n",
        "gpt2.generate_to_file(sess,\n",
        "                      destination_path=gen_file,\n",
        "                      length=500,\n",
        "                      temperature=0.7,\n",
        "                      nsamples=100,\n",
        "                      batch_size=20\n",
        "                      )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0-LRex8lfv1g"
      },
      "source": [
        "# may have to run twice to get file to download\n",
        "files.download(gen_file)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QQAN3M6RT7Kj"
      },
      "source": [
        "## Generate Text From The Pretrained Model\n",
        "\n",
        "If you want to generate text from the pretrained model, not a finetuned model, pass `model_name` to `gpt2.load_gpt2()` and `gpt2.generate()`.\n",
        "\n",
        "This is currently the only way to generate text from the 774M or 1558M models with this notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hsUd_jHgUZnD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 158
        },
        "outputId": "4e0c8a3f-3527-41c4-e3fe-3357f3f8f6c2"
      },
      "source": [
        "model_name = \"774M\"\n",
        "\n",
        "gpt2.download_gpt2(model_name=model_name)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fetching checkpoint: 1.05Mit [00:00, 354Mit/s]                                                      \n",
            "Fetching encoder.json: 1.05Mit [00:00, 131Mit/s]                                                    \n",
            "Fetching hparams.json: 1.05Mit [00:00, 279Mit/s]                                                    \n",
            "Fetching model.ckpt.data-00000-of-00001: 3.10Git [00:23, 131Mit/s]                                  \n",
            "Fetching model.ckpt.index: 1.05Mit [00:00, 380Mit/s]                                                \n",
            "Fetching model.ckpt.meta: 2.10Mit [00:00, 226Mit/s]                                                 \n",
            "Fetching vocab.bpe: 1.05Mit [00:00, 199Mit/s]                                                       \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BAe4NpKNUj2C",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        },
        "outputId": "b09bfe1d-2ff8-4b8a-fffb-273d28d5d4ae"
      },
      "source": [
        "sess = gpt2.start_tf_sess()\n",
        "\n",
        "gpt2.load_gpt2(sess, model_name=model_name)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0828 18:37:58.571830 139905369159552 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to check for files with this prefix.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Loading pretrained model models/774M/model.ckpt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-xInIZKaU104",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 797
        },
        "outputId": "56348e28-7d08-45e3-c859-f26c0efd066d"
      },
      "source": [
        "gpt2.generate(sess,\n",
        "              model_name=model_name,\n",
        "              prefix=\"The secret of life is\",\n",
        "              length=100,\n",
        "              temperature=0.7,\n",
        "              top_p=0.9,\n",
        "              nsamples=5,\n",
        "              batch_size=5\n",
        "              )"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The secret of life is that it's really easy to make it complicated,\" said Bill Nye, the host of the popular science show \"Bill Nye the Science Guy.\" \"And this is one of the reasons why we all need to be smarter about science, because we can't keep up with the amazing things that are going on all the time.\"\n",
            "\n",
            "While Nye is correct that \"everything that's going on all the time\" is making the world a better place, he misses the point. This is not\n",
            "====================\n",
            "The secret of life is in the rhythm of the universe. It's not a mystery. It's not a mystery to me. It's the nature of the universe. It's the beauty of the universe. It's the way the universe works. It's the way the universe is. It's the way the universe is going to work. It's the way the universe is. It's the way the universe is. It's the way the universe is. It's the way the universe is. It's the way\n",
            "====================\n",
            "The secret of life is in the universe.\n",
            "\n",
            "\n",
            "-\n",
            "\n",
            "The Red Devil\n",
            "\n",
            "It's the end of the world as we know it, and the only thing that can save us is a band of super-powered individuals known as the Red Devil.\n",
            "\n",
            "\n",
            "The Red Devil is a group of super-powered individuals who are seeking the secret of life and the only way they know how to do it is by taking on the roles of a variety of different super-powered individuals, each of which has their own\n",
            "====================\n",
            "The secret of life is in the mixing of the elements, and it is the mixing of the elements that makes life possible.\"\n",
            "\n",
            "But in the world of food science, the idea of a \"complex\" or \"complexity\" is almost entirely imaginary.\n",
            "\n",
            "As a scientist, I'm fascinated by the question of how life first began.\n",
            "\n",
            "It's the question that drives my work and the work of the scientists who work on it.\n",
            "\n",
            "My current research is exploring how microbes work in the first moments\n",
            "====================\n",
            "The secret of life is the journey of life, the search for the truth.\n",
            "\n",
            "4.4.2. The last thing you know\n",
            "\n",
            "There is nothing more important than the last thing you know.\n",
            "\n",
            "4.4.3. The little things that make all the difference\n",
            "\n",
            "The little things that make all the difference.\n",
            "\n",
            "4.4.4. The truth is the best teacher\n",
            "\n",
            "The truth is the best teacher.\n",
            "\n",
            "4.4.5. The truth is what\n",
            "====================\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ig-KVgkCDCKD"
      },
      "source": [
        "# Etcetera\n",
        "\n",
        "If the notebook has errors (e.g. GPU Sync Fail), force-kill the Colaboratory virtual machine and restart it with the command below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rIHiVP53FnsX"
      },
      "source": [
        "!kill -9 -1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wmTXWNUygS5E"
      },
      "source": [
        "# LICENSE\n",
        "\n",
        "MIT License\n",
        "\n",
        "Copyright (c) 2019 Max Woolf\n",
        "\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE."
      ]
    }
  ]
}