ML-Intro-Course/17_text_generation.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k9lyz2W7M7Se"
      },
      "source": [
        "# Text-generation with the GPT architecture\n",
        "\n",
        "<a href=\"https://colab.research.google.com/drive/1eriKZDafxRPYj42xV53-jhs-2YfbQzwh\" target=\"_blank\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\">\n",
        "</a>\n",
        "\n",
        "Return to the [castle](https://github.com/Nkluge-correa/TeenyTinyCastle).\n",
        "\n",
        "`Text generation` models are a type of machine learning model that can generate natural language text. These models have a wide range of applications, including `language translation`, `summarization`, and `text generation`. There are several different approaches to text generation, including `statistical models`, `rule-based system`, and, more recently, `neural network-based models`.\n",
        "\n",
        "Neural language models can be trained on large amounts of input text data and use this information to generate new text that is coherent and reflects the patterns and structures found in the training data. The quality of the generated text can vary depending on the _complexity of the model and the amount and quality of the training data_. Overall, text-generation models have the potential to revolutionize many industries by automating many tasks that involve the production and analysis of text.\n",
        "\n",
        "One of the biggest and most current advances in language modeling was made possible by the invention of the `transformer` architecture. A `transformer` model is a type of neural network architecture first described in the 2017 paper \"_[Attention Is All You Need](https://arxiv.org/abs/1706.03762)_\" by Vaswani et al. Transformers were originally used for machine translation, but this adaptable architecture is now used in a wide range of fields and problems.\n",
        "\n",
        "<img src=\"https://machinelearningmastery.com/wp-content/uploads/2021/08/attention_research_1.png\" alt=\"drawing\" height=\"450\"/>\n",
        "\n",
        "[Source](https://machinelearningmastery.com/the-transformer-model/).\n",
        "\n",
        "Before the `transformer`, most neural language models were based on recurrent neural networks ([`RNNs`](https://en.wikipedia.org/wiki/Recurrent_neural_network)). While effective, `RNNs` can be slow and difficult to parallelize, limiting their ability to scale. The `transformer` architecture was designed to overcome these limitations, efficiently processing long data sequences, while also being a very paralelizeble model.\n",
        "\n",
        "The `transformer` architecture has enabled the development of powerful language models such as [`BERT`](https://huggingface.co/docs/transformers/model_doc/bert) and [`GPT-3`](https://arxiv.org/abs/2005.14165), which have achieved state-of-the-art results on a wide range of natural language processing tasks.\n",
        "\n",
        "To learn more about the original `transformer` architecture, go to our [`sequence-to-sequence` machine translation](https://github.com/Nkluge-correa/TeenyTinyCastle/blob/48d415094d30e0e5bc8dde32715bb57428a87d7d/ML-Intro-Course/16_sequence_to_sequence.ipynb) notebook.\n",
        "\n",
        "In this notebook, we will implement a `decoder-only transformer`. `Decoder-only transformers`, such as the `GPT` (_Generative Pre-training Transformer_) series, are models that consist only of the `decoder` portion of the original transformer, which is then trained on a causal language modeling task (i.e., based on $n$ tokens, predict $n+1$).\n",
        "\n",
        "To start building our model, we first need _good-quality text_. Good-quality text is important for the training of language models for several reasons. For example, good-quality text is more likely to reflect the real-world patterns and structures of language, which is important for the model to learn. A model trained on poorly written or grammatically incorrect text may struggle to generate correct and coherent output. Also, a model trained on large amounts of high-quality text may be able to learn language patterns more quickly and with fewer resources than a model trained on low-quality text.\n",
        "\n",
        "For this tutorial, we will create a text dataset using the articles from the [`Stanford Encyclopedia of Philosophy`](https://plato.stanford.edu/). Details on the created text corpus can be found on this [dataset card](https://huggingface.co/datasets/AiresPucrs/stanford-encyclopedia-philosophy), and the\n",
        "full dataset can be downloaded from the Hub. 🤗\n",
        "\n",
        "## Getting a Text Corpus\n",
        "\n",
        "Web scraping is the process of extracting information from websites. This can be done using a variety of programming languages and tools, such as Python and its libraries for web scraping, such as `BeautifulSoup` and `Scrapy`.\n",
        "\n",
        "Web scraping can be useful but can be done in an unethical way and even illegally, so it is important to be aware of the website's terms of use before scraping. Many websites have terms of service that prohibit scraping, so it's important to review the terms of service of a website before scraping it.\n",
        "\n",
        "If we check the [`robots.txt`](https://plato.stanford.edu/robots.txt) file of the SEP, we see what we are allowed to do, and the following code will only scrape permissible content of the SEP.\n",
        "\n",
        "In this example, we will use `BeautifulSoup` to get our text data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DrgPSjDBM7Si",
        "outputId": "069b0a9e-d2ba-488c-966f-c0fbd3a03b01"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m584.5/584.5 kB\u001b[0m \u001b[31m11.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m588.3/588.3 MB\u001b[0m \u001b[31m1.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.8/5.8 MB\u001b[0m \u001b[31m61.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m71.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m51.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.0/6.0 MB\u001b[0m \u001b[31m63.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m439.2/439.2 kB\u001b[0m \u001b[31m32.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m950.8/950.8 kB\u001b[0m \u001b[31m35.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.9/4.9 MB\u001b[0m \u001b[31m43.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m781.3/781.3 kB\u001b[0m \u001b[31m33.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "tensorflow-datasets 4.9.3 requires protobuf>=3.20, but you have protobuf 3.19.6 which is incompatible.\n",
            "tensorflow-metadata 1.14.0 requires protobuf<4.21,>=3.20.3, but you have protobuf 3.19.6 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0m"
          ]
        }
      ],
      "source": [
        "!pip install keras-nlp tensorflow==2.11 tensorflow-text==2.11 --upgrade -q\n",
        "\n",
        "import re\n",
        "import requests\n",
        "import pandas as pd\n",
        "from bs4 import BeautifulSoup\n",
        "\n",
        "# Set the URL to scrape\n",
        "url = \"https://plato.stanford.edu/contents.html\"\n",
        "\n",
        "# Send a request to the URL and retrieve the webpage\n",
        "page = requests.get(url)\n",
        "\n",
        "# Parse the webpage with BeautifulSoup\n",
        "soup = BeautifulSoup(page.text, \"html.parser\")\n",
        "\n",
        "# parse the href addresses and anchors of the html page\n",
        "definitions = soup.find_all(\"a\")\n",
        "quoted = re.compile('\"[^\"]*\"')\n",
        "\n",
        "entries = []\n",
        "for definition in definitions:\n",
        "    definition = str(definition)\n",
        "\n",
        "    # get all the addresses of the links in the 'contents.html' page\n",
        "    for value in quoted.findall(definition):\n",
        "\n",
        "        # get all pages, that have philosophical text\n",
        "        if value[1:-1].startswith(\"entries\"):\n",
        "            entries.append(value[1:-1])\n",
        "\n",
        "# list of all the texts\n",
        "paragraphs = []\n",
        "\n",
        "# list of all the source pages\n",
        "source = []"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ta8F8NCyM7Sj"
      },
      "source": [
        "Now that we have all the entries, we can begin to extract the text from them (note that the entries are allowed according to the [`robots.txt`](https://plato.stanford.edu/robots.txt)).\n",
        "\n",
        "⚠️ ALWAYS BE SURE YOU ARE ALLOWED TO SCRAPE. ⚠️"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "u6Eute3uM7Sj"
      },
      "outputs": [],
      "source": [
        "print(f'Number of content pages: {len(entries)}')\n",
        "# loop over all the pages that have philosophical text\n",
        "for i, entri in enumerate(entries):\n",
        "    url = f\"https://plato.stanford.edu/{entri}\"\n",
        "\n",
        "    # Send a request to the URL and retrieve the webpage\n",
        "    page = requests.get(url)\n",
        "\n",
        "    # Parse the webpage with BeautifulSoup\n",
        "    soup = BeautifulSoup(page.text, \"html.parser\")\n",
        "\n",
        "    # Get all the <p> tags from the parsed html\n",
        "    texts = soup.find_all(\"p\")\n",
        "\n",
        "    # Loop over all the <p> tags from the parsed html\n",
        "    for text in texts:\n",
        "\n",
        "        # remove the html tags from the string\n",
        "        clean_string = re.sub(r'<[^>]*>', '', str(text))\n",
        "\n",
        "        # replace the '\\n' with \" \"\n",
        "        clean_string = clean_string.replace(\"\\n\", \" \")\n",
        "\n",
        "        # append the source and text elements\n",
        "        paragraphs.append(clean_string)\n",
        "        source.append(url)\n",
        "\n",
        "    print(f\"Page {i + 1}. Scrapped page '{url}'!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KWmyjJSBM7Sk"
      },
      "source": [
        "Now, let us create a dataframe with all of our text corpus. First, we will create a \" SEP \" folder in the \"/content\" directory to store our future files."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bGue5drmM7Sk"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "os.mkdir('./SEP')\n",
        "\n",
        "# create a pandas data frame with the data\n",
        "df = pd.DataFrame({'text': paragraphs, 'metadata': source})\n",
        "\n",
        "# drop duplicate text\n",
        "df = df.drop_duplicates()\n",
        "\n",
        "# Clean the URL to get the \"Category\" of the page\n",
        "def clean_url(string):\n",
        "    return string.split('entries/')[1][:-1]\n",
        "\n",
        "# Apply the function to the \"URL\" column\n",
        "df['category'] = df['metadata'].apply(clean_url)\n",
        "\n",
        "# save as a csv file\n",
        "df.to_parquet('SEP/stanford-encyclopedia-philosophy.parquet', compression='gzip')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bUwa-MtdM7Sk"
      },
      "source": [
        "And here is our text corpus. You can download it directly form the Hub using the following commands:\n",
        "\n",
        "```python\n",
        "from datasets import load_dataset\n",
        "\n",
        "dataset = load_dataset(\"AiresPucrs/stanford-encyclopedia-philosophy\")\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "id": "7dLiX_n5M7Sl",
        "outputId": "ef9c154e-5fc9-411b-8e8f-7956a59a5f75"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-12ff7085-e612-4a17-8a6d-b8f152dc5478\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>metadata</th>\n",
              "      <th>category</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>In the philosophical literature, the term “ab...</td>\n",
              "      <td>https://plato.stanford.edu/entries/abduction/</td>\n",
              "      <td>abduction</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>This entry is exclusively concerned with abdu...</td>\n",
              "      <td>https://plato.stanford.edu/entries/abduction/</td>\n",
              "      <td>abduction</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>See also the entry on  scientific discovery, ...</td>\n",
              "      <td>https://plato.stanford.edu/entries/abduction/</td>\n",
              "      <td>abduction</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Most philosophers agree that abduction (in th...</td>\n",
              "      <td>https://plato.stanford.edu/entries/abduction/</td>\n",
              "      <td>abduction</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>You happen to know that Tim and Harry have re...</td>\n",
              "      <td>https://plato.stanford.edu/entries/abduction/</td>\n",
              "      <td>abduction</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>256791</th>\n",
              "      <td>Many thanks to David Chalmers and to Bill Fis...</td>\n",
              "      <td>https://plato.stanford.edu/entries/zombies/</td>\n",
              "      <td>zombies</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>256792</th>\n",
              "      <td>Copyright © 2023 by   Robert Kirk &amp;lt;Robert....</td>\n",
              "      <td>https://plato.stanford.edu/entries/zombies/</td>\n",
              "      <td>zombies</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>256793</th>\n",
              "      <td>View this site from another server:</td>\n",
              "      <td>https://plato.stanford.edu/entries/zombies/</td>\n",
              "      <td>zombies</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>256794</th>\n",
              "      <td>The Stanford Encyclopedia of Philosophy is cop...</td>\n",
              "      <td>https://plato.stanford.edu/entries/zombies/</td>\n",
              "      <td>zombies</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>256795</th>\n",
              "      <td>Library of Congress Catalog Data: ISSN 1095-5054</td>\n",
              "      <td>https://plato.stanford.edu/entries/zombies/</td>\n",
              "      <td>zombies</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>188335 rows × 3 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-12ff7085-e612-4a17-8a6d-b8f152dc5478')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-12ff7085-e612-4a17-8a6d-b8f152dc5478 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-12ff7085-e612-4a17-8a6d-b8f152dc5478');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-fc909fd8-e61d-4c6b-a4d8-9aa9b9e783e5\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-fc909fd8-e61d-4c6b-a4d8-9aa9b9e783e5')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-fc909fd8-e61d-4c6b-a4d8-9aa9b9e783e5 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                                     text  \\\n",
              "0        In the philosophical literature, the term “ab...   \n",
              "1        This entry is exclusively concerned with abdu...   \n",
              "2        See also the entry on  scientific discovery, ...   \n",
              "3        Most philosophers agree that abduction (in th...   \n",
              "4        You happen to know that Tim and Harry have re...   \n",
              "...                                                   ...   \n",
              "256791   Many thanks to David Chalmers and to Bill Fis...   \n",
              "256792   Copyright © 2023 by   Robert Kirk &lt;Robert....   \n",
              "256793                View this site from another server:   \n",
              "256794  The Stanford Encyclopedia of Philosophy is cop...   \n",
              "256795   Library of Congress Catalog Data: ISSN 1095-5054   \n",
              "\n",
              "                                             metadata   category  \n",
              "0       https://plato.stanford.edu/entries/abduction/  abduction  \n",
              "1       https://plato.stanford.edu/entries/abduction/  abduction  \n",
              "2       https://plato.stanford.edu/entries/abduction/  abduction  \n",
              "3       https://plato.stanford.edu/entries/abduction/  abduction  \n",
              "4       https://plato.stanford.edu/entries/abduction/  abduction  \n",
              "...                                               ...        ...  \n",
              "256791    https://plato.stanford.edu/entries/zombies/    zombies  \n",
              "256792    https://plato.stanford.edu/entries/zombies/    zombies  \n",
              "256793    https://plato.stanford.edu/entries/zombies/    zombies  \n",
              "256794    https://plato.stanford.edu/entries/zombies/    zombies  \n",
              "256795    https://plato.stanford.edu/entries/zombies/    zombies  \n",
              "\n",
              "[188335 rows x 3 columns]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "\"\"\"\n",
        "!pip install datasets -q\n",
        "from datasets import load_dataset\n",
        "\n",
        "dataset = load_dataset(\"AiresPucrs/stanford-encyclopedia-philosophy\", split='train')\n",
        "df = dataset.to_pandas()\n",
        "\"\"\"\n",
        "\n",
        "df = pd.read_parquet('SEP/stanford-encyclopedia-philosophy.parquet')\n",
        "display(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R-fu-EtHM7Sl"
      },
      "source": [
        "The loop below will create a dataset folder. The folder will contain a folder for each topic in the SEP dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D34nOPmF3CQp",
        "outputId": "b0caaa2f-fbf3-4c7c-d26b-d97e2e4eea93"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Dataset Folder Created!\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# Define the base directory to create subdirectories\n",
        "base_directory = \"SEP/dataset\"\n",
        "\n",
        "# Check if directory exists or create if it does not exist\n",
        "if not os.path.exists(base_directory):\n",
        "    os.makedirs(base_directory)\n",
        "\n",
        "# Iterate through unique categories in the 'df' DataFrame\n",
        "for category in df.category.unique():\n",
        "    category_directory = os.path.join(base_directory, category)\n",
        "    os.mkdir(category_directory)\n",
        "\n",
        "    dff = df[df['category'] == category]\n",
        "\n",
        "    for i, sample in enumerate(list(dff.text)):\n",
        "        with open(os.path.join(category_directory, f'{i}.txt'), 'w', encoding='utf-8') as fp:\n",
        "            fp.write(sample)\n",
        "\n",
        "print('Dataset Folder Created!')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VyX-WLByM7Sm"
      },
      "source": [
        "Using the entire SEP Corpus to train a language model (if you don't have access to powerful GPUs) can take a long time. As a result, for demonstration purposes, we will train our language model on a subset of our corpus.\n",
        "\n",
        "Our mini-dataset contains only `aesthetics-18th-british`, `aesthetics-18th-french`, `aesthetics-18th-german`, and `aesthetics-19th-romantic`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kOFrCHaMM7Sm",
        "outputId": "aee6f863-1cc3-45b9-fc40-952e6c9da930"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Found 659 files\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "\n",
        "filenames = []\n",
        "\n",
        "directories = [\"SEP/dataset/aesthetics-18th-british\",\n",
        "                \"SEP/dataset/aesthetics-18th-french\",\n",
        "                \"SEP/dataset/aesthetics-18th-german\",\n",
        "                \"SEP/dataset/aesthetics-19th-romantic\"]\n",
        "\n",
        "for directory in directories:\n",
        "    for folder in os.listdir(directory):\n",
        "        filenames.append(os.path.join(directory, folder))\n",
        "\n",
        "print(f\"Found {len(filenames)} files\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dZW-3mvwM7Sm"
      },
      "source": [
        "All the found files are `txt` with some text about the topics selected above.\n",
        "\n",
        "Now, let us shuffle the order of our samples and create a dataset using the `tf.data.TextLineDataset`, which loads text from text files and creates a dataset where each line of the files becomes an element of the dataset. We also selected a small `batch_size` to avoid OOM (OUT-OF-MEMORY) problems (in case you are using a GPU)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dmgpVwfMM7Sn"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "import tensorflow as tf\n",
        "\n",
        "batch_size = 16\n",
        "\n",
        "random.shuffle(filenames)\n",
        "\n",
        "text_ds = tf.data.TextLineDataset(filenames)\n",
        "text_ds = text_ds.shuffle(buffer_size=256)\n",
        "text_ds = text_ds.batch(batch_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HR9W90h8M7Sn"
      },
      "source": [
        "Because we are using a small dataset, we will have a small vocabulary (in this example, we end up with a vocabulary with 8600 unique tokens). This is a very small vocabulary compared to famous large language models (Pythia, Llama, Claude, GPT-4, etc.).\n",
        "\n",
        "> **Note:** Most Large Language Models have tokenizers trained via [Byte-pair encoding](https://en.wikipedia.org/wiki/Byte_pair_encoding). Check out our other [repositories to learn how](https://github.com/Nkluge-correa/Aira) to create such tokenizers.\n",
        "\n",
        "We create our vocabulary using the `tf.keras.layers.TextVectorization`, passing a `custom_standardization` function to lower strings and parse punctuations. Then we adapt the `TextVectorization` layer to our dataset and get our vocabulary out of it. We save the vocabulary in a `txt` file for later use. From this vocabulary, we can detokenize the sequences produced by our language model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GdMTBANPM7Sn",
        "outputId": "f1a463e8-480b-4fe0-a64f-8f336aa93b34"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Found 8600 unique tokens!\n"
          ]
        }
      ],
      "source": [
        "import string\n",
        "\n",
        "# Will cut sequences with more than 500 tokens\n",
        "sequence_length = 500\n",
        "\n",
        "# Maximum vocabulary size\n",
        "vocab_size = 8700\n",
        "\n",
        "# Lower all strings and parse punctuation\n",
        "def custom_standardization(input_string):\n",
        "    lowercased = tf.strings.lower(input_string)\n",
        "    return tf.strings.regex_replace(lowercased, f\"([{string.punctuation}])\", r\" \\1\")\n",
        "\n",
        "from keras.layers import TextVectorization\n",
        "\n",
        "# Create a vectorization layer and adapt it to the text\n",
        "vectorize_layer = TextVectorization(\n",
        "    standardize=custom_standardization,\n",
        "    max_tokens=vocab_size - 1,\n",
        "    output_mode=\"int\",\n",
        "    output_sequence_length=sequence_length + 1,\n",
        ")\n",
        "\n",
        "# Fit the TextVectorization layer to the dataset\n",
        "vectorize_layer.adapt(text_ds)\n",
        "\n",
        "# Get words back from token indices\n",
        "vocab = vectorize_layer.get_vocabulary()\n",
        "\n",
        "print(f'Found {len(vocab)} unique tokens!')\n",
        "\n",
        "# Save the vocabulary as a text file\n",
        "with open(f'vocabulary.txt', 'w', encoding='utf-8') as fp:\n",
        "    for word in vocab:\n",
        "        fp.write(\"%s\\n\" % word)\n",
        "    fp.close()\n",
        "\n",
        "# Index to detokenize tokens\n",
        "vocab_index = {}\n",
        "for index, word in enumerate(vocab):\n",
        "    vocab_index[word] = index"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zcAqgVWdM7Sn"
      },
      "source": [
        "To prepare our dataset, we shift word sequences by $1$ position so that the target for the position $i$ is a word at position $i+1$. The model will use all words up to position $i$ to predict the next. Thus, our language model is forced to make predictions in a causal way.\n",
        "\n",
        "> Note: This is also called causal modeling, i.e., only past tokens can be used to infer the next in the sequence."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lOb4-UOeM7Sn",
        "outputId": "ab6d5ecf-b005-41e7-82ba-5bb7a95d1493"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Dataset ready!\n"
          ]
        }
      ],
      "source": [
        "def prepare_lm_dataset(text):\n",
        "    \"\"\"\n",
        "    Prepares a language modeling dataset by tokenizing the input text.\n",
        "\n",
        "    Args:\n",
        "        text (str or tf.Tensor): The input text to be tokenized.\n",
        "\n",
        "    Returns:\n",
        "        tuple: A tuple containing two elements: the input sequences (x) and the target sequences (y).\n",
        "    \"\"\"\n",
        "    text = tf.expand_dims(text, -1)\n",
        "    tokenized_sentences = vectorize_layer(text)\n",
        "    x = tokenized_sentences[:, :-1]\n",
        "    y = tokenized_sentences[:, 1:]\n",
        "    return x, y\n",
        "\n",
        "\n",
        "text_ds = text_ds.map(prepare_lm_dataset)\n",
        "text_ds = text_ds.prefetch(tf.data.AUTOTUNE)\n",
        "\n",
        "print('Dataset ready!')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q1GdibV8M7Sn"
      },
      "source": [
        "In this notebook, we will create a `decoder-only transformer` based on the `GPT` architecture. Our model will use some of the same components we used in our [`sequence-to-sequence`](https://github.com/Nkluge-correa/TeenyTinyCastle/blob/master/ML-Intro-Course/16_sequence_to_sequence.ipynb), like the `PositionalEmbedding` layer (a way to inject temporal information into our model), and the `TransformerDecoder` block.\n",
        "\n",
        "The combination of these simple blocks gives rise to our `mini-GPT`. If you wish to create a more robust model and give it more data, you could stack more `TransformerDecoder` blocks and create residual connections among them."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7Oyo6U7nM7So"
      },
      "outputs": [],
      "source": [
        "from tensorflow import keras\n",
        "from keras import layers\n",
        "\n",
        "class PositionalEmbedding(layers.Layer):\n",
        "    \"\"\"\n",
        "    This class creates a positional embedding layer that adds positional information to the input embeddings.\n",
        "    It takes in the sequence length, input dimension, and output dimension as arguments.\n",
        "    The call method takes in the inputs and returns the sum of the token embeddings and positional embeddings.\n",
        "    The compute_mask method returns a boolean mask tensor based on the inputs.\n",
        "    The get_config method returns the configuration of the layer.\n",
        "    \"\"\"\n",
        "    def __init__(self, sequence_length, input_dim, output_dim, **kwargs):\n",
        "        super().__init__(**kwargs)\n",
        "        self.token_embeddings = layers.Embedding(\n",
        "            input_dim=input_dim, output_dim=output_dim)\n",
        "        self.position_embeddings = layers.Embedding(\n",
        "            input_dim=sequence_length, output_dim=output_dim)\n",
        "        self.sequence_length = sequence_length\n",
        "        self.input_dim = input_dim\n",
        "        self.output_dim = output_dim\n",
        "\n",
        "    def call(self, inputs):\n",
        "        length = tf.shape(inputs)[-1]\n",
        "        positions = tf.range(start=0, limit=length, delta=1)\n",
        "        embedded_tokens = self.token_embeddings(inputs)\n",
        "        embedded_positions = self.position_embeddings(positions)\n",
        "        return embedded_tokens + embedded_positions\n",
        "\n",
        "    def compute_mask(self, inputs, mask=None):\n",
        "        return tf.math.not_equal(inputs, 0)\n",
        "\n",
        "    def get_config(self):\n",
        "        config = super(PositionalEmbedding, self).get_config()\n",
        "        config.update({\n",
        "            \"output_dim\": self.output_dim,\n",
        "            \"sequence_length\": self.sequence_length,\n",
        "            \"input_dim\": self.input_dim,\n",
        "        })\n",
        "        return config\n",
        "\n",
        "\n",
        "class TransformerDecoder(layers.Layer):\n",
        "    \"\"\"\n",
        "    TransformerDecoder is a class that implements the decoder block of the Transformer model.\n",
        "    It takes in the input sequence, encoder outputs and an optional mask and returns the decoder output.\n",
        "\n",
        "    Args:\n",
        "        embed_dim (int): The dimensionality of the embedding space.\n",
        "        dense_dim (int): The dimensionality of the dense layer.\n",
        "        num_heads (int): The number of attention heads.\n",
        "\n",
        "    Returns:\n",
        "        The decoder output.\n",
        "    \"\"\"\n",
        "    def __init__(self, embed_dim, dense_dim, num_heads, **kwargs):\n",
        "        super().__init__(**kwargs)\n",
        "        self.embed_dim = embed_dim\n",
        "        self.dense_dim = dense_dim\n",
        "        self.num_heads = num_heads\n",
        "        self.attention_1 = layers.MultiHeadAttention(\n",
        "          num_heads=num_heads, key_dim=embed_dim)\n",
        "        self.attention_2 = layers.MultiHeadAttention(\n",
        "          num_heads=num_heads, key_dim=embed_dim)\n",
        "        self.dense_proj = keras.Sequential(\n",
        "            [layers.Dense(dense_dim, activation=\"relu\"),\n",
        "             layers.Dense(embed_dim),]\n",
        "        )\n",
        "        self.layernorm_1 = layers.LayerNormalization(epsilon=1e-6)\n",
        "        self.layernorm_2 = layers.LayerNormalization(epsilon=1e-6)\n",
        "        self.layernorm_3 = layers.LayerNormalization(epsilon=1e-6)\n",
        "        self.dropout1 = layers.Dropout(0.1)\n",
        "        self.dropout2 = layers.Dropout(0.1)\n",
        "        self.dropout3 = layers.Dropout(0.1)\n",
        "        self.supports_masking = True\n",
        "\n",
        "    def get_config(self):\n",
        "        config = super(TransformerDecoder, self).get_config()\n",
        "        config.update({\n",
        "            \"embed_dim\": self.embed_dim,\n",
        "            \"num_heads\": self.num_heads,\n",
        "            \"dense_dim\": self.dense_dim,\n",
        "        })\n",
        "        return config\n",
        "\n",
        "    def get_causal_attention_mask(self, inputs):\n",
        "        input_shape = tf.shape(inputs)\n",
        "        batch_size, sequence_length = input_shape[0], input_shape[1]\n",
        "        i = tf.range(sequence_length)[:, tf.newaxis]\n",
        "        j = tf.range(sequence_length)\n",
        "        mask = tf.cast(i >= j, dtype=\"int32\")\n",
        "        mask = tf.reshape(mask, (1, input_shape[1], input_shape[1]))\n",
        "        mult = tf.concat(\n",
        "            [tf.expand_dims(batch_size, -1),\n",
        "             tf.constant([1, 1], dtype=tf.int32)], axis=0)\n",
        "        return tf.tile(mask, mult)\n",
        "\n",
        "    def call(self, inputs, encoder_outputs, mask=None):\n",
        "        causal_mask = self.get_causal_attention_mask(inputs)\n",
        "        if mask is not None:\n",
        "            padding_mask = tf.cast(\n",
        "                mask[:, tf.newaxis, :], dtype=\"int32\")\n",
        "            padding_mask = tf.minimum(padding_mask, causal_mask)\n",
        "        attention_output_1 = self.attention_1(inputs, inputs, attention_mask=causal_mask)\n",
        "        attention_output_1 = self.dropout1(attention_output_1)\n",
        "        attention_output_1 = self.layernorm_1(inputs + attention_output_1)\n",
        "        attention_output_2 = self.attention_2(query=attention_output_1,\n",
        "                                                value=encoder_outputs,\n",
        "                                                key=encoder_outputs,\n",
        "                                                attention_mask=padding_mask)\n",
        "        attention_output_2 = self.dropout2(attention_output_2)\n",
        "        attention_output_2 = self.layernorm_2(attention_output_1 + attention_output_2)\n",
        "        proj_output = self.dense_proj(attention_output_2)\n",
        "        proj_output = self.dropout3(proj_output)\n",
        "        return self.layernorm_3(attention_output_2 + proj_output)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zYP0Nhl2M7So"
      },
      "source": [
        "We end up with a 6.6M parameters model, which is small for a modern-day, state-of-the-art language model. Keeping things simple, this model has only two attention heads, one decoder block, a vocabulary (_targets of the final dense network_) of 8600 tokens, embedding dimensions of 256, and a dense-latent-dimension of 2048 ($Vocab_{8600}, Embedd_{256}, D_{2048}$)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1ye602wWM7So",
        "outputId": "9ee7ae2d-568c-4143-9b38-40e8890b0f64"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend\n",
            "Model: \"model\"\n",
            "__________________________________________________________________________________________________\n",
            " Layer (type)                   Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            " input_1 (InputLayer)           [(None, 500)]        0           []                               \n",
            "                                                                                                  \n",
            " positional_embedding (Position  (None, 500, 256)    2355200     ['input_1[0][0]']                \n",
            " alEmbedding)                                                                                     \n",
            "                                                                                                  \n",
            " transformer_decoder (Transform  (None, 500, 256)    2104576     ['positional_embedding[0][0]',   \n",
            " erDecoder)                                                       'positional_embedding[0][0]']   \n",
            "                                                                                                  \n",
            " dense_2 (Dense)                (None, 500, 8600)    2210200     ['transformer_decoder[0][0]']    \n",
            "                                                                                                  \n",
            "==================================================================================================\n",
            "Total params: 6,669,976\n",
            "Trainable params: 6,669,976\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "from tensorflow import keras\n",
        "import keras_nlp\n",
        "\n",
        "sequence_length = 500\n",
        "embed_dim = 256\n",
        "latent_dim = 2048\n",
        "num_heads = 2\n",
        "\n",
        "inputs = keras.Input(shape=(sequence_length,), dtype=tf.int32)\n",
        "x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(inputs)\n",
        "x = TransformerDecoder(embed_dim, latent_dim, num_heads)(x, x)\n",
        "outputs = layers.Dense(len(vocab), activation=\"softmax\")(x)\n",
        "model = keras.Model(inputs, outputs=outputs)\n",
        "\n",
        "perplexity = keras_nlp.metrics.Perplexity(name=\"perplexity\")\n",
        "\n",
        "model.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "                optimizer=\"adam\", metrics=perplexity)\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LtcSKmjqM7So"
      },
      "source": [
        "Now, the last thing we need is to train our model. We are using `keras.callbacks` to save the best model in 30 epochs (for such a small dataset, we don't need more). The callback will monitor the `perplexity` of the model, which is the metric we use to evaluate performance.\n",
        "\n",
        "`Perplexity` is a metric that measures how well a probabilistic model (such as a language model) predicts a sample. `Perplexity` is defined as 2 to the power of the cross-entropy, which measures the difference between the sample's predicted and true probability distribution.\n",
        "\n",
        "The formula for `perplexity` is simply the inverse of the exponentiation of the cross-entropy:\n",
        "\n",
        "$$Perplexity = 2^{-cross\\;entropy}$$\n",
        "\n",
        "$$Cross\\;Entropy = -\\frac{1}{n}\\sum_{i=1}^{n}\\log P(w_i)$$\n",
        "\n",
        "where\n",
        "\n",
        "- $n$ is the length of the sample.\n",
        "- $P(w_i)$ is the predicted probability of the $i$-th word in the sample according to the language model (_sum is taken over all words in the sample_).\n",
        "\n",
        "`Perplexity` reveals how well a model predicts the next word in a sequence and, thus, how well it knows the language. In general, it is used as an evaluation metric in language modeling tasks."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N2g2RLGZM7So",
        "outputId": "d8300645-4e73-4583-ed20-ae2203698e57"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Version:  2.11.0\n",
            "Eager mode:  True\n",
            "GPU is available\n",
            "Epoch 1/30\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/keras/backend.py:5585: UserWarning: \"`sparse_categorical_crossentropy` received `from_logits=True`, but the `output` argument was produced by a Softmax activation and thus does not represent logits. Was this intended?\n",
            "  output, from_logits = _get_logits(\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "42/42 [==============================] - 22s 260ms/step - loss: 6.9390 - perplexity: 1031.6973\n",
            "Epoch 2/30\n",
            "42/42 [==============================] - 11s 264ms/step - loss: 6.0547 - perplexity: 426.0944\n",
            "Epoch 3/30\n",
            "42/42 [==============================] - 9s 207ms/step - loss: 5.2796 - perplexity: 196.2975\n",
            "Epoch 4/30\n",
            "42/42 [==============================] - 10s 243ms/step - loss: 4.7148 - perplexity: 111.5857\n",
            "Epoch 5/30\n",
            "42/42 [==============================] - 8s 193ms/step - loss: 4.2863 - perplexity: 72.6973\n",
            "Epoch 6/30\n",
            "42/42 [==============================] - 10s 236ms/step - loss: 3.8747 - perplexity: 48.1692\n",
            "Epoch 7/30\n",
            "42/42 [==============================] - 8s 190ms/step - loss: 3.4763 - perplexity: 32.3396\n",
            "Epoch 8/30\n",
            "42/42 [==============================] - 10s 236ms/step - loss: 3.0898 - perplexity: 21.9717\n",
            "Epoch 9/30\n",
            "42/42 [==============================] - 7s 163ms/step - loss: 2.7191 - perplexity: 15.1671\n",
            "Epoch 10/30\n",
            "42/42 [==============================] - 9s 206ms/step - loss: 2.4019 - perplexity: 11.0446\n",
            "Epoch 11/30\n",
            "42/42 [==============================] - 7s 156ms/step - loss: 2.0383 - perplexity: 7.6778\n",
            "Epoch 12/30\n",
            "42/42 [==============================] - 8s 195ms/step - loss: 1.7149 - perplexity: 5.5563\n",
            "Epoch 13/30\n",
            "42/42 [==============================] - 8s 185ms/step - loss: 1.4432 - perplexity: 4.2342\n",
            "Epoch 14/30\n",
            "42/42 [==============================] - 8s 182ms/step - loss: 1.2151 - perplexity: 3.3706\n",
            "Epoch 15/30\n",
            "42/42 [==============================] - 6s 151ms/step - loss: 0.9650 - perplexity: 2.6247\n",
            "Epoch 16/30\n",
            "42/42 [==============================] - 8s 190ms/step - loss: 0.7553 - perplexity: 2.1282\n",
            "Epoch 17/30\n",
            "42/42 [==============================] - 6s 151ms/step - loss: 0.5903 - perplexity: 1.8045\n",
            "Epoch 18/30\n",
            "42/42 [==============================] - 8s 188ms/step - loss: 0.4612 - perplexity: 1.5860\n",
            "Epoch 19/30\n",
            "42/42 [==============================] - 6s 151ms/step - loss: 0.3486 - perplexity: 1.4171\n",
            "Epoch 20/30\n",
            "42/42 [==============================] - 7s 173ms/step - loss: 0.2707 - perplexity: 1.3108\n",
            "Epoch 21/30\n",
            "42/42 [==============================] - 6s 144ms/step - loss: 0.2126 - perplexity: 1.2368\n",
            "Epoch 22/30\n",
            "42/42 [==============================] - 7s 157ms/step - loss: 0.1688 - perplexity: 1.1838\n",
            "Epoch 23/30\n",
            "42/42 [==============================] - 7s 162ms/step - loss: 0.1324 - perplexity: 1.1416\n",
            "Epoch 24/30\n",
            "42/42 [==============================] - 6s 144ms/step - loss: 0.1048 - perplexity: 1.1105\n",
            "Epoch 25/30\n",
            "42/42 [==============================] - 7s 164ms/step - loss: 0.0844 - perplexity: 1.0880\n",
            "Epoch 26/30\n",
            "42/42 [==============================] - 6s 132ms/step - loss: 0.0702 - perplexity: 1.0727\n",
            "Epoch 27/30\n",
            "42/42 [==============================] - 6s 143ms/step - loss: 0.0598 - perplexity: 1.0616\n",
            "Epoch 28/30\n",
            "42/42 [==============================] - 6s 145ms/step - loss: 0.0555 - perplexity: 1.0571\n",
            "Epoch 29/30\n",
            "42/42 [==============================] - 6s 148ms/step - loss: 0.0520 - perplexity: 1.0534\n",
            "Epoch 30/30\n",
            "42/42 [==============================] - 6s 145ms/step - loss: 0.0465 - perplexity: 1.0476\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7fddf21ac130>"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "print(\"Version: \", tf.__version__)\n",
        "print(\"Eager mode: \", tf.executing_eagerly())\n",
        "print(\"GPU is\", \"available\" if tf.config.list_physical_devices('GPU') else \"NOT AVAILABLE\")\n",
        "\n",
        "callbacks = [keras.callbacks.ModelCheckpoint(\"text_gen.h5\",\n",
        "                                                save_best_only=True,\n",
        "                                                monitor=\"perplexity\",\n",
        "                                                patience=3,\n",
        "                                                restore_best_weights=True)]\n",
        "\n",
        "model.fit(text_ds, verbose=1, epochs=30, callbacks=callbacks)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QcNeqKmpM7So"
      },
      "source": [
        "Having a trained language model on texts related to aesthetic philosophy, we can now use it to generate some text.\n",
        "\n",
        "To load models created using subclass functions from the Keras API, you need to pass your classes as custom objects (after building the classes ...):\n",
        "\n",
        "```python\n",
        "\n",
        "model = keras.models.load_model(\"your_model.keras\",\n",
        "    custom_objects={\"TransformerDecoder\": TransformerDecoder,\n",
        "        \"PositionalEmbedding\": PositionalEmbedding,\n",
        "        \"perplexity\": keras_nlp.metrics.Perplexity})\n",
        "\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "svJdBqFmM7Sp"
      },
      "outputs": [],
      "source": [
        "from keras.layers import TextVectorization\n",
        "from tensorflow import keras\n",
        "import tensorflow as tf\n",
        "from keras import layers\n",
        "import numpy as np\n",
        "import keras_nlp\n",
        "\n",
        "TextGenerator = keras.models.load_model(\"text_gen.h5\",\n",
        "    custom_objects={\"PositionalEmbedding\": PositionalEmbedding,\n",
        "        \"TransformerDecoder\": TransformerDecoder,\n",
        "        \"perplexity\": keras_nlp.metrics.Perplexity})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hCfnpn7KM7Sp"
      },
      "source": [
        "We also load our vocabulary to create a detokenization function for the outputs of our model. With this vocabulary, we create a TextVectorization layer to tokenize our prompt inputs, passing our vocabulary so we don't have to adapt it again."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YMiazr-XM7Sp"
      },
      "outputs": [],
      "source": [
        "with open('vocabulary.txt', encoding='utf-8') as fp:\n",
        "    vocab = [line.strip() for line in fp]\n",
        "    fp.close()\n",
        "\n",
        "vocab_index = {}\n",
        "for index, word in enumerate(vocab):\n",
        "    vocab_index[word] = index\n",
        "\n",
        "text_vectorization = TextVectorization(max_tokens=len(vocab),\n",
        "                                        output_mode=\"int\",\n",
        "                                        output_sequence_length=500,\n",
        "                                        vocabulary=vocab)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ljjDmCCF8nF3"
      },
      "source": [
        "With all of these parts ready, we can create functions to:\n",
        "\n",
        "1. Sample from our model (thus producing less deterministic outputs).\n",
        "2. Detokenize our samples.\n",
        "3. Generate some text! 🎉"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        },
        "id": "PlDMpR2QM7Sp",
        "outputId": "e0120d60-f44a-49a8-ddde-3f80539356ee"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'debates about artistic matters were greatly influenced by the new spaces and means of communication that emerged in the seventeenth and eighteenth centuries . critics expressed their judgments in published treatises and in periodicals such as le mercure galant ; philosophical ideas were also developed in oral conversations between members of the newly founded royal were greatly influenced by the new spaces and means of communication that emerged in the seventeenth and eighteenth centuries . critics expressed their judgments in published treatises and in periodicals such as le mercure galant ; philosophical ideas were also developed in oral conversations between members of the newly founded royal'"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "\n",
        "def sample_from(logits, chose_from):\n",
        "    \"\"\"\n",
        "    This function allows us to sample from the\n",
        "    probability distribution output of our\n",
        "    model with a \"chose_from = 1\", the model will\n",
        "    always argmax. But with a value higher than 1,\n",
        "    this function will sample a token, randomly from\n",
        "    the top n (chose_from = n) of the distribution\n",
        "    \"\"\"\n",
        "\n",
        "    logits, indices = tf.math.top_k(logits, k=chose_from, sorted=True)\n",
        "    indices = np.asarray(indices).astype(\"int32\")\n",
        "    preds = keras.activations.softmax(tf.expand_dims(logits, 0))[0]\n",
        "    preds = np.asarray(preds).astype(\"float32\")\n",
        "    return np.random.choice(indices, p=preds)\n",
        "\n",
        "\n",
        "def detokenize(number):\n",
        "    \"\"\"\n",
        "    Function to turn tokens back into words ...\n",
        "    \"\"\"\n",
        "    return vocab[number]\n",
        "\n",
        "def generate_text(start_tokens, generate_tokens, vobab, chose_from):\n",
        "    \"\"\"\n",
        "    This function takes as input a sequence of \"start_tokens\"\n",
        "    (a prompt), a number of tokens to be generated, a vocabulary,\n",
        "    and the \"chose_from\" parameter. The function will output a\n",
        "    sequence of words, generated by the model, using the sampling\n",
        "    method established. You can change the number of the \"chose_from\"\n",
        "    argument to make the output less repetitive and more random.\n",
        "    \"\"\"\n",
        "\n",
        "    start_tokens = [_ for _ in start_tokens]\n",
        "    num_tokens_generated = 0\n",
        "    tokens_generated = []\n",
        "\n",
        "    while num_tokens_generated <= generate_tokens:\n",
        "\n",
        "        pad_len = sequence_length - len(start_tokens)\n",
        "        sample_index = len(start_tokens) - 1\n",
        "        if pad_len < 0:\n",
        "            x = start_tokens[:sequence_length]\n",
        "            sample_index = sequence_length - 1\n",
        "        elif pad_len > 0:\n",
        "            x = start_tokens + [0] * pad_len\n",
        "        else:\n",
        "            x = start_tokens\n",
        "\n",
        "        x = np.array([x])\n",
        "        y = TextGenerator.predict(x, verbose=0)\n",
        "        sample_token = sample_from(y[0][sample_index], chose_from)\n",
        "        tokens_generated.append(sample_token)\n",
        "        start_tokens.append(sample_token)\n",
        "        num_tokens_generated = len(tokens_generated)\n",
        "\n",
        "    return \" \".join([detokenize(_) for _ in start_tokens + tokens_generated])\n",
        "\n",
        "start_prompt = \"debates about artistic matters\"\n",
        "start_tokens = [vocab_index.get(_, 1) for _ in start_prompt.split()]\n",
        "generate_tokens = 50\n",
        "\n",
        "generate_text(start_tokens, generate_tokens, vocab, 1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ewaoq4D8956c"
      },
      "source": [
        "As expected, the model knows \"_something about 17th-century aesthetics_.\" However, if you increase the sampling parameter you will see that the model will start to generate gibberish. This is to be expected, given that we're dealing with a small model trained on even less text.\n",
        "\n",
        "In the end, good language models require a lot of training and data. If you would like to train more capable models, you need to invest in computing, given that training rounds can easily last days, weeks, and even months. In this [repository](https://github.com/Nkluge-correa/Aira), we have the code for training models like BERT and GPT-2 from scratch using the `transformers` library.\n",
        "\n",
        "You can find two already models (`bert-base-wikitext`, `bert-base-bookcorpus`) trained on distinct datasets.\n",
        "\n",
        "> Note: These models were trained on an RTX 3070 for approximately 15 days with a batch size of 8.\n",
        "\n",
        "Let us briefly test one of these models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qS3qffsdBxKU",
        "outputId": "d4c2a0b7-2109-4ca9-f363-778a91aab5e0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Result 1:\n",
            "Score: 0.9921931624412537\n",
            "Token: 3007\n",
            "Token String: capital\n",
            "Sequence: paris is the capital of france.\n",
            "\n",
            "\n",
            "Result 2:\n",
            "Score: 0.0008999903220683336\n",
            "Token: 2803\n",
            "Token String: centre\n",
            "Sequence: paris is the centre of france.\n",
            "\n",
            "\n",
            "Result 3:\n",
            "Score: 0.0006859182612970471\n",
            "Token: 2540\n",
            "Token String: heart\n",
            "Sequence: paris is the heart of france.\n",
            "\n",
            "\n",
            "Result 4:\n",
            "Score: 0.0004766975180245936\n",
            "Token: 2415\n",
            "Token String: center\n",
            "Sequence: paris is the center of france.\n",
            "\n",
            "\n",
            "Result 5:\n",
            "Score: 0.0004332299577072263\n",
            "Token: 2148\n",
            "Token String: south\n",
            "Sequence: paris is the south of france.\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "from transformers import pipeline\n",
        "\n",
        "model = 'AiresPucrs/bert-base-wikitext'\n",
        "pipe = pipeline('fill-mask', model=model, tokenizer=model)\n",
        "\n",
        "def unmask(string):\n",
        "    outputs = pipe(string)\n",
        "    for i, result in enumerate(outputs):\n",
        "        print(f\"Result {i+1}:\")\n",
        "        print(f\"Score: {result['score']}\")\n",
        "        print(f\"Token: {result['token']}\")\n",
        "        print(f\"Token String: {result['token_str']}\")\n",
        "        print(f\"Sequence: {result['sequence']}\")\n",
        "        print(\"\\n\")\n",
        "\n",
        "unmask(\"Paris is the [MASK] of France.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yN6MI4kKM7Sp"
      },
      "source": [
        "To summarize, developing a `language model` is a complex task that requires knowledge of `natural language processing`, machine learning, and resources, like data and computing power. Also, remember that developing a good language model requires a large amount of data and computational resources, so don't be discouraged if your first attempts don't yield cutting-edge results.\n",
        "\n",
        "---\n",
        "\n",
        "Return to the [castle](https://github.com/Nkluge-correa/TeenyTinyCastle)."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.13"
    },
    "orig_nbformat": 4,
    "vscode": {
      "interpreter": {
        "hash": "aca09746cf57686f00a55ae76e987247ecfb5dd0b3b2e2474d8dbbf0c5e3377e"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}